Analyzing Users Data Captured in Learning Management
Systems
1
1 Presented at the DAILE13 Workshop on Data Analysis and Interpretation for Learning Environments , Alpine Rendez-Vous 2013, Villard-de-Lans, France, January 28th – 30th.
Agathe Merceron
Beuth University of Applied Sciences Luxemburgerstrasse 10 13353 Berlin +493045045105
ABSTRACT
Though Learning Management Systems are widely used in education, the usage data they store is not analyzed in a routine basis by different stakeholders to retrieve pedagogical information. The aim of this contribution is to discuss which functionalities an analysis tool should offer, which techniques are mature enough to implement these functionalities and how to structure them. The analysis tool should be decoupled from any particular LMS, therefore the need of a data model at the interface of the two systems.
Categories and Subject Descriptors
H.1.1.2 [Information Systems]: Models and Principles – User/Machine Systems human information processing.
General Terms
Management, Design, Human Factors.
Keywords
Learning Management Systems, Learning Analytics, Tools.
1.
INTRODUCTION
Though Learning Management Systems (LMS) are widely used in education, the usage data they store is not analyzed in a routine basis by different stakeholders to retrieve pedagogical information that could support reflection and help improve the learning experience of students or the teaching experience of teachers. For example, if a teacher notices that some resource she has uploaded is hardly used, she might do some further analysis: does it seem to have a positive impact on the mark of the final exam for the few students who accessed it? If not, she might consider deleting it from the course for the next semester; if yes, she might change her teaching style so that students consult this resource more. For this kind of routine analysis to happen, tools are needed that stakeholders can handle to conduct the analysis they need. An analysis tool should be decoupled from any particular LMS, therefore the need of a data model at the interface of the two systems. The data model should represent users’ data stored commonly by LMS. The purpose of the analysis tool is not to track any particular student but to support reflection of stakeholders for strategic thinking or future action. An analysis must provide several functionalities that match the needs of stakeholders and these functionalities have to be implemented by techniques that are robust and mature enough so that results can
be easily interpreted. As noticed in [11] the fear of misinterpreting results is a reason not to use analysis tools. Furthermore analysis functionalities have to be flexible enough and yet well structured for the tool to be intuitive and user friendly, similar to a text editor or a browser: a glance at the menu bar tells the possibilities of the tool.
This paper presents work in this direction focusing primarily on study program managers, content providers and teachers as stakeholders. It presents succinctly a model for usage data stored by Learning Management Systems to be mapped into. Mainly it discusses functionalities structured in five areas that such a tool could/should offer as well as analysis techniques to implement these functionalities. These five areas could constitute the menu bar of such a tool.
1.1
Background
The starting point for the functionalities presented in this contribution is a catalogue established with different stakeholders: study program managers, content providers, teachers and researchers [2]. This catalogue describes about 80 requirements expressed informally in natural language to such a tool. These requirements have been analyzed and structured in the five areas presented below. It is interesting to note that most of the works presented in the educational data mining conferences on data stored by LMS can be mapped in these five areas.
1.2
Related Works
A number of research works analyzing data stored by LMS has already been published. They investigate how some specific information can be retrieved. They are important as they might give clues on which routine analysis could be conducted and on which techniques are mature enough. To cite a few presented in the educational data mining conferences, [10] investigates whether it is possible to predict if students can answer a question without pressing the help button. The impact on the exam mark of using specific learning objects is studied in [5, 9, 15]. Many works such as [1, 16, 17, 20] investigate how to predict performance in the final exam.
A number of tools to be used by different stakeholders are also under development. AAT [7] for example is intended for instructional designers and can be connected to various LMS via templates; a user can select a functionality or pattern from a list, or define a new one; a pattern is based on some SQL query; currently the list of patterns does not seem to be structured, however analyses can be stored in profiles. A tool specific to the
LMS Moodle is under development in [14]; the functionalities offered by the tool from a teacher’s perspective are not clear. GLASS [13] and [2] present further tools under development; GLASS is based on the CAM schema [19] while [2] is based on the data model presented below. To the best of my understanding these tools do not have yet a definitive set of functionalities structured in some way.
The work described in [11] is most similar to the present approach. It presents requirements coming from the following stakeholders: administrators, study program managers, teachers and students. An interesting aspect of [11] is to present use cases for monitoring. The indicators presented for teachers are covered by the area interactions with the learning offer presented below. [11] does not present requirements regarding the areas clustering students, understanding or predicting performance described in this contribution. Furthermore [11] presents different tools to implement indicators coming from different stakeholders, whereas the ultimate aim of the present work is to come up with a single tool versatile enough to be useful for and usable by different stakeholders.
2.
DATA MODEL
It is not uncommon in Germany for a company or an educational institution to use several learning management systems in parallel. The analysis tool should not be restricted to one of them. This is a first reason to define a data model in which to import usage data from a particular LMS into. Second, laws concerning data privacy and security are quite strict in Europe. Personalized log data stored by LMS have to be erased after a period of time that can be as short as a few days. The import module anonymizes the data, which allows to keep historical data and analyze them, as the aim of the tool is not to track students but to discover trends. A third reason is to organize the data in a way better suited for analysis. The data model makes several requirements to an LMS that are now exposed. First, users can register in the LMS and the learning offer is structured in courses, possibly also in degrees and departments. A course can be offered in different degrees and a degree may contain several courses. Users can enroll or sign in courses and sign off courses. Users can have roles like “lecturer”, “administrator”, “tutor”, “student” and so on. A user may have different roles in different courses. For example a user can be a tutor in the course “Introduction to Programming” and a student in the course “Early American History”. An LMS may contain groups that are associated to courses. Students enroll in those groups. An LMS contains different kinds of learning objects such as forums, wikis, chats, resources (files that can be viewed such as texts, slides, pictures etc.), quizzes or assignments. All these learning objects are associated to courses. Thus a resource for example, can be used in several courses. A quiz may contain one or more questions and, conversely, a given question can be associated to several quizzes. These general assumptions cover the particular case of LMS where learning objects exist only inside one given course. In this particular case an association table contains only one tuple. An LMS logs or stores interactions of
users. For any given interaction, the LMS stores the identification of the user, of the course, of the learning object, as well as the timestamp, the nature of the interaction (“view” which corresponds to a simple hit, “download”, “update”, “creation”, “attempt”, “submit” and so on), the marks and the contribution when relevant, message in a forum or answer in a quiz for instance.
The data model adopted in [2, 3] bears similarities to a fact constellation schema [8] and is a slight extension of what has been published in [9]. It contains three kinds of tables: tables to describe objects found in LMS (user, department, degree, course, resource, quiz and so on), these tables can be seen as dimension tables; tables to describe interactions with learning objects (usually an entry of such a table is the user ID, the learning object ID, a time stamp, the nature of the interaction like view, post, attempt etc. and, when applicable user’s input), these tables can be seen as fact tables; and third, association tables to describe associations between objects (like a table containing all the courses belong to one degree). For each particular LMS it is then necessary to implement a connector that import the LMS usage data into this data model.
These requirements to the LMS have proven to be realistic enough to allow for the implementation of three connectors: one for the Moodle LMS, one for the Clix LMS and one for the Chemgapedia learning portal [4,3]. Not all tables need to be filled. For example data extracted from Chemgapedia do not contain any quiz and the corresponding tables simply do not exist.
3.
PEDAGOGICAL INFORMATION AND
POSSIBLE ANALYSIS TECHNIQUES
Synthetizing the requirements coming from the project [2] and taking into account analysis questions tackled in the educational data mining community lead to the following five areas to structure the pedagogical information that is the most interesting for teachers and study program managers: Viewing what is in the learning offer, exploring how users interact with the learning offer, clustering students, understanding performance and predicting performance. Most of the concrete examples given in this section have been obtained with the prototypes [3, 5]. None of these prototypes contain yet all the functionalities envisioned in this contribution.3.1
What is in the learning offer?
Study program managers need to have an overview of what is in the learning offer. The data model makes it possible to retrieve a list of all department, degrees and courses offered in the LMS, and, inside a course, of all learning objects uploaded there. This overview can be filtered to obtain for example, only a list of the quizzes in a specific course. This information can be obtained with simple queries and the result returned as a table.
3.2
Interaction with the learning offer
Information on how the learning offered is used is needed by several stakeholders. For instance, study program managers are interested in questions like: do teachers post enough messages in forums? Study program managers, content providers and teachers are interested to know whether and how students use learning objects in courses. Content providers have questions like: in which sequence do students access the different topics of a course?
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Conference’10, Month 1–2, 2010, City, State, Country. Copyright 2010 ACM 1-58113-000-0/00/0010 …$15.00.
It turns out that requirements concerning interactions with the learning offer have two perspectives. The first one is quantitative and bears similarities to reporting sales in commercial applications. Managers want to know the total sales per year, then details over time, per quarter or month, details per types of items like electronics, details per items themselves, and possibly details per location. Instead of sold items LMS have learning objects that can be accessed by several kinds of users and in several ways: students or teachers can read or post a message in a forum for instance. The second perspective has similarities to Web-Mining: how do students navigate through learning objects? These two perspectives are tackled in turn.
3.2.1
Indicators for interactions
3.2.1.1
Overview, filters and details
Three useful ways of measuring interactions emerged from the requirements: (i) summing all accesses - multiple accesses of users to a single object are all counted; (ii) counting the number of users who have accessed the objects; (iii) summing the duration of access to objects. In each case, overview, filters and details give useful information as explained below taking the case (i), summing all accesses, as an example.
The coarsest overview is given by the total number of interactions with the LMS. This gives a single number. Drilling down the structure departments, degrees and courses gives the total number of interactions at each of these levels. Filters help focusing on particular aspects of the interactions with learning objects inside a course: (i) a particular type of learning object like forums or file resources for instance, (ii) a particular period of time like Christmas holidays, (iii) particular action like “view”, or “post” for forums, or “attempt” for a quiz (iv) particular users like users with the role “student”. Details can show each particular item inside the chosen filter.
Several kinds of diagrams to visualize the results are useful. Figure 1 shows the number of accesses in a course with a filter on the type of learning object set to “resource”, a filter to restrict the time to a semester, a filter on the action “view”, and a filter on users with the role “student”, detailed per resource [5]. The histogram is ordered so that at a glance it is possible to catch that the resource most viewed by students during the semester was the first set of slides on the left called “slides01”.
Figure 1. Number of view interactions by students to resources of a course during one semester
Figure 2 shows the number of accesses in a course with the filter on the type of learning object set to “quiz”, a filter to restrict the time to a semester, no filter on action, and a filter on users with the role “student”, detailed per quiz and student [5]. Because there is no filter on action, all interactions with quizzes, viewing as well
as attempting a quiz, are counted. The different colors give the number of interactions. This diagram shows that most students interacted with quizzes 1 to 7. Student 11373, third from left, is an exception.
Figure 2. Number of interactions by students with quizzes of a course during a semester
Of course it is also possible to calculate more figures like average number of accesses and so on.
The same approach “overview, filters and details” works for (ii) counting students who have accessed the objects and (iii) for duration. Duration has to be handled with care as this number does not necessarily tells the real time students have studied a particular object. Furthermore it is not always calculable.
3.2.1.2
Techniques
Simple queries or techniques from Data Warehouses and Online Analytical Processing (OLAP) [8] can be used to retrieve this information.
3.2.2
Navigation
Information on paths is needed by teachers and instructional designers to answer questions such as: what are the entering and outgoing objects? Do students follow pre-defined paths?
The model exposed in section 2 contains all data to extract all paths of all users through the objects of a course. A path is a sequence of tuples ordered by access time. A tuple has three items: a learning object, access time and duration of access. As above it is possible to filter the paths over time, a path contains only the tuples whose timestamp are in a given period, over duration, a path contains only tuples whose duration is in a given range, over type of object, a path is returned only if it contains at least an object of the filtered type, over action, or over users. Table 1 shows an overview of the length of paths through all learning objects on an interval of two months of a 1st semester course taught in face-to-face teaching at Beuth University of Applied Sciences, Berlin in winter semester 2012. 37 students were enrolled in the course.
Table 1. Overview of students’ paths through all learning objects in one course during two months
Nb. Paths of length 0 – 10: 5 Nb. Paths of length 11 – 20: 10 Nb. Paths of length 21 – 30: 6 Nb. Paths of length 31 – 40: 7 Nb. Paths of length 41 – 50: 4 Nb. Paths of length 51 – 60: 2 Nb. Paths of length 61 – 70: 1 Nb. Paths of length 81 – 90: 2
A visualization of paths as given with Google Analytics is not likely to work because in many educational settings students are completely free in the way they navigate through the LMS or the learning portal. Furthermore it might be more useful to show a summary of the paths instead of a set of individual paths. One possible summary is aggregating all the paths as shown in Figure 3. Circles are learning objects, the color gives their type. There is a link between two learning objects if some user navigated from one to the other. The bigger the circle the more the LO was accessed. Mouse over a circle gives the name and type of the LO and the number of accesses, mouse over a link shows its direction. With this summary, the sequence of an individual path is lost.
Figure 3. Aggregated paths of 37 students through all learning objects of a course during 2 months
Another possible summary is to show the frequent sequences in the sense of [8] as explained now. Let us suppose that an LMS has six objects o1 till o6 and that there are two paths; the path of student s1 is: <o3, o2, o1, o3> and the path of students s2 is: <o2, o3, o4>. Then <o2>, <o3> and <o2, o3> are 3 sequences that are contained in both paths, their support is 2. Sequence <o3, o3> is contained in the path of student s1 only, its support is 1. The sequence <o2, o3> is called a closed sequence because it strictly contains others sequences with the same support. Closed frequent sequences might be useful for teachers, as they give them the information on the sub-paths followed by a number of students. Note that closed frequent sequences give only the information about sequence. Nothing is known about time of access or duration. It could be that s1 accessed the objects in January and spent never more than 2 minutes on each object while student s2 accessed them in February and spend no less than 10 minutes on each object if filters on time and duration allows for such ranges. Therefore it is important to set the right filters on the paths before extracting closed frequent sequences.
As an example, the closed frequent sequences with a support of at least 25% extracted from the 37 paths of Table 1 had a length from 1 to 9, among them six closed frequent sequences of length 8 and one of length 9. These longest sequences had a support of 10
or 11 (little less than 30% of the students) and were showing that sample exams were accessed before their solutions.
3.2.2.1
Techniques
The extraction of paths has to be implemented using the Data Model of section 2. There are a number of algorithms that extract (closed) frequent sequences like The BIDE-Algorithm [16]. First experiences integrating this algorithm in the LeMo tool [3] show that choosing a support so that frequent paths are retrieved in a few minutes can be challenging. Problems have occured when a number of long paths contain similar objects.
3.3
Finding groups of students
It is interesting for study program managers to know whether there are students with a similar behavior, which can be defined in various ways. Also quite often teachers need to group their students. Sometimes they need to put similar students together in one group; sometimes on the contrary, they need to form groups with students as diverse as possible. Students can be similar or different along several dimensions: the type of learning objects they interact with and the action they performed while interacting, how often or how long they interact with learning objects, their navigation, the marks they have earned and so on. All these features can be used to compare students. It is up to teachers to choose on which dimensions students should be compared.
3.3.1
Techniques
Clustering allows to find groups or clusters of similar individuals. Two clustering methods are often used: K-means clustering and EM-Clustering [8]. These two methods need K, the number of expected clusters. Fortunately, teachers quite often know the number of groups they want to have.
A problem with any clustering method is, that a set of groups will always be found, even if the data is scattered randomly. Hence the quality of the set of found clusters has to be checked. To overcome this problem, clusters should be found not only for K, the value given by the teacher, but also for variations of it, like K+1, K-1 and so on. For each set of clusters found its quality should be measured with for example the Sum of Squared Error (SSE) [8]. The set of clusters with the sharpest drop of SSE should be returned. If there is no such a sharp drop, meaningful clusters might not exist.
3.4
Understanding Performance
Requirements concerning understanding performance can be divided into two groups. The first group is similar to interactions with learning objects, except that now a teacher needs to know the marks obtained by students in quizzes and assignments, and to know the number of mistakes that are made while solving quizzes. The second aspect has to do with investigating learning style and success. These two aspects are tackled in turn.
3.4.1
Reports on marks and mistakes
“Overview, filter and details” can be transferred to marks for quizzes or assignments calculating average and standard deviation. Details are to compare them: are there tests where students achieve better/poorer marks? Details on students is to know whether students are homogeneous: do they succeed similarly on the same quizzes or assignments?
Filter on the action “attempt” logged each time a student attempts a quiz, or on the action “upload” each time students upload an answer to an assignment shows how often quizzes or assignments have been solved. Using the answers stored to quizzes it should be possible to sum the wrong answers or mistakes, giving details per
quizzes and / or per students. It is thus possible to compare quizzes and students regarding mistakes made.
3.4.2
Impact of interaction on performance
Very important for teachers is to understand whether some particular ways of interacting with the learning objects, or more broadly learning styles, lead to better success than other ways. One approach is to split students into groups according to their results: students in one group have similar results, then explore interactions with the learning offer one group after the other as exposed in the previous sections and then compare the findings. Another approach borrows the idea of A/B-Tests in ecommerce applications or multivariate statistics. For example, suppose that a teacher wants to investigate whether attempting the non-compulsory self-test1 has a positive impact on the mark of the exam. Students should be split into two groups, those who have attempted self-test1 and those who did not attempt it. Then average mark in the final exam is computed for both groups and the results can be compared. If size of groups allow (at least 30 students per group) it should be checked whether the difference is statistically significant. This approach can be used in a very flexible way, as shown in Table 2 which gives the minimum mark, maximum mark, average and standard deviation to a final exam in general, first line, then for students having solved Exercise 1, second line, then for students having solved Exercise 2 and so on [9]. The last column gives the average mark for students who did not solve any exercise, first line, then for students who did not solve Exercise 1, second line, and so on.
Table 2. Completing self-evaluation exercisesand marks in the exam
Exercise min max mean s. deviat. meanNoEx
General 1 13 8.63 4.34 7 Ex1 1 13 9.48 4.06 7.33 Ex2 1 13 9.56 3.97 7.95 Ex3 1 13 9.2 4.4 8.26 Ex4 1 13 8.56 4.77 8.68 Ex5 1 13 9.21 4.54 8.29 Ex6 1 13 10.91 3.4 7.70 Ex7 9 13 11.67 1.41 7.69
As another example, suppose that a teacher wants to investigate whether a particular navigation is beneficial to learning. Again students should be split into two groups, those whose path contains the special sequence, and those who do not. As before average mark in the final exam can be compared and the result checked for significance if groups are big enough.
3.4.3
Techniques
Queries or techniques from data Warehouses are appropriate to extract a subgroup that has accessed a particular object like self-test1.
I am not aware of any algorithm to extract group of students whose path contains a given sequence. However it is possible to program it as a linear search through all students’ paths, once these have been extracted.
3.5
Predicting Performance
Equally important for teachers is to be aware of students most likely to do poorly in a course so that they can be proactive and help them to achieve a better performance.
Most of the works published on that topic analyze courses in which a number of quizzes and assignments have to be done by students and are graded by the teacher before the final exam. The results on those quizzes and assignments, and possibly use of other objects like script-resources, forums and so on, are used to predict performance in the final exam. Clearly, if there are very few assignments, data contained in the LMS might be too poor for this kind of investigation. Further, the model built over at least one semester will be used for prediction in the next semester, hence the course has to keep the same objects.
3.5.1
Techniques
Prediction is done using classification techniques. For teachers classification techniques giving a result that is easy to understand and interpret are preferable. This is why decision trees [8] are particularly well suited. The data of one semester will be used to build the tree, and this tree can be used by teachers in the following semester to track students at risk. This requires however that no changes in the learning objects are made. Better results are obtained if marks in the final exam can take a small number of different values compared to the number of students. When this condition is not met, values should be grouped and a range for the mark, instead of the mark itself, can be predicted. It is important to return not only the decision tree to the teacher, but also the confusion matrix, so that teachers can judge the accuracy obtained by the tree.
4.
CONCLUDING REMARKS
So far the data model presented here has proven adequate to map users’ data from the LMSs Moodle and Clix, as well as from the Learning Portal ChemgaPedia [3]. Is this model also suited for other educational systems like intelligent tutoring systems, or simulation environments? A main difference between LMS or learning portal on one side and intelligent tutoring systems or simulation environments on the other side is the learner model, possibly also the teacher model, which makes the latter systems more complex. Would it be beneficial to substitute the data model presented here by the CAM schema [19]?
The functionalities presented in this contribution came up from the requirements expressed by the stakeholders of the LeMo project [2], taking also into consideration works related to LMS published in the educational data mining conferences. They answer or give some support to answer about 70% of the requirements of the LeMo project [2]. The answer to some questions is not obvious like “Have students deepen the course?” even challenging like “Is there any lack space or missing link in the material?” coming from content providers. Techniques to analyze usage of communication tools such as forums are presently limited to the general techniques given in the area interactions, which does correspond to most of the requirements obtained from our partners: “Wow many have used the forum? When?” etc. It could be worthwhile to have a supplementary area dedicated to communication / collaboration to analyze the network of the communication: who communicates with who, or the content of the communication: what topics are addressed, what opinions are exchanged. A few questions concern the content of communication like “are there discussions / questions that are recurrent in forums?”
5.
ACKNOWLEDGMENTS
This work is partly supported by the Institute für Angewandte Forschung Berlin and the European Regional Development Fund for the Berlin state project “LeMo” as well as by the “Berlin Senatsverwaltung für Wirtschaft, Technologie und Forschung” with funding from the European Social Fund. I thank my colleagues Liane Beuster, Helena Dierenfeld, Margarita Elkina, Albrecht Fortenbacher, Leonard Kappe, Andreas Pursian, Sebastian Schwarzrock and Boris Wenzlaff for their coorperation.
6.
REFERENCES
[1] Agapito, J.B., Sosnovsky, S., Ortigosa, A.. 2009. Detecting Symptoms of Low Performance Using Production Rules. In
Proceedings of the 2th International Conference on
Educational Data Mining (Cordoba, Spain , July 6-10, 2009), EDM2009, 31-40.
[2] Beuster, E., Elkina, M., Fortenbacher, A., Kappe, L., Merceron, A., Schwarzrock, S., Wenzlaff, B. 2012. LeMo-Lernprozessmonitoring auf personalisierenden und nicht personalisierenden Lernplattformen. In Proceedings of the GML2 Grundfragen des Multimedialen Lehrens and Lernens Conference (Berlin, Germany, March 15-16, 2012) . Waxmann Verlag, 63-76.
http://www.gml-2012.de/tagungsband/Tagungsband_GML2012_web.pdf [3] Beuster, E., Elkina, M., Fortenbacher, A., Kappe, L.,
Merceron, A., Schwarzrock, S., Wenzlaff, B. 2012. Prototyp einer plattformunabhängigen Learning Analytics Applikation – fokussiert auf Nutzungsanalyse und Pfadanalyse. In
Proceedings of the Elearning Symposium 2012 (Potsdam Germany, November 17, 2012), to appear.
[4] ChemgaPedia homepage, http://www.chemgapedia.de [5] Dierenfeld, H., Merceron, A. 2012. Learning Analytics with
Excel Pivot Tables. In Proceedings of the 1st Moodle
Research Conference (Heraklion, Greece, September 14-15, 2012), 115-121.
[6] Dierenfeld, H., Merceron, A. 2012. Information Retrieval in Learning Systems.
http://wissen.beuth-hochschule.de/datamining/node/4
[7] Graf. S., Ives, C., Rahman, N. and Ferri, A.. 2011. AAT – A Tool for Accessing and Analysing Students’ Behaviour Data in Learning Systems. In Proceedings of the Conference on Learning Analytics & Knowledge (Banff, Alberta, Canada, February 27 – March 01, 2011). LAK2011, ACM, New York, NY, USA.
[8] Han, J. and Kamber, M.. 2007. Data Mining: Concepts and Techniques. Morgan Kaufman Publishers.
[9] Krüger, A., Merceron, A., Wolf, B. 2010. A Data Model to Ease Analysis and Mining of Educational Data. In
Proceedings of the 3th International Conference on
Educational Data Mining (Pittsburg, USA , June 11-13), EDM2010, 131-140.
[10]Mavrikis, M. 2008. Data-driven modelling of students' interactions in an ILE. In Proceedings of the 1th International Conference on Educational Data Mining (Montreal, Canada, June 20-21, 2008), EDM2008, 87-96.
[11]Mazza, R., Bettoni, M., Faré, M. and Mazzola, L. 2012. MOCLog – Monitoring Online Courses with log data. In
Proceedings of the 1st Moodle Research Conference (Heraklion, Greece, September 14-15, 2012), 132-139. [12]Merceron, A., Yacef, K. 2008. Interestingness Measures for
Association Rules in Educational Data. In Proceedings of the 1th International Conference on Educational Data Mining (Montreal, Canada, June 20-21, 2008), EDM2008, S. 57-66. [13]Pardo, A. Delgado Kloos, C., Sánchez De Castro, D., Derick
Leony, De La Fuente Valentín, L. 2012. GLASS: A Learning Analytics Visualization Tool. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (Vancouver, Canada, April 30 – May 3 2012). ACM, New York, NY, 2012.
[14]Pedraza-Perez, R., Romero, C., Ventura, V. 2011.A Java Desktop Tool for Mining Moodle Data. In Proceedings of the 4th International Conference on Educational Data Mining, (Eindhoven, the Netherlands, July 6-8), EDM11, 319-320. [15]Romero, C. ,Romero, J.R., Luna, J.M., Ventura, S. 2010.
Mining Rare Association Rules from e-Learning Data. In
Proceedings of the 3th International Conference on Educational Data Mining (Pittsburg, USA , June 11-13), EDM2010, 171-180.
[16]Romero, C., Ventura, S., Espejo, P.G., Hervas, C. 2008. Data Mining Algorithms to Classify Students. In Proceedings of the 1th International Conference on Educational Data Mining (Montreal, Canada, June 20-21, 2008), EDM2008, 8-17.
[17]Romero, C., Ventura, S., Vasilyeva, E., Pechenizkiy, M.
2010. Class Association Rules Mining from Students’ Test
Data. In Proceedings of the 3th International Conference on
Educational Data Mining (Pittsburg, USA , June 11-13), EDM2010, 317-318.
[18]Wang, J., Han, J. 2004. BIDE: Efficient mining of frequent
closed sets. In Proceeding ICDE '04 Proceedings of the 20th
International Conference on Data Engineering. IEEE Computer Society Washington, DC, USA, 79-90. www.cs.uiuc.edu/hanj/pdf/icde04_bide.pdf [19]Wolpers, M., Najjar, J., Verbert, K., Duval, E. 2007.
Tracking Actual Usage: the Attention Metadata Approach.
Educational Technology & Society, 10 (3), 106-121. [20]Zafra, A., Ventura, S. 2009. Predicting Student Grades in
Learning Management Systems with Multiple Instance Genetic Programming. In Proceedings of the 2nd International Conference on Educational Data Mining (Cordoba, Spain. July 1-3). EDM09, 309-318.
