Since the take over of VLEsas the main environment for distance learning (Keppell, Souter & Riddle,2011), learners have achieved a higher degree of freedom to control their learning pro- cess and adapt their learning experience to their own needs. Often, we can see how VLEs are used for personalization and self-regulated approaches (McLoughlin & Lee,2010).VLEspermit students to access learning resources, with a potential amount of additional features and tools that might not be mandatory, and that allow them to personalize their learning paths. Furthermore,
VLEsallow students to communicate and collaborate remotely on learning activities (Dabbagh & Kitsantas,2012). Over the last years, pedagogy has increased the weight on giving more respon-
sibility and control to learners (Kay,2001) which is beneficial for their actual learning outcomes (Carneiro, Lefrere, Steffens & Underwood,2012). The main idea with self-regulated learning is that students should master a process that involves goal setting and planning, monitoring and con- trol processes, as well as reflection and evaluation processes (Schon,1984;Bolton,2010). Within this self-regulated settings students can decide which items or activities they want to use. We can roughly divide it in regular or mandatory activities, that are those required to be completed by students in order to achieve a passing grade (e.g. graded exercises or videos) and those who are completely optional and might not even be related to the learning process (e.g. setting an avatar picture). This degree of optionality depends on each specific case study, e.g., in some occasions forum activity might be mandatory, in other cases completely optional.
If we assume that one of the main objectives ofMOOCsandSPOCsis that students complete the proposed courseware in a correct way (e.g., their interaction with videos or educational activ- ities). Therefore, it is necessary to define metrics that can accurately measure the effectiveness of students with the courseware. These metrics can help to determine how students progress in the course according to the proposed are activities. Nonetheless, in the literature we find that most metrics that are used to evaluate the effectiveness of students are very simple (e.g, num- ber of videos completed or number of exercises solved correctly) and usually these metrics are not adapted to the specificities of the educational context. For example, the study carried out by
Dyckhoff et al. (2013) shows a compilation of indicators used in different studies in the literature, showing that most of them are simple indicators such as number of threads started by a student, number of assignments submitted or number of pages viewed. These indicators do not take into account how educational resources and activities were structured or how they are related to each other.
The traditional educational literature defines the concept of effectiveness from a perspective of amount of learning, if we quote the work ofHiltz & Arbaugh(2003) the definition is as fol- lows, “how much did the students learn, how well did they master skills and how well can they apply knowledge”. The concept of effectiveness applies for different educational settings such as face to face lessons or blended learning, but it might have become even more important for pure online learning courses where instructors cannot establish physical bonds and analyze the behavior of students in class so easily. Consequently, there is a need to design alternative methods to measure students’ effectiveness (Ni,2013;Swan,2003). One of the most common methods to measure learning effectiveness is the application of achievement tests or surveys (Moody & Sin- dre,2003). Nevertheless, this might not be always available. In addition, each environment might need specific definitions to measure the effectiveness e.g., Serrano-Laguna, Torrente, Moreno- Ger & Fern´andez-Manj´on(2012) uses the source of data from an educational game to feed aLA
system to infer knowledge about the effectiveness of the students. One possibility is to analyze the effectiveness separately as suggested by Swan. Another possibility by Rourke, Anderson, Garrison & Archer (2007) is to measure effectiveness in terms of interactivity with peers (so- cial presence), with instructors (teaching presence) and with contents (cognitive presence). This
ware can help to delve into different behavioral profiles such as for example ‘copy and paster’, ‘hint abuser’, ‘hint avoider’, ‘student misuse’, ‘video avoider’, ‘unreflective user’ or ‘procrasti- nator’ (Blikstein,2011;Aleven, McLaren, Roll & Koedinger,2004;Aleven et al.,2006;Mu˜noz- Merino et al., 2013;Baker, Corbett, Koedinger & Wagner, 2004; Baker, Corbett & Koedinger,
2004;Tervakari, Marttila, Kailanto, Huhtam¨aki, Koro & Silius,2013). We can analyze the influ- ence of these different indicators on different outcomes, e.g., factors that might affect teaching effectiveness (Kyriakides, Christoforou & Charalambous, 2013), factors of student persistence (Hart,2012), relationship of different behaviors with learning gains (Aleven et al.,2006) or anal- ysis of what items can increase student engagement (Wankel & Blessinger,2012). However, most e-learning platforms are still providing just rough insight (usually just the number and the grade of the activities completed) regarding the interaction of students with the educational resources. There is a need for more precise strategies to measure the effectiveness of students that can take into account the structure of the activities and other specificities such as the relationship between the different items in a course. As part of this dissertation, we have analyzed the relationship between the effectiveness of students with other variables and also with the purpose of student profiling (Mu˜noz-Merino, Ruip´erez-Valiente, Alario-Hoyos, P´erez-Sanagust´ın & Kloos, 2014;
Mu˜noz-Merino, Ruip´erez-Valiente, Alario-Hoyos, P´erez-Sanagust´ın & Delgado Kloos,2015). Additionally to the aforementioned learning activities, there are other activities that might not be mandatory or required to effectively complete the learning process. These activities can be defined as optional for students. For example, Mu˜noz-Merino, Delgado Kloos, Seepold & Garc´ıa(2006) analyzed which tools and functionalities that are provided by theVLEsMoodle3 and .LRN4are the most important regarding students’ perception. Some of the most highly rated were optional activities such as the use of forums or visualizations regarding their status. This shows that students also care about extra functionalities. Koedinger et al. (2015) compared the effect of passive and active learning. They found that only watching videos can be predictive of dropout and those who completed activities were more successful than just watching videos or pages. In addition, they also found that the combination of both passive and active learning lead to the highest success rates. Santos, Klerkx, Duval, Gago & Rodr´ıguez(2014) analyzed the activities conducted by learners in two languageMOOCs and they found that a higher activity in the forum correlated with students’ success. This is in line with the findings of the study conducted by Cheng, Par´e, Collimore & Joordens(2011) with over 2.000 students that found that students who participated voluntarily in forums also performed better in the course. Other works that explored activities that can be regarded as optional, are for example the one carried
3https://moodle.org 4http://www.dotlrn.org/
out by Gaˇsevi´c, Mirriahi & Dawson(2014) in terms of video annotation. They compared two courses, in the first one annotations were graded and in the second annotations were non-graded. Their findings suggest that students in the group of graded annotations, were able to use and develop more complex language indicators as a result of a potential more complex cognitive process. The study by Coetzee, Fox, Hearst & Hartmann(2014) with a reputation system for forum activities, suggested that students who were actively using the forum performed better and at the same time the use of the reputation system produced faster and more numerous post responses. On the other side, a study by Davies & Graff (2005) suggested the contrary, that forum activity alone is not enough to lead to higher grades, at least in their context. A study by
Mu˜noz-Organero, Mu˜noz-Merino & Kloos(2010) found that participating in e-learning activities (such as forums) or uploading a profile photograph was positively correlated with the motivation and final grade of students. During this dissertation we analyze the relationship between the use of optional activities and learning outcomes (Ruip´erez-Valiente, Mu˜noz-Merino, Delgado Kloos, Niemann & Scheffel,2014;Ruip´erez-Valiente, Mu˜noz-Merino, Delgado Kloos, Niemann, Scheffel & Wolpers, 2016). More specifically, we look into the use of feedback, votes, badge display, avatar image and setting up learning goals in Khan Academy (this optional activities are described in Subsection3.1.1.1). We present the relationship between using certain regular and optional activities with other learning indicators. We also delve into how the behavior and activity of students might relate to learning outcomes.