8. Conclusions and Future Work
8.2. Future Work
There are many potential areas of future work as a follow up of the results achieved during this dissertation. In the direction of visual analytics with our platformsALAS-KAandANALYSE, one potential future action is the design and implementation of new and novel indicators as well as new visualizations e.g., a more in depth analysis of the social activity in these environments including for example social network analysis visualizations of their activity in the discussion forums using tools such as qgraph10. We would also like to integrate other tools to perform more advanced statistical analysis. Generally speaking, we would also want to implement several of the recommendations received by the respondents of the evaluation survey such as enabling different languages and clearing up the available interactivity of each visualization. A very challenging project would be to develop a modular and common framework of indicators and visualization that can be easily adapted to different courses and platforms, this would increment interoperability greatly. Another interesting line of work would be to compare the usability of these platforms between ‘digital natives’ and ‘forced digital immigrants’.
Based on our findings in prediction models and detectors we would like to integrate our lessons learned withinALAS-KAandANALYSE. For example, we can introduce a recommender system that when negative behaviors are detected, it can provide feedback to the student regard- ing how to improve his past negative action or an early warning that when a student is detected to be in risk of not achieving a certificate send a warning, so that the student can turnover this situation before it is too late. As part of our prediction model of certificate accomplishment, we analyzed the evolution of this model per week, but we would also like to carry out a more gran- ular approach, maybe a day-to-day analysis which would also help into analyzing the effect of other variables such as deadlines or days of the week. We would also like to try out forecasting algorithms that can be useful to find trends in the evolution of data as time goes by. One of the
8http://www.anc.ed.ac.uk/
9http://www.ed.ac.uk/profile/dragan-gasevic 10http://sachaepskamp.com/qgraph
most challenging projects would be to delve into the development of prediction models that can extrapolate to different topics, courses and even platforms. One idea to accomplish that would be the use of ratios instead of absolute variables, and use those as the input variables of the prediction model. The final stage of these ideas would be to perform A/B experiments in which a treatment group would receive some sort of recommendation or warning (e.g., positive or negative behav- iors, warning regarding risk of not getting a certificate) and the control group would not; then we can analyze if the treatment group improved their learning outcomes compared to the control group (higher learning gains or higher certificate accomplishment ratio).
In terms of the interaction of students with regular activities we would like to work towards defining new metrics that are able to efficiently characterize the effectiveness of students. For that purpose new experiments including pre-test and post-test to estimate students’ learning might be good, since we would be able to analyze which effectiveness metrics are more correlated with the actual learning achievement of students. In terms of optional activities, we plan to formulate a common framework that can be applicable to otherVLEsand also replicate our experiments in other platforms. We also want to conduct A/B testing experiments with optional activities where some students will have them available and others will not, so that we can measure with more reliability the actual impact of optional activities in the learning process of students.
Related to the behavior with badges, we would like to extend our models to be applicable to more badge types. Additionally, we would like to replicate our experiment in other environments to see if we obtain similar findings that can help us generalize. Another interesting direction would be to survey students regarding their interest about badges, and check for correlations between our metrics and the reported interest. This can be helpful to validate the effectiveness of our badge metrics. Finally, we would also like to delve into assessing the influence of interest in badges on learning achievement, specially in those cases of students that had really high intentionality repetitive badges.
Our research on online academic dishonesty also opens many potential future work lines. The most straightforward would be to examine both in terms of CAMEOandclose submitteralgo- rithms a larger portfolio ofMOOCs, but also maybe other types of online learning courses, such as on-campus blended learning or corporate training. We can gain insight in terms of general- ization and to find trends across courses depending on other factors such as university, topic or platform delivering the course. A broader analysis might enable also the implementation ofML
models that can work well across courses, this would permit developing run-time detectors that could be embedded withinVLEsor learning analytics systems such asANALYSE. We would also like to corroborate our findings and the effectiveness of our detectors by interviewing some of the detected students, although this would problematic for several reasons such as students might feel that their rights are violated, or they might also lie due to embarrassment and fear to potential repercussions due to their illicit behavior. On a more general way, there are probably many other methods that students might be using to facilitate their way into certificates, thus more work to- wards developing detectors would be useful. Finally, pedagogy-wise we would like to quantify
ANALYSE
BLOCK 1. Specific questions that require that you interact and carry out a certain task with ANALYSE in order to be able to give an adequate response. Each question is associated with an specific visualization:
1. Course Summary. What percentage of students have acquired a proficiency grade in ‘Homework’ assignments.
2. Students’ Grades. What is the grade of student ‘Verified’ in the midterm exam? 3. Problem Time Distribution. In which problems have the student ‘Audit’ spent the
most time? How much time?
4. Video Time Watched. Which video has highest difference between different video watched and total video watched by all students?
5. Repetitions of Video Intervals. In the video ‘Radioactive’, which approximate interval of seconds has been watched more times by all the class?
6. Video Time Distribution. In which video did the student ‘Staff’ spend more time? 7. Video Events Distribution. In the video ‘Passenger - Let her go’, what is the approx-
imate range of seconds where we can find more ‘Change Speed’ events by ‘Audit’ student?
8. Exercise and Video Progression. Check the progression for the student ‘jruipere’. Which score was higher on the date ‘15/02/2015’, video progress or exercise grades? 9. Daily Time on Exercises and Videos. Which day did users engage the largest amount
of time on problems with the platform? How much time?
10. Course Accesses. Which section of the course has the highest number of accesses by student ‘Honor’?
11. Chapter Time. How much ‘Graded time’ time has been spent by all students in section 2?
12. Students Time Schedule. Which is the time interval in which the student ‘JoseRuiperez’ has us28ed the platform the most? How many minutes?
BLOCK 2. Now that you have used all visualizations, rate from 1 (not useful at all) to 5 (very useful) the usefulness of each one of the visualizations from the perspective of an instructor:
13. Course Summary. 14. Students’ Grades.
15. Problem Time Distribution. 16. Video Time Watched.
17. Repetitions of Video Intervals. 18. Video Time Distribution. 19. Video Events Distribution. 20. Exercise and Video Progression. 21. Daily Time on Exercises and Videos. 22. Course Accesses.
23. Chapter Time.
24. Students Time Schedule.
BLOCK 3. Specific questions about the usability of the learning analytics extension. Mark from 1 (strongly disagree) to 5 (strongly agree) your degree of agreement with the following statements from the perspective of an instructor:
25. I think that the use of this application would help to evaluate students that are taking an online course more easily.
26. I think these visualizations are useful and help understand the learning process of students.
27. I think that these visualizations can be used to detect problems in learning resources of an online course (exercises, videos, etc).
BLOCK 4. System Usability Scale questionnaire. Mark from 1 (strongly disagree) to 5 (strongly agree) your degree of agreement with the following statements from the perspective of an instructor:
28. I think that I would like to use this web application frequently. 29. I found the web application unnecessarily complex.
33. I thought there was too much inconsistency in this web application.
34. I would imagine that most people would learn to use this web application very quickly. 35. I found the web application very cumbersome to use.
36. I felt very confident using the web application.
37. I needed to learn a lot of things before I could get going with this web application. BLOCK 5. Questions regarding best features and potential improvements allowing an open text response:
38. Which features and/or visualizations do you think are the most useful/important after interacting with ANALYSE?
39. Which features could be improved and what new functionality could be implemented to improve ANALYSE?
B.1.
JCR-Indexed
The journal publications indexed in theJournal Citations Report (JCR)are the following: 1. Ruip´erez-Valiente, J.A., Mu˜noz-Merino, P.J., Leony, D. & Delgado Kloos, C. 2015.
ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior. 47, 139–148 (Ruip´erez- Valiente et al.,2015). Impact factor as of 2015: 2.88, JCR(21/129), Q1, category: psychol- ogy, multidisciplinary.
2. Leony, D., Mu˜noz-Merino, P. J., Ruip´erez-Valiente, J. A., Pardo, A., Mart´ın-Caro, D. A., & Delgado Kloos, C. (2015). Detection and evaluation of emotions in Massive Open Online Courses. Journal of Universal Computer Science, 21(5), 638–655 (Derick Leony et al.,
2015). Impact factor as of 2015: 0.546, JCR(89/106), Q4, category: computer science, software engineering.
3. Mu˜noz-Merino, P. J., Ruip´erez-Valiente, J. A., Alario-Hoyos, C., P´erez-Sanagust´ın, M., & Delgado Kloos, C. (2015). Precise Effectiveness Strategy for analyzing the effective- ness of students with educational resources and activities in MOOCs. Computers in Hu- man Behavior, 47, 108–118 (Mu˜noz-Merino et al.,2015). Impact factor as of 2015: 2.88, JCR(21/129), Q1, category: psychology, multidisciplinary.
4. Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Delgado Kloos, C., Niemann, K., Schef- fel, M., & Wolpers, M. (2016). Analyzing the Impact of Using Optional Activities in Self-Regulated Learning. IEEE Transactions on Learning Technologies, 9(3), 231–243 (Ruip´erez-Valiente et al.,2016). Impact factor as of 2015: 1.129, JCR(78/231), Q2, cate- gory: education & educational research.
5. Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Gasc´on-Pinedo, J. A., & Delgado Kloos, C. (2016). Scaling to Massiveness with ANALYSE: A Learning Analytics Tool for Open edX.
IEEE Transactions on Human-Machine Systems(Ruip´erez-Valiente et al., 2016). Impact factor as of 2015: 1.8, JCR(5/22), Q1, category: computer science, cybernetics.
6. Mu˜noz-Merino, P. J., Ruip´erez-Valiente, J. A., Delgado Kloos, C., Auger, M. A., Briz, S., Castro, V. De, & Santalla, S. N. (2016). Flipping the Classroom to Improve Learning with MOOCs Technology. Computer Applications in Engineering Education, 25(1), 15–25 (Mu˜noz-Merino et al.,2016). Impact factor as of 2015: 0.935, JCR(45/85), Q3, category: engineering, multidisciplinary.
7. Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Pijeira D´ıaz, H. J., Santofimia Ruiz, J., & Delgado Kloos, C. (2017). Evaluation of a Learning Analytics Application for Open edX Platform. Computer Science and Information Systems, 14(1), 51–73 (Ruip´erez-Valiente et al.,2017). Impact factor as of 2015: 0.623, JCR(83/106), Q4, category: computer sci- ence, software engineering.
8. Alexandron, G., Ruip´erez-Valiente, J. A., Chen, Z., Pedro J. Mu˜noz-Merino, & Pritchard, D. E. (2017). Copying@Scale: Using Harvesting Accounts for Collecting Correct Answers in a MOOC. Computers & Education, 108, 96–114 (Alexandron et al., 2017). Impact factor as of 2015: 2.881, JCR(14/104), Q1, category: computer science, interdisciplinary applications.
9. Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., & Delgado Kloos, C. (2017). Detect- ing and Clustering Students by their Gamification Behavior with Badges: A Case Study in Engineering Education (In press). The International Journal of Engineering Educa- tion, (Engineering Behind Technology-Based Educational Innovations) (Ruip´erez-Valiente et al., 2017). Impact factor as of 2015: 0.559, JCR(61/85), Q3, category: engineering, multidisciplinary.
10. Mu˜noz-Merino, P. J., M´endez Rodr´ıguez, E., Delgado Kloos, C., & Ruip´erez-Valiente, J. A. (2017). Design, Implementation and Evaluation of SPOCs at the Universidad Carlos III de Madrid. Journal of Universal Computer Science, 23(2), 167–186 (Mu˜noz-Merino et al.,
2017). Impact factor as of 2015: 0.546, JCR(89/106), Q4, category: computer science, software engineering.