8. Management Education and Development Competitive
Using Learning Analytics to Inform Interventions for ‘At Risk’ Online Students
Ms Sue Whale
UNE Business School, University of New England, Armidale, Australia Email: [email protected]
Associate Professor Josie Fisher
UNE Business School, University of New England, Armidale, Australia Email: [email protected]
Dr Fredy-Roberto Valenzuela
UNE Business School, University of New England, Armidale, Australia Email: [email protected]
8. Management Education and Development Competitive
Using Learning Analytics to Inform Interventions for ‘At Risk’ Online Students*
ABSTRACT:
This paper describes a project designed to explore the use of learning analytics in real time to enhance students’ learning experience. The approach was designed to be implemented by individual subject coordinators during the teaching period using student data which is readily available from the learning management system. This project demonstrates how a simple approach to tracking students’ activities and timely implementation of interventions for ‘at risk’ students can encourage engagement and result in an enhanced online learning experience.
Keywords:
e-learning, business education, business schoolsINTRODUCTION
This paper describes preliminary findings of a larger project funded by the Office for Teaching and Learning. The research question we sought to answer was whether learning analytics could provide teaching staff with key points in time and criteria to identify ‘at risk’ students who could be individually contacted during the teaching period thereby increasing their level of engagement and satisfaction.
Learning analytics focuses on the learning process including the relationship between the learner, content, institution and educator (see, for example, Ferguson & Buckingham Shum 2012; Siemens & Long 2011).
This paper describes the development and implementation of an approach to support students in online learning utilising the learning analytics available in the learning management system (LMS).
METHODS
In designing the project we utilised an approach identified by Siemens (2011) involving the use of analytics to make predictions based on historical student activity. We drew on these predictions and undertook a process of adaptation, personalisation and intervention to enhance students’ learning experience. This approach is consistent with Clow’s (2012, p. 1) “Learning Analytics Cycle” which
identifies a four step cycle: identifying learners; generating data; producing metrics, analytics or visualisations; and using the outcomes to inform interventions thereby closing the loop and beginning a new cycle.
The first task involved an analysis of student behavior and development of assumptions about how these behaviours might impact student success. Tracking data from the LMS was then extracted to identify students’ access and participation in terms of the behaviours identified. A series of personalised
interventions were developed and implemented by subject coordinators. Students identified as being ‘at risk’ were individually contacted and offered additional assistance. Students’ responses to this approach were positive in both undergraduate and postgraduate subjects. Student interaction with the learning materials and activities increased and there was an increase in student satisfaction.
BRIEF LITERATURE OVERVIEW
Openness and accessibility of distance or online education are all too often associated with significantly lower rates of successful unit and course completion than campus-based institutions (Powell et al. 1990).
As Staley and Trinkle (2011, p.10) point out “[s]tudent success does not arise by chance. It is the result of an intentional, structured, and proactive set of strategies that are coherent and systematic in nature and carefully aligned to the same goal”.
With increased enrolment numbers and study options the diversity of the student population has also increased. There is strong and consistent evidence that students who successfully pass a subject are more likely to re-enroll and undertake further study than those who fail. There is also evidence of a correlation between learners’ engagement in effective learning practices and their academic success (Campbell et al.
2010). Research by Shin (2012) and Morris and Zuluaga (2003) indicates that the perception distance students have of psychological presence of teaching staff, peers and the institution in which they are
enrolled has a significant impact on student learning, satisfaction and motivation. While Richardson (2003) cites claims that online learning is not as effective as traditional classroom learning due to lack of interaction and feelings of disconnection from others, increasing social presence and interaction in online learning can be facilitated and lead to positive experiences for students (Valenzuela, Fisher & Whale 2013, p. 387).
Formal systems of ‘student coaching’ are becoming more prevalent both nationally and internationally, and are proving to be very effective mechanisms for engaging and retaining students. Research by Bettinger and Baker (2011) in the US found that coaching resulted in a 15% increase in retention. An Australian study at Queensland University of Technology demonstrated similar results (Nelson et al., 2009).
In contrast to such institution-based activities, our project explored the use of simple, real time learning analytics by individual subject coordinators as part of their learning and teaching activities in order to inform interventions which were implemented within the teaching period. The impact of these interventions on student engagement and satisfaction across three online subjects offered by the University of New England (UNE) Business School was explored.
INTERVENTIONS AND DISCUSSION
Online learning is increasing in higher education, offering benefits to students including flexibility of time management and access, however, unlike face to face delivery, where student presence, interaction and progress can be monitored informally by teaching staff by simply surveying the room, online learning means student behaviours are often invisible. Results of our previous research (Fisher, Valenzuela &
Whale 2012) were used to establish patterns of behaviour and resulting success and satisfaction of students. This baseline data and analysis in subjects offered by the UNE Business School indicated that a
significant proportion of students exhibiting the following behaviours are likely to be less successful in their online studies:
• Limited access to the learning management system (LMS) and learning materials in the first few weeks of the teaching period
• Poor results in early assessment tasks
• Limited access to materials provided for major assessment tasks
• Inconsistent access to the LMS across the teaching period
While identifying these behaviours as indicators of less successful outcomes is not new, the contribution of this project is that these ‘triggers’ were used to inform interventions which were implemented by teaching staff and evaluated within the teaching period, thus providing an opportunity to explore their potential impact on students’ learning experience.
As the student cohorts in this project closely resembled those in the previous study, the first three behaviours were used as indicators that students may benefit from prompts or interactions which encouraged students to engage with the study materials earlier rather than later and additional assistance was offered to support these students in their online learning. The timing of planned interventions and the opportunities for students to change behaviours and achieve success were considered. Early assessment tasks were therefore targeted while late ones were not. A personalised approach was adopted as it was considered to be more likely to encourage contact from students and increase opportunities for engaging them in their learning.
Each of the subjects involved in this project (one undergraduate and two postgraduate) was delivered fully online. Limited access to the LMS in the first two weeks was therefore considered to be a strong indicator that students had not yet engaged with the learning materials and was the first behavior targeted
with an intervention. Those students who had not yet engaged were contacted by phone as first preference and via an email from the subject coordinator if phone contact was not made.
A total of 43 students were identified as meeting this first criterion – limited access to the LMS and learning materials in the first 2-3 weeks of the teaching period. These students were contacted personally through phone calls to remind them that the teaching period had commenced and to determine if there were any access, technical or other issues that teaching staff could assist with. The questions asked of these students included:
- Have you looked at unit materials?
- Have you had any problems accessing or working your way through the materials?
- Do you have the textbook for the unit yet?
- If they haven't accessed the site and don't think they will continue with the unit, why are they thinking of
withdrawing?
- If they plan to withdraw, can we do anything to assist with getting started and continuing?
Personalised emails were sent from the unit coordinators to the 26 students who could not be contacted by phone. Part of the email sent to students read as follows:
‘Dear (student’s first name),
I just want to check that everything is proceeding well for you in Trimester 1. If you have any concerns about the unit content, obtaining the textbook or Moodle please let me know via return email.
Regards,
(first name of subject coordinator)’
Email responses from students were followed up within 24 hours. Students contacted by phone and/or email were grateful for the reminder to access the LMS and appreciative of the personal contact received, as evidenced by the following comments:
‘thank you very much for your email…thank you for your support’
‘thank you very much for getting my focus back’
‘sincere thanks for your email – I appreciate your interest’ (Student Comments, 2013).
Students readily discussed their situations and following this interaction the majority (93%) of students promptly accessed the LMS multiple times.
The second behavior targeted was poor performance or non-participation in early assessment tasks. While this was not possible for every unit due to the timing of assessments, students who received non-passing grades or who did not participate in early assessments were contacted. The subjects of study included in this phase of the project had assessment items that were low in weightings, so it was still possible for students to successfully complete the subject. Students who met this criterion were each contacted by the subject coordinator. This intervention consisted of an email offering additional support, further
explanation or, in some, cases a discussion of personal circumstances. The email sent to these students read as follows:
‘Dear XXX,
The Online Test marked the ‘official’ end of Module One. The content, however, will be applied throughout Module Two, so an understanding of the main theories and concepts introduced in Topics 1.1 – 1.5 will be required. If you did not do as well as you had hoped in the Online Test, I suggest you carefully review the questions you answered wrongly. If you have difficulty in understanding where you went wrong, or you have any other questions, please contact me.
Regards,
(first name of subject coordinator)’
As already mentioned, this intervention was not enacted for every assessment task – it focused on early tasks and those which were not weighted heavily. Since performance in these early assessment tasks provides a strong indicator of success in the remainder of the unit, ensuring students engaged with the material and understood the key concepts was vital. There were two points of contact with students for this intervention associated with two different assessment tasks. In the first case 22 students were
contacted. Of these, 20 successfully continued in the unit of study to completion. The second point of contact was directed to 18 students, of whom 16 successfully completed the unit of study.
The third focus was on access to materials relating to major assessments. Details of assessment tasks are provided in a specific location in the LMS, and students who had not accessed this information two-three weeks prior to the due date for the assignment were considered to be at risk of not completing the task satisfactorily. Students were contacted by email if they fell into this category. An example of the email sent to these students can be seen below.
‘Dear XXX,
I just want to remind you that assessment one has to be submitted on 26th March. Attached you will find a document with tips regarding that assessment. Please get in contact with me if you have any question regarding that
assessment.
Regards,
(first name of unit coordinator)’
The result of this contact was that seven of the 14 students contacted promptly engaged with the materials and commenced preparation for the upcoming task. However, nine of the students contacted did not subsequently complete the task (withdrawing from the unit instead). The five students who did complete the assessment achieved an average grade of 79%. Typical responses from students were:
‘Thanks for reminding me about the assessment’
‘The tips were very useful’
That nine of the 14 students contacted failed to complete the task was an unexpected outcome and we intend to follow up with these students to try to identify their reasons for withdrawing. It may be relevant that this intervention related to a first year undergraduate subject which typically has higher non-
completion rates than postgraduate subjects.
All students enrolled in the subjects that were part of this project were surveyed to identify whether those students who were the recipients of interventions rated their overall learning experience higher than students who did not receive interventions. Results showed a significant difference (p<0.05) between these student groups. The mean for students who were targeted for one or more intervention was 4.3 (out of 5) and 4.0 for students who were not targeted for an intervention.
CONCLUSION
This project demonstrates a simple approach to the use of learning analytics by teaching staff aimed at enhancing online students’ engagement and satisfaction. It demonstrates that tracking students’
activities and the timely implementation of interventions has the potential to influence students’
behaviours and enhance their online learning experience. Students were contacted personally, rather than by an automated means and students’ unsolicited responses to the interventions revealed that this approach was appreciated.
The next phase of this project is to further analyse the survey results and the in-depth interviews which are currently taking place to better understand students’ views related to the value of the interventions. The project will then be reviewed, revised and expanded to a larger number of units to enable an evaluation of the impact of interventions on students’ online experiences and outcomes across a broader sample.
While this project demonstrated that students’ learning experiences were enhanced by the
interventions, this added significantly to the workload involved in delivering the subjects compared to previous offerings of the subjects. Unless the larger project confirms the value of this
personalised approach to student support, and resources can be allocated to this activity, it would
not be sustainable on a larger scale.
*Support for this project has been provided by the Australian Government Office for Learning and Teaching. The views in this project do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.
References
Bettinger, EP., & Baker, R. (2011). The effects of student coaching in College: An evaluation of a randomized experiment in student mentoring. Stanford University School of Education, March 2011.
Retrieved 25 June, 2013 from https://ed.stanford.edu/sites/default/files/bettinger_baker_030711.pdf Campbell, J. P., DeBlois, P.B., & Oblinger, D. (2010). Academic analytics: A new tool for a new era.
Educause Review, 42(4), 40-57.
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. Retrieved from http://dougclow.wordpress.com/2011/02/28/the-learning-analytics-cycle/
Ferguson, R., & Buckingham Shum, S. (2012). Social learning analytics: Five approaches. In
Proceedings of the 2nd International Conference on Learning Analytics & Knowledge (LAK12).
Vancouver, BC, Canada.
Fisher, J., Valenzuela, F., & Whale, S. (2012) ‘Enhancing distance education student outcomes utilising learning analytics: A case study’, Australian New Zealand Academy of Management 26th Annual Conference, Perth, 5-7 December, 2012, pp. 1-16.
Morris, E.J.S., & Zuluaga, C.P. (2003). Educational effectiveness of 100% online I.T. courses. In G.
Crisp, D. Thiele, I. Scholten, S. Barker and J. Baron (Eds.), Interact, integrate, impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. Adelaide.
Nelson, K., Duncan, M., & Clarke, J. (2009). Student success: The identification and support of first year students at risk of attrition. Studies in Learning, Evaluation Innovation and Development, 6(1), 1-15.
Powell, R., Conway, C., & Ross, L. (1990). Effects of student predisposing characteristics on student success. The Journal of Distance Education, 5(1) 5 - 19.
Richardson, J.C. (2003). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7(1), 68-88.
Shin, N. (2012). Transactional presence as a critical predictor of success in distance learning. Distance Education, 24(1) 69-86.
Siemens, G. (2011). What are learning analytics? Elearnspace. Retrieved from:
http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/
Siemens, G., & Long, P. (2011). Penetrating the fog. Analytics in learning and education. Educause Review, 46(5). Retrieved from
http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume46/Pene tratingtheFogAnalyticsinLe/235017
Staley, D.J., & Trinkle, D.A. (2011). The changing landscape of higher education. Educause Review, 46(1), January/February 2011. Retrieved from http://www.educause.edu/ero/article/changing- landscape-higher-education
Valenzuela, F., Fisher, J., & Whale, S. (2013). Lecturers’ social presence and personality in the online environment: The perceptions of off-campus postgraduate and on and off-campus undergraduate management students. In B. Tynan, J. Willems & R. James (Eds.), Outlooks and opportunities in blended and distance learning (pp. 383-402). Hershey, PA: Information Science Reference.
