Learning analytics and study behaviour: A pilot study

Proceedings ascilite 2011 Hobart: Full Paper

₉₉₇

Learning analytics and study behaviour: A pilot study

Rob Phillips, Dorit Maor, Wendy Cumming-Potvin, Pauline Roberts, Jan Herrington,

Murdoch University

Greg Preston,

The University of Newcastle

Elizabeth Moore, Laura Perry

Murdoch University

The analysis of student access to learning management systems and web-based lecture capture systems is a growing area of interest for teachers in higher education wishing to improve the student learning experience. The data trails left by students as they engage in these environments can be accessed and analysed for meaning. This paper describes a study conducted as part of a wider multi-university study into student study behaviour. It offers a detailed snapshot of four students whose access to Lectopia recordings were tracked and analysed, and who were

subsequently interviewed to confirm or disconfirm assumptions made about their study methods from the analysis. The data revealed that a surface analysis using learning analytics was largely insufficient to determine student study characteristics, but qualitative data provided rich information to supplement the analysis. Suggestions are made for further research into how this emerging methodology can be further developed and strengthened.

Keywords: learning analytics, academic analytics, student engagement, technology-enhanced learning, lecture-capture systems, study patterns, e-learning

Background/ Intro

Learning technologies are now pervasive in universities in the developed world, but we have little evidence of their effectiveness in improving learning outcomes. In many Australian universities, a blended approach is taken, providing a mix of online and face-to-face study opportunities (Littlejohn & Pegler, 2007). In some Australian universities, students study externally, by distance education, and learning technologies enable remote learners to communicate with each other and their teachers. The most common learning technologies in use in Australia are Learning Management Systems (LMSs such as Blackboard or Moodle) and web-based lecture-capture technologies (such as Lectopia).

Proceedings ascilite 2011 Hobart: Full Paper

₉₉₈

Much e-learning research over the years has been based on quantitative data largely derived from the

perceptions of students, which may not provide accurate indicators of learning. Such studies rarely indicate the causality of effects. On the other hand, qualitative approaches rely on descriptive data and focus on individual contexts. Emerging research, such as the study described in this paper, seeks to combine data from technology usage logs with descriptive methods, to develop a richer understanding of how students engage with e-learning environments.

E-learning environments automatically capture system-based records of users‘ activities, recording who accessed what, and when. Use of this data is termed usage logs, audit trails, learning analytics or academic analytics (Goldstein & Katz, 2005; Kennedy & Judd, 2004; Oblinger & Campbell, 2007). Teachers can use this data to reconstruct an individual student‘s online presence in great detail. However, the meaning of the masses of data that can now be collected is not always clear. Usage log data can be used to track how students use lecture recording systems (this work) and web-based learning management systems (LMS) (See, for example, Dawson, McWilliam, & Tan, 2008; Phillips, 2006; Phillips, Baudains, & van Keulen, 2002). Usage data can also be used to analyse how social networks form during online discussion forums, e.g. Dawson‘s SNAPP tool (Dawson, Bakharia, & Heathcote, 2010).

However, usage logs simply record users‘ behaviour in an e-learning environment, but they do not explain why that behaviour occurs. Kennedy and Judd (2004, p. 19) explain that, ―at their most basic level audit trails measure the behavioural responses and activities of users‖, without explaining why they do what they do. So, while usage logs of learners‘ activities in e-learning environments are increasingly used, and data is often relatively simple to generate, care should be taken in analysing and interpreting this data. This paper contributes to knowledge in this area.

Previous work

The impact of web-based lecture-capture technology on the teaching and learning enterprise has been a popular subject of research in recent years. A large study across four universities (Gosper, et al., 2008) is just one of many recent studies about technologies such as Lectopia (See Taplin, Low, & Brown, 2011 for a recent review). The overwhelming finding of this research into lecture-capture technology is that students find these tools beneficial in terms of both flexibility and assisting their study schedules. However, teaching staff have viewed this technology negatively in some cases, because of falling attendance at face-to-face classes.

A recognised shortcoming of much of this research is that it has focussed on the technology per se, rather than the learning environment as a whole (Gosper, et al., 2008). This was the impetus for our current work, which holistically examines a unit of study, and uses learning analytics to gain a richer understanding of what students actually do in a technology-enhanced learning environment.

In this research we are primarily interested in learning processes, rather than learning outcomes. That is, the way that students interact with the learning environment and the learning tasks which are embedded in that

environment. We are interested in the learning activities that students undertake as they engage with learning tasks: what the learner actually does, whether intended or not. This includes interaction with the learning environment; engagement with designed learning tasks; how this engagement occurs (e.g., individually, in groups, as directed by the teacher); and self-directed review and reflection activities. We contrast these studying

learning processes with cognitive (or internal) learning processes: ―psychological processes which lead to greater competence or understanding‖ (Goodyear & Retalis, 2010). Goodyear and Retalis proposed that learning processes are ―tightly bound up with‖ (p. 12) – but are not the same as – studying activity, which is our focus in this work.

Lectopia usage patterns

We have previously reported (Phillips, et al., 2010) on the development of a learning analytic tool which ‗mines‘ data recorded by the Lectopia lecture recording technology and aggregates this data in a week-by-week manner. Patterns of use are then displayed graphically for a whole class or individual students. Our initial work (Phillips, et al., 2010) identified eight main conceptual usage behaviours that distinguish between student activities in Lectopia. Conscientious students access Lectopia regularly. Good-intentioned and Repentant

students have some weeks of regular use, at either the beginning or end of the teaching period. Other students access recordings only in dedicated blocks – they are Bingers. Crammers leave their engagement with

Proceedings ascilite 2011 Hobart: Full Paper

₉₉₉

recordings until just before the examination period. Other students may access recordings once (One-hit wonders) or not at all (Disengaged), or their pattern of use may not fit any of the other categories (Random users).

These categories provide indicators of behaviour, but they do not explain that behaviour. For example, bingeing students could be very effective in practice, balancing their study, work and family commitments, and studying when they find time. On the other hand, a bingeing student could be falling behind in their work because of poor time management and prioritisation skills, and their efforts could be ineffective. A simple of review of the numbers does not provide this depth of understanding.

Our continuing work has applied this analytic tool to pilot studies of student behaviour in three units across two universities. This paper reports on one of those pilots, drawing on usage data to identify and interview students with diverse patterns of behaviour, in order to validate the use of this tool.

Method

Our work aims to investigate how students engage with, and study in, e-learning environments. We address this in part in this paper, but we are more interested in validating the Lectopia learning analytic tool we have developed, and its use in a suite of methods. We have two specific research questions: ‗How useful is the Lectopia analysis tool in identifying student behaviour patterns?‘ and ‗What can this pilot study tell us about student study behaviour in the context of this investigation?‘

The design of the study was informed by a pragmatic, mixed-methods paradigm of inquiry, using a modified design-based research approach, which has emerged in recent years as a suitable approach to educational research, in particular e-learning research (Herrington, Reeves, & Oliver, 2010; Phillips, McNaught, &

Kennedy, 2011). While a design-based research approach often starts with the initial design of an innovation, in this case we take a reconstructive approach (van den Akker, 1999), using existing information to derive the baseline for the study. Design-based research also has an explicit interest in theory development, albeit as relatively humble ‗design principles‘.

We accessed a range of sources of data to address our research questions: the Lectopia analysis tool described above; the Social Network Analysis tools developed by Dawson et al. (2010) to analyse student behaviour in discussion forums; standard usage reports from the LMS; assessment results; interviews with unit coordinators; and semi-structured interviews with a sample of students.

An essential element of the research design hinges on the ability to identify students so that various sources of data can be cross-referenced (e.g., learning analytics and grades) to their interview responses. Ethics approval was received for this approach, and students were informed about the research, and the possibility of being interviewed, at the start of semester. Interviews were scheduled after all unit activities and assessment had ceased, that is, after the end of semester.

The baseline criterion for categorising students (Phillips, et al., 2010) was to distinguish between behaviour on two dimensions:

attendance and non-attendance at lectures use and non-use of Lectopia

Data for the former were taken from attendance record sheets filled in at each lecture. However, as will be seen from the interview data, not all students did this. Data on Lectopia usage was obtained from the Lectopia analysis tool.

Towards the end of the semester, we began to collate class attendance data and Lectopia usage patterns. We used this to create a shortlist of students with different behaviour categories whom we would approach as possible interviewees. It proved problematic to get students to agree to be interviewed, because the timing of the interviews in the break between semesters meant that many students were unavailable. Other students with high Lectopia use were enrolled externally, and their geographic distance meant travel to the interview location was difficult. While we approached ‗backup‘ students in each category, we were unable to interview as wide a cross-section of behaviours as initially planned. Our stratified sample became, instead, a convenience sample of those who were available for interview within the available timeframe, which impinged on the variety of behaviour categories available.

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₀

The semi-structured interviews sought some background information and questioned students about general study habits, before discussing the learning analytic data relating to lecture attendance, Lectopia hits and LMS sessions. We trialed the interview approach with three students. This trial included video-recording the

interview so that gesturing around the usage charts could be captured. The visual aspects of the video recording did not yield a great deal of useful information, so subsequent interviews were simply voice-recorded. The interview concluded with questions related to students‘ perceptions of performance overall in the unit in terms of final result, and whether this meant their chosen strategy was one that they considered worked for them in that unit of study.

The interviews were transcribed and analysed using a constant comparative method (Glaser & Strauss, 1967; Merriam, 1998). Each interview was initially coded, by two researchers according to temporary themes

emerging from the data. Successive interviews were constantly compared to these themes, and within and across categories to find meaningful patterns in the data. Refined themes were then finalised to provide meaning, and to describe and explain phenomena within the data. Data were also broadly considered within the three-stage framework suggested by Miles and Huberman (1994): data reduction, data display and conclusion drawing and verification. The qualitative interview analysis was then combined with the quantitative usage data and assessment results to develop a richer understanding of each case. Finally, one of the unit coordinators was interviewed to comment on any contextual issues that might have influenced the analysis.

The pilot study

The semester unit of study (the focus of this paper) was a third year unit on the sociology of education, offered at a metropolitan Western Australian university. The unit had three cohorts: ~150 internal students at the main campus; ~50 internal students at a regional campus; and ~100 external (distance education) students. This particular unit was chosen because it appeared to be well-designed, had a clear and comprehensive study guide and made use of both LMS and Lectopia technologies. Further, the unit coordinators at both campuses were interested in the research, in order to better understand the impact of their teaching.

The teaching activities each week comprised a one hour lecture (recorded through Lectopia) and a two hour workshop for internal students. External students accessed the Lectopia recordings, and carried out interactive activities through discussion forums in the LMS (Blackboard Campus Edition 8). Online participation was assessed for external students but not for internal students. The weekly topics related to issues that impact on education contexts within Australia, supported by numerous readings, which students were expected to read in preparation for workshop or online discussions.

The unit assessment was outcomes-based and relatively innovative in that it aimed to position pre-service teachers as ‗practical theorists‘ (See Bell & Patterson, 1998), so they can act as informed and responsible teaching professionals. Assignments were scaffolded, research-focused and promoted critical reflection on practice during two in-school practicums (pracs) during the semester. The assessment consisted of a short research proposal (in Week 3); a critical autobiography of pre-service teachers‘ prior educational experiences; and a 2000 word research paper. This final report encouraged students to reflect on the values that they hold and how these may conflict with the views of other stakeholders in the educational system. A final exam completed the assessment requirements.

Results

Lectopia behaviour patterns

The PHP-based Lectopia Usage tool described in Phillips et al. (2010) was used to generate data for the unit studied, by performing database queries on the log data recorded by Lectopia. This data was downloaded and imported into Excel for further manipulation through various formulae and pivot tables. A series of macros were developed to automate the generation of this data and subsequent graphs. The pivot tables allow graphs to be generated for the entire class of students, and for individuals. The Excel macros also provide alternative ways of drilling down into the data, so that the researchers could see at a glance the nature of each ‗hit‘ (or access to a Lectopia event):

the format of each hit (download/stream/MP3/MP4, etc.)

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₁

the type of each hit (whether it is an initial hit or a repeat hit on a recording)

the number of the lecture that is accessed in which week

Figure 1 shows one representation of the overall data for all 109 students in the pilot unit who accessed Lectopia. This shows the total number of hits on the Lectopia system (y-axis) against the week of the semester (x-axis). The semester is structured as 10 weeks of formal teaching (Weeks 1-3, 6-9 & 12-14), with four non-teaching weeks (shown on Fig. 1) used for non-teaching pracs. The study and exam period extended from weeks 15-17. Figure 1 also indicates assessment dates. In addition, the time between the lecture recording and the time recordings were accesssed is shown as an extra dimension in Figure 1, with three intervals: on the day of the lecture, during the first week, and after the first week.

Figure 1 shows varying patterns of use across the semester. Lectopia use was relatively high in Weeks 2 and 6, prior to submission of Assignments 1 and 2, respectively. Overall, Lectopia use was quite low during the first non-teaching period. On the other hand, Lectopia use was relatively high during the second non-teaching period, and also in Week 14 – the week for submitting the final assignment. By far the highest use was during the week of the examination. The delay before listening to a lecture recording is also shown in Figure 1. In the first three weeks of semester, the majority of access was in the first week of ‗publication‘. As the semester progressed, approximately 50% of recordings were accessed more than one week after the lecture was recorded.

While we could speculate about some of these patterns, it would not necessarily be productive, because at this stage we do not have enough evidence to support our speculations. Instead, as described above, we selected a sample of students for interview, to probe their behaviour more deeply.

Table 1 summarises the characteristics of students who consented to be interviewed, in terms of their enrolment type, their self-reported attendance at lectures, their observed hits on the Lectopia system, the behaviour category we assigned to them based on their pattern of Lectopia hits, and their final mark in the unit. Only one student (D) self-reported regular attendance at lectures according to the attendance sheets. Lectopia use ranged from none (E) to substantial, with three students (A, B, F) recording more hits on Lectopia than the ten available lecture recordings.

Manual observation of LMS usage reports indicated that all six interviewees accessed the LMS to some extent, largely to download unit materials. The six students also largely ‗lurked‘ in the discussion forums, and tended not to post messages (Dennen, 2008). The SNAPP tool therefore provided little useful information. This was arguably predictable, because the unit design did not require internal students to contribute to forums, and all of our sample students were enrolled internally.

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₂

Student A was classified as Conscientious. As shown in Figure 2, she accessed lecture recordings on 8

occasions, during most of the teaching weeks. Figure 2 also indicates that Student A reviewed lecture recordings a second time three times. Students B and F were categorised as Crammers, because they accessed all the recordings only in Week 17, with no access prior to that. Students C and D were categorised as Random, because they only accessed Lectopia on a few, dispersed occasions. These two students were the only ones who indicated they had attended on-campus lectures. Student E was classified as Disengaged, because she did not access Lectopia, and no lecture attendance was recorded.

Table 1. Summary of the six students selected for interview

Student Background Attendance Lectopia Hits Lectopia Category Mark

A Grad Dip. Mature-aged - 11 Conscientious 88

B B.Ed. regional campus - 17 Crammer 61

C B. Ed. main campus 1 3 Random 70

D B. Ed. main campus 8 5 Random 64

E B. Ed. main campus - - Disengaged 51

F B. Ed. main campus - 15 Crammer 66

Interview data

The six students were interviewed for 45-60 minutes. During the interview, each student was presented with data on their own personal usage patterns, and advised of their predicted pattern descriptor (see Table 1). Each student was able to comment on whether this was an apt description of their study pattern, and could refute the label. Here, we present the data for four students with four distinct Lectopia usage patterns (Students A, B, C and E). Students D and F are not considered here for space reasons, and because their behaviours were similar to students C and B, respectively.

Student A

Student A was a mature-aged student with school-aged children. She attended the first lecture, but subsequently used Lectopia while working from home. She did this because this was the only class she had that day, and she could save an hour‘s travel time, but also because she was highly motivated, and felt able to learn

independently: ―I‘m more of an independent learner than a group learner‖.

Student A agreed with her characterisation as Conscientious. She adopted a routine of studying every day from 9am-1pm, while her children were at school.

This is my study time, what I do each week, listen to lectures, do the notes, do the readings – try to keep up to date if not a week ahead with the readings. It‘s called being a mature-aged student who likes to study, helps me to keep up with my study.

Her study patterns included regular LMS access to download required and optional readings. Over time, she stopped accessing the optional readings because she didn‘t think these were relevant. As an individual learner, Student A logged on to the discussion board ―just to see if anything important‖ was there, but did not contribute to, or actively participate in, the discussion.

Student A is an experienced learner who was highly engaged with the unit and its materials, despite not Fig. 2. Repeat or initial hits by week for Student A

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₃

attending lectures and not contributing to discussion forums. This student appreciated the flexibility offered by Lectopia, and used her self-efficacy skills to achieve a final grade of High Distinction.

Student B

Student B was also a full-time, mature-aged student with family responsibilities, enrolled at the regional campus and working part time as an Education Assistant in a school. She was categorised as a Crammer, but disputed this, claiming that she had attended all lectures and tutorials (her attendance was not recorded). Her reported behaviour was systematic and conscientious. She followed the study guide and allocated 1.5 hours early in the morning to do the readings.

Student B used the LMS to download unit materials, but did not find the discussion forum helpful. She admitted struggling with understanding the unit material because of family and time pressures, and she only just passed the two major assignments. Because she was worried about failing, she put in a special effort to revise for the exam, and achieved a distinction in this, for an overall Credit grade.

This student strategically used Lectopia as a revision tool. She downloaded all the lectures in Week 17, for revision purposes:

I had the lecture notes and grabbed a red pen and took it differently, but by listening to one after the other, I think it helped a lot.

An advantage that Student B found, as a regional student, was that the lecture recordings were of the main campus lecturer. This assisted her revision because the lectures were presented in quite different ways.

…it was good though because I had one lecturer – she‘s very good but then when I listened to it, it was another lecturer so it was similar content but in a different way.

Student B found that not only did her revision with Lectopia help her examination result, it also assisted her to consolidate the learning outcomes intended in the assignments. She reflected:

It was hard for me to believe schools weren‘t doing their best. I ... couldn‘t believe there was more to it. It took a while for me to understand that and be critical of things. ... I guess I thought schools were doing everything right [but] this was all about investigation and what schools could do better. I think they are doing their best but this unit says that they are not.

Student C

Student C was a full-time internal student at the main campus. She was categorised as a Random student because she had infrequent access to Lectopia and incomplete attendance data. However, she claimed that she was a Conscientious student because she attended the majority of the lectures. She used Lectopia to listen to and take notes from recordings of lectures that she had missed. Student C systematically used the Unit Guide, the textbook and the LMS to guide her learning. She would read the weekly preamble to check what to cover, and do the readings prior to the lecture. She appreciated the amount of detail on the LMS compared to other units of study.

The structure [of this unit] with the readings online, the external links, the questions – just everything was in the one spot and that was awesome.

Student C accessed optional materials as needed, within workload constraints.

If they were hard concepts or I felt like there were lots of gaps ... I‘d be more likely to go to the online readings but then as the semester goes on and stuff piles up, I just didn‘t.

She also regularly read the discussion forum, but only posted twice, because she thought, in general, that the students who posted the questions could have answered them themselves by reading unit materials. Unlike Students A and B, Student C enjoyed interactions with her peers to enable her to ―bounce ideas‖, but also was able to work independently. Student C demonstrated a high level of commitment and motivation ―I take the reader with me and study anytime.‖ Her learning habits include devoting a block of time to learning, but more often she ―fitted 5-10 minutes every now and then.‖ This commitment led to a High Distinction grade in the research paper and a Distinction result overall.

Student E

Student E was an internal full-time 2nd year student on the main campus. She was categorised as Disengaged

because there was no evidence of class attendance, and she did not access Lectopia, but she claimed to attend most of the lectures. Student E engaged with unit materials through the LMS and was a prolific internet user: ―I

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₄

pretty much use the internet for everything apart from the books we have to have‖. She downloaded lecture notes and annotated them in lectures. She also read over 130 forum posts and contributed to forums three times. This student purposely chose not to use Lectopia, partly because ―I went to most of [the lectures]‖, and partly because of an awareness of her own learning challenges in previous units:

I have it downloaded and don‘t listen to it, or listen to it for 5 minutes and then phase out. That‘s why I go [to lectures] because I‘ve tried listening to them and I tend not to, so I try to get there as much as I can. (E)

Student E appeared to be relatively disengaged with the unit. As a second year student in a unit mostly ‗populated‘ by third year students, she felt isolated and used the LMS to compensate:

Most of my friends are not doing [the unit]. That‘s why I went to LMS on this because it‘s my only class I‘ve got with these people.

She noted that discussion forums gave her a sense of how others were tackling their study, and found that often others online had the information she needed so she did not need to ask the teacher:

[Discussion forums] were one of the best bits when I first started. Especially when you are on your own you are not sure you are doing the exact right thing, and to have someone to bounce off or to check your opinion and they come back saying either no or yes. So it‘s online so you can do it anytime. Like if you are working on your assignment at 3am or something, you can write it up there and someone will answer eventually.

While she might have been classified as Disengaged with Lectopia, but did attend lectures, her reported behaviour was less engaged than other students interviewed, and this showed in her final grade – a bare Pass. She self-reported problems with self-efficacy, and was still grappling with the discipline needed for successful study:

When it comes to assignment time ... I start to get lazy, leave it for a week or two and have to sit down for longer. ... No, I got worse [grades] than I thought. Towards the end of the semester I got really lazy and in the last assignment I did not use LMS as much as I should have. Usually when I‘ve finished an assignment I go on to LMS and check the marking guide and the criteria for it ... [It‘s] frustrating but it‘s because I got lazy and didn‘t go through and check it again.

Discussion

The previous section makes it clear that the four students exhibited very different study behaviours, with varying degrees of success. Student A made a conscious decision that she could study effectively without face-to-face contact, and she used Lectopia and other unit resources to facilitate this, with great success. Student B attended face-to-face classes, but was struggling with the unit content because she was over-stretched in terms of her own time. She used Lectopia to recover from poor assessment results during the semester, and, in the process, came to some core understandings that the unit set out to facilitate. Student C attended most face-to-face classes and used Lectopia only to catch up on classes she missed. Her approach to blended learning made extensive use of LMS-based learning resources, diligent self-study and peer interactions to become a successful independent learner. Student E attended most face-to-face classes but did not use Lectopia because she found it challenging to concentrate on recorded audio. Instead, she made extensive use of the LMS and relatively high use of the discussion forum to try to engage with a unit that few of her friends were enrolled in. She found it difficult to engage in the unit and reported problems in self-regulation, which led to a low pass mark.

The results provide solid evidence towards the second research question. The four students displayed different approaches to the use of technology to assist their study in this unit. All four students reported here (and another two in this case study, and nine more in two further case studies) were endeavouring in different ways to engage with their studies, using technology and unit resources in various ways. Some students were more successful than others at this, but we do not try to attribute a particular technology use to this success or lack thereof. Instead, the four cases reported here start to illustrate some of the complexity of the modern, technology-enhanced learning environment.

This complexity points to the need for an ongoing program of study into this area. At the same time, however, this pilot study has validated both our learning analytic tool (first research question) and our mixed methods approach, combining direct observations of technology-usage behaviour with student perceptions (through interviews) to gain a deeper understanding of student study behaviour.

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₅

Shortcomings

Despite the promise of this approach, this pilot study has also identified a number of shortcomings in the research design, which we intend to address in subsequent studies. Table 2 lists a number of limitations in the research design, together with suggested improvements.

Table 2. Suggested improvements to the research design

Shortcoming Improvement

Not all students recorded their attendance at on-campus lectures.

Lecturer to emphasise importance of recording attendance. Research assistant remains in class until attendance recorded.

Ethics issues preclude interviewing students during semester. Hard to get students to agree to interview.

Choose a larger potential sample and spend more time contacting students.

Sample of students interviewed was too narrow. Hard to identify disengaged students.

Interview more students. Use other measures to identify different behaviour patterns. See below.

Difficult to get external students to agree to interview.

Use phone or webconference interviews. Range of units of study too narrow and role of unit

designer underplayed.

Repeat the study across a range of units with different characteristics.

Interviews need to probe reasons for study behaviour in more depth.

Interview students about the unit-specific context (e.g., Why did you not listen to Lectopia in Week 8?) and about ‗whole of unit‘ issues (What did you do in workshops? Did you do your readings before class?)

Apart from some easily rectifiable process issues, two major implications arose from this analysis: a need for the broadening of the mechanisms for identifying student behaviour patterns; and the application of the

methodology to other contexts.

There is little need for students to visit a Lectopia recording multiple times, and it, therefore, produces relatively sparse learning analytic data. This contrasts with LMS use, where students may have multiple reasons for accessing content and learning tools in a given week as they engage with the growing online community in the unit. LMS usage data is thus richer than Lectopia data and more useful as a predictive tool. A key requirement of subsequent work is to apply the same approaches used here, and elsewhere (Phillips, et al., 2002), to the analysis of LMS data.

A second mechanism for identifying student study patterns is through an initial survey of students mid-semester. The survey could probe perceptions about how students generally use educational technologies in their study. It would investigate a range of dimensions of study behaviour: interactions with the learning environment; attendance patterns; study patterns; computer usage; work hours and expectations of success. When the results of this survey are combined with LMS and Lectopia usage data, we should have a rich mechanism to identify students with diverse study behaviours at all levels of engagement, and another level of data to use in selection of students for interview.

The final methodological modification would be to study a wider range of units to investigate the impact of disciplinary characteristics and teacher beliefs on the teaching approach and the learning design. Potential units could be selected from relatively large undergraduate units across a mixture of disciplines. Selection of units could best be achieved through document analysis of study guides together with expert review by the project team in terms of how well they appear to facilitate student learning in their particular context through alignment of pedagogy, tasks and resources.

Conclusion

Educators in higher education and other sectors are, more and more, using learning management systems and lecture-capture technologies to offer students a truly flexible and rich learning experience. While teachers in these environments already have access to quite detailed information on individual student access, particularly the what and the when of student access, they currently do not have a great deal of information on the how and the why.

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₆

This study endeavoured to further understanding of student engagement within a blended learning environment. It aimed to offer some insights into learning analytics, describe how they can be used to build student activity profiles, and demonstrate that initial analysis can be fleshed out in much greater depth and accuracy in tandem with qualitative research methods. Interviews with selected students provided accurate and insightful

information on their studying habits and learning strategies.

We believe that further research into learning analytics (both our own as suggested here, and the work of many other scholars working in this field throughout the world) will refine its ability to accurately diagnose

problematic student access and identify potential at-risk behaviour. Learning analytics will further expand to easily provide teachers with information about how and why students are using unit tasks and resources, and it will be a useful evaluative tool to enable continuous improvement of learning environments for on-campus and distant students.

References

Bell, J. W., & Patterson, A. (1998). Students as Researchers: Breaking the Binds. Paper presented at the Australian Association for Research in Education (AARE) Conference, Research in Education: Does it Count?, Adelaide, Australia.

Dawson, S., Bakharia, A., & Heathcote, E. (2010). SNAPP: Realising the affordances of real-time SNA within networked learning environments. In H. V. Dirckinck-Holmfeld L, Jones C, de Laat M, McConnell D & Ryberg T (Ed.), Proceedings of the 7th International Conference on Networked Learning.

Dawson, S., McWilliam, E., & Tan, J. P. L. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice Hello! Where are you in the landscape of educational technology? Proceedings ascilite. Melbourne.

Dennen, V. P. (2008). Pedagogical lurking: Student engagement in non-posting discussion behavior. Computers in Human Behavior, 24, 1624–1633.

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter.

Goldstein, P. J., & Katz, R. N. (2005). Academic Analytics: The Uses of Management Information and Technology in Higher Education, ECAR Research Study 8 Available from

http://net.educause.edu/ir/library/pdf/ers0508/rs/ers0508w.pdf

Goodyear, P., & Retalis, S. (2010). Learning, technology and design. In P. Goodyear & S. Retalis (Eds.),

Technology-enhanced learning: design patterns and pattern languages (Vol. 2, pp. 1-28). Rotterdam: Sense Publishers.

Gosper, M., Green, D., McNeill, M., Phillips, R. A., Preston, G., & Woo, K. (2008). Final Report: The Impact of Web-Based Lecture Technologies on Current and Future Practices in Learning and Teaching. Sydney: Australian Learning and Teaching Council.

Herrington, J., Reeves, T. C., & Oliver, R. (2010). A Guide to Authentic e-Learning. New York and London: Routledge.

Kennedy, G. E., & Judd, T. S. (2004). Making sense of audit trail data. Australasian Journal of Educational Technology, 20(1), 18-32.

Littlejohn, A., & Pegler, C. (2007). Preparing for blended e-learning. Abingdon, U.K.: Routledge.

Merriam, S. B. (1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass Inc.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.

Oblinger, D. G., & Campbell, J. P. (2007). Academic Analytics. Retrieved 30 June, 2010, from http://connect.educause.edu/library/abstract/AcademicAnalytics/45275

Phillips, R. A. (2006). Tools used in Learning Management Systems: analysis of WebCT usage logs. In L. Markauskaite, P. Goodyear & P. Reimann (Eds.), The 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education: Who's Learning? Whose Technology? (pp. 663-673). Sydney: Sydney University Press.

Phillips, R. A., Baudains, C., & van Keulen, M. (2002). An Evaluation of Student Learning in a Web-Supported Unit on Plant Diversity. In A. Williamson, C. Gunn, A. Young & T. Clear (Eds.), 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 525-534). Auckland, New Zealand.

Phillips, R. A., McNaught, C., & Kennedy, G. (2011). Evaluating e-learning: Guiding research and practice. New York and London: Routledge.

Proceedings ascilite 2011 Hobart: Full Paper

₁₀₀₇

academic analytic tools to investigate studying behaviours in technology-supported learning environments. In C. H. Steel, M. J. Keppell, P. Gerbic & S. Housego (Eds.), Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010 (pp. 761-771). Sydney: ascilite.

Taplin, R. H., Low, L. H., & Brown, A. M. (2011). Students‘ satisfaction and valuation of web-based lecture recording technologies. Australasian Journal of Educational Technology, 27(2), 175-191.

van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. M. Branch, K. Gustafson, N. Nieveen & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1-14). Dordrecht: Kluwer Academic.

Acknowledgments

This work was funded by the Echo360 International Research Grants Program, and by a grant from the Murdoch University Strategic Research Fund. We also acknowledge the contribution of our colleagues Maree Gosper and Margot McNeill to the broader research project of which this work is part. The programming was done by Reef Turner, with technical support from the Perth office of Echo360. We also acknowledge the technical assistance of the Lectopia administrators at Murdoch and Macquarie Universities, Zig Tan and David Morgan,

respectively.

Author contact details:

Rob Phillips [email protected]

Please cite as: Phillips, R., Maor, D, Cumming-Potvin., Roberts, P., Herrington, J., Preston, G. & Moore, E. (2011). Learning analytics and study behaviour: A pilot study. In G. Williams, P. Statham, N. Brown & B. Cleland (Eds.), Changing Demands, Changing Directions. Proceedings ascilite Hobart 2011. (pp.997-1007). http://www.ascilite.org.au/conferences/hobart11/procs/Phillips-concise.pdf

Copyright © 2011 Rob Phillips, Dorit Maor, Wendy Cumming-Potvin, Pauline Roberts, Jan Herrington, Greg Preston & Elizabeth Moore.

The author(s) assign to ascilite and educational non-profit institutions, a non-exclusive licence to use this document for personal use and in courses of instruction, provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ascilite to publish this document on the ascilite web site and in other formats for the Proceedings ascilite Hobart 2011. Any other use is prohibited without the express permission of the author(s).