Analytics in Education

Answers to the question ‘What do you think of, when you hear the term learning analytics?’ are diverse. Some people interpret the meaning of ‘learning analytics’ as ‘learning about analytics’, while others describe it as ‘analyzing the process of learning’ or ‘analyzing the way people learn’, or ‘a method to improve learning’. All of these explanations are somehow correct, but need to be specified in more detail. It is true: researchers are still ‘learning about analytics’. LA is an emerging research field, which gathers researchers at its own international conference ‘Learning Analytics and Knowledge’ (LAK) since February 2011 (SoLAR 2013a). Also, it is a hot topic of the Horizon Reports 2011 to 2013 (L. Johnson et al. 2011; L. Johnson, Adams, and Cummins 2012; L. Johnson et al. 2013). But how can LA be distinguished from analytics in general or from related research fields, which seem to be the same or rather similar at first glance? The following sections introduce research domains that are related to LA, in order to compare them to and differentiate them from LA subsequently.

4.3.1 Analytics in General

Analytics have a major impact on a variety of fields, industries, and events. For example, in the 2012 U.S. presidential elections big data and analytics played a crucial role (Shen 2013). Some statisticians predicted “that Obama would prevail with close to 99 percent certainty based on aggregated poll data” (Shen 2013) a few days before Election Day. Analytics was not only used for prediction, but also it was an integral part of Obama’s political campaign. Large amounts of data were gathered and combined for systematically targeted promotions and fund raising,

e.g., through picking specific stars as fund-raisers (Georg Clooney and Sarah Jessica Parker) to appeal to certain donors (Shen 2013).

Analytics also is commonly used in sports, e.g. basketball, for example to evaluate the performance of players (Cade 2012). Another important area for potential benefits of analytics is healthcare. For example, analytics can help in medical research to learn more about certain diseases.

LA reuses and remanufactures methods and technologies from established fields, like analytics, web analytics, business intelligence, and data mining, statistics, social network analysis and recommender systems (Chatti et al. 2012a). For distinguishing LA from other fields it is important to know them. Therefore, these fields are described briefly before looking into more closely related fields, like educational data mining, information visualization and academic analytics in the following sections (section 4.3.2 to 4.3.4):

• ‘Analytics’ “is the discovery and communication of meaningful patterns in data” (Wikipedia 2013c). ‘Web analytics’ is specifically concerned with analyzing the usage behavior and visitors profiles of web pages. Different metrics (key performance indicators), e.g., conversion rate, measure the effectiveness of web pages or marketing campaigns concerning specific goals. Important data sources are, e.g., log files, cookies, or client-based data derived from tiny one-pixel-graphics and JavaScript.

• ‘Business Intelligence’, also known as ‘Management Information Systems’, is a term that is mainly used in an economic context (Kemper 2004). The main goal of business intelligence systems is to assist with decision-making and planning based on accurate, current, and automatically generated business reports. Hence, common methods used are different forms of data processing and mining.

• ‘Knowledge Discovery in Databases’ and ‘Data Mining’ pursue the goal to generate patterns and new useful knowledge from databases in minimal time (Ester and Sander 2013). This field is closely related to ‘statistics’. But data mining results may even be wrong from statistics point of view, since its methods are approximations that accept a certain loss of statistical correctness for runtime reasons. Related topics are also, e.g., data management, preprocessing, modeling, post-processing, and visualization. • ‘Social network analysis’ provides tools to explore and analyze networks of persons based on graph-based visualizations (social network diagrams). This way, it supports the detection of relevant actors as well as connections between individuals and items. The interest in analyzing virtual social networks increases in recent years. “This led to the development of different methods to study relationships between people, groups, organisations- and other knowledge-processing entities on the Web” (D’Andrea, Ferri, and Grifoni 2009, p. 3).

• ‘Recommender systems’ is a research area within data mining and machine learning. It can help users discover information or items they

might not have found themselves. Therefore, recommender systems gather all kinds of data about users to analyze and to suggest products, context- specific resources, other users’ profiles etc. (Zanker, Felfernig, and Friedrich 2011).

4.3.2 Educational Data Mining

In the research field of ‘Educational Data Mining (EDM)’, researchers are examining the development of methods for the generation of information from data of educational contexts. EDM is therefore the combination of data mining techniques with educational data. It aims to better understand students' learning processes and settings in which they learn (EDM 2013).

LA and EDM are quite similar regarding the analysis domain, data, processes, and objectives. Both fields focus on the educational domain, work with data originating from educational environments, and convert this data into relevant information with the aim of improving the learning process. However, the techniques used for LA can be quite different from those used in EDM. EDM basically focuses on the application of typical data mining techniques (i.e. clustering, classification, and association rule mining) to support teachers and students in analyzing the learning process. But EDM is mostly targeting the objectives of researchers, since it is more concerned with the development and evaluations of new data mining methods than with testing its usefulness in practical situations. In addition to data mining techniques, LA further includes other methods, such as statistics, visualization tools or social network analysis (SNA) techniques, and puts them into practice for studying their actual effectiveness on the improvement of teaching and learning. Therefore, further ‘real world’ issues, like data privacy, are important.

C. Romero and Ventura (2007), Baker and Yacef (2009), and Cristóbal Romero and Ventura (2010) provide excellent reviews of how EDM has developed in recent years as well as the major trends in EDM research up to 2009.

4.3.3 Information Visualization

Numerous studies have stated that large tables are not a particular user-friendly form of presenting LA information (Mazza and Dimitrova 2007; Ali et al. 2012; Cristóbal Romero, Ventura, and García 2008). For practical purposes, meaningful visualizations that are easy to interpret are required. Creating knowledge on the design of such visualizations is the objective of the research area ‘information visualization (IV)’.

According to Card, Mackinlay, and Shneiderman (1999), information visualization is “[t]he use of computer-supported, interactive, visual representations of abstract data to amplify cognition” (p. 7). A dictionary defines visualization as “the act or process of interpreting in visual terms or of putting into visible form” (Merriam-Webster 2013). The first definition emphasizes the

usage of tools to amplify cognition and the latter points out the activity of interpretation during the usage of visualizations. Hence, with the help of external visualizations users are “using vision to think”, as the title of a famous book regarding IV suggests (Card, Mackinlay, and Shneiderman 1999). Another way to put it, according to Norman (1993), is that the invention of external aids, such as diagrams, has increased human memory, thought, and reasoning. Nevertheless, visualizations can also confuse their users. Therefore, it is important to understand, how they are used properly (Few 2007). Card, Mackinlay, and Shneiderman (1999) propose six ways in which visualizations can amplify cognition:

“(1) by increasing the memory and processing resources available to the user, (2) by reducing the search for information, (3) by using visual representations to enhance the detection of patterns, (4) by enabling perceptual inference operations, (5) by using perceptual attention mechanisms for monitoring, and (6) by encoding information in a manipulable medium” (p. 16).

However, the main research question of the field of IV is how to convert data into interactive graphical representations, while preserving and emphasizing the intended meaning (Mazza 2004). An answer to this question “depends on the nature of the data, the type of information to be represented and its use, but more consistently, it depends on the creativity of the designer of the graphical representation” (Mazza 2004, p. 29).

Figure 2 shows how data can be mapped from raw data to visual form. It depicts a simple reference model for information visualization systems by Card, Mackinlay, and Shneiderman (1999). Data transformations convert raw data into data tables, visual mappings transform them into visual structures, and view transformations generate views through parameter selection (e.g., position, scaling, and clipping) (p. 17).

Findings from IV are important for presenting analytics outcomes because they can help to show information based on data and condensed in small spaces. Information visualizations are an integral part of LA, since they provide possibilities to engage users in monitoring and analyzing activities, while quickly conveying relevant information. Stephen Few (2006) emphasizes the need for good design principles for interactive dashboards and charts. This involves, knowing your audience and their goals, along with studying how visual perception works, and how to take advantage of this. “When properly designed for effective visual communication, dashboards support a level of awareness—a picture of what’s going on—that could never be stitched together from traditional reports.” (Few 2007, p. 5) Since LA tools often are types of dashboards, his findings are especially relevant for LA.

4.3.4 Academic Analytics

Before this work goes into details on LA, a closely related and by definition overlapping term needs to be defined. According to J. Campbell and Oblinger (2007) academic analytics:

“[…] can help institutions address student success and accountability while better fulfilling their academic missions. Academic systems generate a wide array of data that can predict retention and graduation. Academic analytics marries that data with statistical techniques and predictive modeling to help faculty and advisors determine which students may face academic difficulty, allowing interventions to help them succeed.” (Abstract)

By replacing ‘academic’ with ‘learning’ this definition could also be used for LA. Therefore, academic analytics, which was introduced by (Goldstein and Katz 2005), needs to be differentiated in more detail to be able to draw the fine line between both fields.

One way to do this is to have a look at the motivation behind academic analytics. Higher education institutions experience a growing demand for accountability and they have to document and present data on their accomplishments (J. Campbell and Oblinger 2007; J. P. Campbell, DeBlois, and Oblinger 2007), if they want to be capable of competing for students. This need is based on political and financial reasons. Example analytics implementations are predicting enrollment, student success, or student retention (J. P. Campbell, DeBlois, and Oblinger 2007). Against this background, academic analytics serve analogue purposes as business analytics or business intelligence.

Nonetheless, academic analytics serves the improvement of learning as well by pursuing the goals of increasing students success and graduation rates. Related projects use diverse sets of educational data, e.g., to predict students’ risks status, allowing faculty and advisors to intervene (Arnold 2010; Tally 2009). While academic analytics is situated more at the level of higher education or even

national decision making, LA is rather situated at a local level of teaching and learning in actual courses. It is concerned with the support of the actors of specific courses, like teachers and learners, by providing analytics for awareness, reflection, and action – as investigated by this thesis.