Ethics and Learning Analytics: Charting the (Un)Charted

Open Research Online

The Open University’s repository of research publications

and other research outputs

Ethics and Learning Analytics: Charting the

(Un)Charted

Book Section

How to cite:

Prinsloo, Paul and Slade, Sharon (2017). Ethics and Learning Analytics: Charting the (Un)Charted. In: Lang, Charles;

Siemens, George; Wise, Alyssa and Gaˇsevi´

c, Dragan eds. Handbook of Learning Analytics. SOLAR, pp. 49–57.

For guidance on citations see FAQs.

c

[not recorded]

Version: Version of Record

Link(s) to article on publisher’s website:

http://dx.doi.org/doi:10.18608/hla17

https://solaresearch.org/wp-content/uploads/2017/05/hla17.pdf

Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyright

owners. For more information on Open Research Online’s data policy on reuse of materials please consult the policies

page.

,QWKH1HZ0HGLD&RQVRUWLXPÚV+RUL]RQ5HSRUW (NMC, 2011) pointed to the increasing importance of learning analytics as an emerging technology, which has since developed from a mid-range technology or WUHQGWRRQHWREHUHDOL]HGZLWKLQDÜRQH\HDURUOHVVÝ WLPHIUDPH10&S7KRXJKWKHUHDUHFOHDU linkages between learning analytics and the more HVWDEOLVKHGāHOGRIHGXFDWLRQDOGDWDPLQLQJWKHUH are also important distinctions regarding, inter alia, automation, aims, origins, techniques, and methods 6LHPHQV %DNHU$VWKHāHOGRIOHDUQLQJDQ-DO\WLFVKDVGHYHORSHGDVDGLVWLQFWāHOGRIUHVHDUFK and practice (see van Barneveld, Arnold, & Campbell, 2012), so too thinking around ethical issues has slowly PRYHGLQIURPWKHPDUJLQV6ODGHDQG3ULQVORR established one of the earliest frameworks developed with a focus on ethics in learning analytics. Since then, WKHQXPEHURIDXWKRUVSXEOLVKLQJLQWKLVVXEāHOGKDV VLJQLāFDQWO\LQFUHDVHGUHVXOWLQJLQDJURZLQJQXPEHU RIIUDPHZRUNVFRGHVRISUDFWLFHVWD[RQRPLHVDQG JXLGHOLQHV*DÇHYLİ'DZVRQ -RYDQRYLİ ,QWKHZLGHUFRQWH[WRISXEOLFFRQFHUQVVXUURXQGLQJ

increasing surveillance and the (un)warranted col-OHFWLRQDQDO\VLVDQGXVHRISHUVRQDOGDWDÜIHDUVDQG UHDOLWLHVDUHRIWHQLQGLVWLQJXLVKDEO\PL[HGXSOHDGLQJ to an atmosphere of uncertainty among potential ben-HāFLDULHVÝ'UDFKVOHU *UHOOHUS*DÇHYLİHW DODOVRVXJJHVWWKDWIXUWKHUFKDOOHQJHVUHPDLQ ÜWREHDGGUHVVHGLQRUGHUWRIXUWKHUDLGXSWDNHDQG LQWHJUDWLRQLQWRHGXFDWLRQDOSUDFWLFHÝDQGVHHHWKLFV DQGSULYDF\DVLPSRUWDQWHQDEOHUVWRWKHāHOGRIOHDUQ-LQJDQDO\WLFVÜUDWKHUWKDQEDUULHUVÝS :HEULHĂ\VLWXDWHVHWWKHFRQWH[WIRUFRQVLGHULQJWKH ethical implications of learning analytics, before PDSSLQJRXUSHUVRQDOUHVHDUFKMRXUQH\LQWKHāHOG We then consider recent developments and conclude E\ĂDJJLQJDVHOHFWLRQRILVVXHVWKDWFRQWLQXHWRUHTXLUH broader and more critical engagement.

There is some consensus that the future of learning will be digital, distributed, and data-driven such that

SETTING THE CONTEXT: WHY

ETHICS IS RELEVANT

Chapter 4: Ethics and Learning Analytics:

Charting the (Un)Charted

Paul Prinsloo,

1

_{Sharon Slade}

2

$VWKHāHOGRIOHDUQLQJDQDO\WLFVPDWXUHVDQGGLVFRXUVHVVXUURXQGLQJWKHVFRSHGHāQLWLRQ FKDOOHQJHVDQGRSSRUWXQLWLHVRIOHDUQLQJDQDO\WLFVEHFRPHPRUHQXDQFHGWKHUHLVEHQHāW both in reviewing how far we have come in considering associated ethical issues and in looking ahead. This chapter provides an overview of how our own thinking has developed DQGPDSVRXUMRXUQH\DJDLQVWEURDGHUGHYHORSPHQWVLQWKHāHOG$JDLQVWDEDFNGURSRI technological advances and increasing concerns around pervasive surveillance and the role and unintended consequences of algorithms, the development of research in learning analytics as an ethical and moral practice provides a rich picture of fears and realities. More importantly, we begin to see ethics and privacy as crucial enablers within learning DQDO\WLFV7KHFKDSWHUEULHĂ\ORFDWHVHWKLFVLQOHDUQLQJDQDO\WLFVLQWKHEURDGHUFRQWH[W of the forces shaping higher education and the roles of data and evidence before tracking RXUSHUVRQDOUHVHDUFKMRXUQH\KLJKOLJKWLQJFXUUHQWZRUNLQWKHāHOGDQGFRQFOXGLQJE\ mapping future issues for consideration.

Keywords:(WKLFVDUWLāFLDOLQWHOOLJHQFHGDWDELJGDWDVWXGHQWV

ABSTRACT

1 _{Department of Business Management, University of South Africa, South Africa} 2 _{Faculty of Business and Law, The Open University, United Kingdom}

HANDBOOK OF LEARNING ANALYTICS PG 50

HGXFDWLRQÜHQDEOHVTXDOLW\RIOLIHDQGPHDQLQJIXOHP-SOR\PHQWWKURXJKDH[FHSWLRQDOTXDOLW\UHVHDUFKE sophisticated data collection and; (c) advanced machine OHDUQLQJDQGKXPDQOHDUQLQJDQDO\VLVVXSSRUWÝ6LH-PHQV6OLGH$OWKRXJKFRQVLGHUDWLRQVRIWKH ethical implications of learning analytics were initially RQWKHPDUJLQVRIWKHāHOGWKHSURPLQHQFHRIHWKLFV has come a long way and is increasingly foregrounded *DÇHYLİHWDO,QDFRQWH[WZKHUHPXFKLVWR be said for the potential economicEHQHāWVIRUERWK students and the institution) of more successful OHDUQLQJH[SHULHQFHVUHVXOWLQJIURPLQFUHDVHGGDWD harvesting, we should not ignore the possibilities of ÜGDWDSUR[\LQGXFHGKDUGVKLSâZKHQWKHGHWDLORE-WDLQHGIURPWKHGDWDSUR[\FRPHVWRGLVDGYDQWDJHLWV HPERGLHGUHIHUHQWLQVRPHZD\Ý6PLWKSDOVR VHH5XJJLHUR6WUDXVVEDQG:DWWHUV Ethical implications around the collection, analysis, DQGXVHRIVWXGHQWGDWDVKRXOGWDNHFRJQL]DQFHRIWKH SRWHQWLDOO\FRQĂLFWLQJLQWHUHVWVDQGFODLPVRIDUDQJHRI VWDNHKROGHUVVXFKDVVWXGHQWVDQGLQVWLWXWLRQV9LHZV RQWKHEHQHāWVULVNVDQGSRWHQWLDOIRUKDUPUHVXOWLQJ from the collection, analysis, and use of student data will depend on the interests and perceptions of the particular stakeholder. In this chapter, we hope to provide insight into the different positionalities, claims, and interests of primarily students and institutions.

Although now becoming more established, ethics and the need to question how student data is used and under what conditions was very much a marginal LVVXHLQWKHHDUO\\HDUVRIWKHāHOG7KHāUVWDWWHPSWV WRH[SORUHZLGHULVVXHVDURXQGOHDUQLQJDQDO\WLFV ZHUHSUHVHQWHGDW/$.ÚLQ9DQFRXYHU7KHODUJH majority of sessions at this conference remained focused on developmental work. There was some mention of stakeholder perceptions of the ways in ZKLFKVWXGHQWGDWDFRXOGEHXVHGQRWDEO\'UDFKVOHU DQG*UHOOHUWKRXJKWKHLUSDSHUVXJJHVWHGWKDW surveyed stakeholder considerations were largely focused on privacy and not considered particularly FRQWHQWLRXV$IXUWKHUSDSHU3ULQVORR6ODGH *DOSLQ 2012) touched upon the need to consider the impacts of all stakeholders on students’ learning journeys in order to increase the success of students’ learning. 7KHQRWLRQRIÜWKLUGVSDFHÝSURYLGHGDXVHIXOKHXULVWLF to map the challenges and opportunities, but also WKHSDUDGR[HVRIOHDUQLQJDQDO\WLFVDQGLWVSRWHQWLDO impact on student success and retention. At the same FRQIHUHQFHDQH[SORUDWRU\ZRUNVKRS6ODGH *DOSLQ EXLOWXSRQHDUO\ZRUNE\&DPSEHOO'H%ORLVDQG 2EOLQJHUDLPLQJWRFRQVLGHUDQGH[SDQGXSRQ

a number of assumed relevant ethical issues from different stakeholder perspectives.

:RUNLQPRYHGRQWRDQH[DPLQDWLRQRIH[LVW-ing institutional policy frameworks that set out the purposes for how data would be used and protected 3ULQVORR 6ODGH7KHJURZLQJDGYHQWRIOHDUQ-LQJDQDO\WLFVKDGVHHQXVHVRIVWXGHQWGDWDH[SDQGLQJ rapidly. In general, policies relating to institutional use of student data had not kept pace, nor taken account RIWKHJURZLQJQHHGWRUHFRJQL]HHWKLFDOFRQFHUQV focusing mainly on data governance, data security, DQGSULYDF\LVVXHV7KHUHYLHZLGHQWLāHGFOHDUJDSV DQGWKHLQVXIāFLHQF\RIH[LVWLQJSROLF\

Taking a sociocritical perspective on the use of learn-LQJDQDO\WLFV6ODGHDQG3ULQVORRFRQVLGHUHGD QXPEHURILVVXHVDIIHFWLQJWKHVFRSHDQGGHāQLWLRQRI the ethical use of learning analytics. A range of ethical issues was grouped within three broad, overlapping categories, namely:

• The location and interpretation of data

• ,QIRUPHGFRQVHQWSULYDF\DQGWKHGHLGHQWLā-cation of data

• 7KHPDQDJHPHQWFODVVLāFDWLRQDQGVWRUDJHRIGDWD 6ODGHDQG3ULQVORRSURSRVHGDIUDPHZRUNEDVHG RQWKHIROORZLQJVL[SULQFLSOHV

1. Learning analytics as moral practice — focusing not only on what is effective, but on what is ap-propriate and morally necessary

2. Students as agents — to be engaged as collabora-tors and not as mere recipients of interventions and services

3. Student identity and performance as temporal G\QDPLFFRQVWUXFWVØUHFRJQL]LQJWKDWDQDO\WLFV provides a snapshot view of a learner at a particular WLPHDQGFRQWH[W

4. 6WXGHQWVXFFHVVDVDFRPSOH[PXOWLGLPHQVLRQDO phenomenon

5. Transparency as important — regarding the pur-poses for which data will be used, under what conditions, access to data, and the protection of an individual’s identity

 That higher education cannot afford not to use data These principles offer a useful starting position, but ought sensibly to be supported by consideration of a number of practical considerations, such as the development of a thorough understanding of who EHQHāWVDQGXQGHUZKDWFRQGLWLRQVHVWDEOLVKPHQW RILQVWLWXWLRQDOSRVLWLRQVRQFRQVHQWGHLGHQWLāFDWLRQ and opting out; issues around vulnerability and harm (e.g., inadvertent labelling); systems of redress (for both student and institution); data collection, analyses,

ESTABLISHING ETHICAL

PRINCI-PLES: HOW FAR HAVE WE COME?

access, and storage (e.g., security issues and avoiding perpetuation of bias); and governance and resource allocation (including clarity around the key drivers IRUÜVXFFHVVÝDQGZKDWVXFFHVVPHDQVH[LVWLQJFRQ-straints, and the conditions that must be met). This latter aspect of resource allocation was carefully H[SORUHGLQDODWHUSDSHUFRQVLGHULQJWKHFRQFHSWRI HGXFDWLRQDOWULDJH3ULQVORR 6ODGHD$OWKRXJK learning analytics offers theoretical opportunities for HEIs (higher education institutions) to proactively identify and support students at risk of failing or drop-SLQJRXWWKH\GRVRLQDFRQWH[WZKHUHE\UHVRXUFHV are (increasingly) limited. The challenge then is where

best to direct support resources and on what basis that

decision is made. The concept of educational triage as a means of directing support toward students most OLNHO\WRÜVXUYLYHÝUHTXLUHVFDUHIXOFRQVLGHUDWLRQRI DQXPEHURIUHODWHGDQGFRPSOH[LVVXHVVXFKDVWKH balance between respecting student autonomy and, at the same time, ensuring the long-term sustain-DELOLW\RIWKHLQVWLWXWLRQWKHQRWLRQRIEHQHāFHQFHWR always act in the student’s best interest); the need for QRQPDOHāFHQFHLQĂLFWLQJWKHOHDVWKDUPSRVVLEOHWR UHDFKDEHQHāFLDORXWFRPHDQGPDLQWDLQLQJDVHQVHRI distributive justice (understanding that demographic characteristics have and do impact support provided DQGDVVXPSWLRQVPDGHDQGWKHQHHGWRUHFRJQL]H and address this).

An increasing awareness of learning analytics as a means of doing something to the student without that student necessarily knowing triggered further H[SORUDWLRQRILVVXHVDURXQGVXUYHLOODQFHVWXGHQW SULYDF\DQGLQVWLWXWLRQDODFFRXQWDELOLW\3ULQVORR  Slade, 2014b). The resulting discussion challenged assumptions around learning analytics as a produc-er of accurate, objective, fully complete pictures of student learning, and also reviewed the potentially unequal relationship between institution and student. ,QFRQVLGHULQJH[LVWLQJIUDPHZRUNVUHJDUGLQJWKHXVH DQGDQDO\VLVRISHUVRQDOGDWDWKHVWXG\VXJJHVWHGVL[ elements that could form a basis for a student-centred learning analytics:

1. 7KHXVHRIDJJUHJDWHGQRQSHUVRQDOL]HGGDWDLV essential in delivering effective and appropriate teaching and learning, but students should be able to make informed opt in/out decisions

2. Students should have full(er) knowledge of which data is collected and how it is used

3. Students should ensure that their personal data records are complete and up to date

4. The surveillance of activities and the harvesting of data must not harm student progress

5. Algorithmic output should be subject to (potential)

human review, and corrected if needed

 /HDUQLQJDQDO\WLFVHVVHQWLDOO\SURYLGHVFRQWH[W DQGWLPHVSHFLāFSURYLVLRQDOLQFRPSOHWHSLFWXUHV of students, and algorithms should be frequently reviewed and validated

,VVXHVDURXQGVXUYHLOODQFHDQGWKHQHHGWRUHFRJQL]H students as active agents in the use of their own data ZDVH[SOLFLWO\DGGUHVVHGZLWKLQWKHGHYHORSPHQWRI7KH Open University (OU; 2014) policy on the ethical use of student data for learning analytics. As part of the stakeholder consultation, a representative group of 50 VWXGHQWVH[SORUHGWKHLUXQGHUVWDQGLQJRIWKHZD\VLQ which data is used to support students in completing their study goals over a three-week period. A study of UHVSRQVHV6ODGH 3ULQVORRIRXQGWKDWVWXGHQWV DSSHDUHGODUJHO\XQDZDUHRIWKHH[WHQWWRZKLFKGDWD was already actively collected and used, and they raised a number of concerns. The major concern related to the potential to actively consent (or not), with a PDMRULW\RIVWXGHQWVH[SUHVVLQJDZLVKIRUDULJKWWR opt out. This direct involvement of student voices in shaping a policy dealing with the ethics of learning analytics offered unique insight into the ways in which students regard their data — as a valuable entity to be carefully protected and even more carefully applied. *LYHQWKDWWKHVDPSOHLQ6ODGHDQG3ULQVORRPD\ not be fully representative of the total population, the RXWFRPHVFDQQRWEHJHQHUDOL]HGDFURVVLQVWLWXWLRQDO DQGJHRSROLWLFDOFRQWH[WV

In response to this growing awareness of student con-FHUQ3ULQVORRDQG6ODGHTXHVWLRQHGZKHWKHURXU assumptions and understanding of issues surround-LQJVWXGHQWDWWLWXGHVWRSULYDF\PD\EHLQĂXHQFHG by both the apparent ease with which the public appear to share the detail of their lives and by our largely paternalistic institutional cultures. The study H[SORUHGLVVXHVDURXQGFRQVHQWDQGWKHVHHPLQJO\ simple choice to allow students to opt-in or opt-out of having their data tracked. As a foundation for the discussion, the terms and conditions of three massive open online course (MOOC) providers were reviewed to establish information given to users regarding the XVHVRIWKHLUGDWD7KLVH[WHQGHGLQWRDGLVFXVVLRQRI how HEIs can move toward an approach that engages and more fully informs students of the implications of learning analytics on their personal data. A similar WKHPHZDVSXUVXHGLQ3ULQVORRDQG6ODGHD7KLV paper challenged the tendency for many HEIs to adopt DQDXWKRULWDULDQDSSURDFKWRVWXGHQWGDWD'HVSLWH the rapid growth in the deployment of learning ana-lytics, few HEIs have regulatory frameworks in place and/or are fully transparent regarding the scope of VWXGHQWGDWDFROOHFWHGDQDO\]HGXVHGDQGVKDUHG

6WXGHQWYXOQHUDELOLW\ZDVH[SORUHGLQWKHQH[XVEH-HANDBOOK OF LEARNING ANALYTICS PG 52

WZHHQUHDOL]LQJWKHSRWHQWLDORIOHDUQLQJDQDO\WLFV WKHILGXFLDU\GXW\RI+(,VLQWKHFRQWH[WRIWKHLU asymmetrical information and power relations with VWXGHQWVDQGWKHFRPSOH[LWLHVVXUURXQGLQJVWXGHQW agency in learning analytics. The aim was to consider ways in which student vulnerability may be addressed, increasing student agency, and empowering them as active participants in learning analytics — moving from TXDQWLāHGGDWDREMHFWVWRTXDOLāHGDQGTXDOLI\LQJ VHOYHVVHHDOVR3ULQVORR 6ODGHE

It is broadly accepted that the increasing value of GDWDDVDVKDUDEOHFRPPRGLW\ZLWKDQLQFUHDVLQJH[-change value has overtaken our legal and traditional HWKLFDOIUDPHZRUNV=KDQJÜ'HHSHFRQRPLF SUHVVXUHVDUHGULYLQJWKHLQWHQVLāFDWLRQRIFRQQHFWLRQ DQGPRQLWRULQJRQOLQHÝ&RXOGU\SDUDQG Ü:KDWÚVQHHGHGLVPRUHFROOHFWLYHUHĂHFWLRQRQWKH costs of capitalism’s new data relations for our very SRVVLELOLWLHVRIHWKLFDOOLIHÝ&RXOGU\SDU$V such, there have been attempts in different geopolitical DQGLQVWLWXWLRQDOFRQWH[WVWRJUDSSOHZLWKWKHHWKLFDO LPSOLFDWLRQVRIOHDUQLQJDQDO\WLFV6FODWHU3HDVJRRG DQG0XOODQIRUH[DPSOHUHYLHZSUDFWLFHVZLWKLQ higher education in the United States, Australia, and WKH8QLWHG.LQJGRP7KH\VXPPDUL]HWKHLUāQGLQJV E\LQGLFDWLQJWKDWOHDUQLQJDQDO\WLFVPDNHVVLJQLāFDQW contributions for 1) quality assurance and quality im-provement; 2) boosting retention rates; 3) assessing and acting upon differential outcomes among the student population; and 4) the development and introduction of adaptive learning. The report acknowledges the many opportunities, but also highlights threats such DVÜHWKLFDODQGGDWDSULYDF\LVVXHVÙRYHUDQDO\VLVÚDQG WKHODFNRIJHQHUDOL]DELOLW\RIWKHUHVXOWVSRVVLELOLWLHV IRUPLVFODVVLāFDWLRQRISDWWHUQVDQGFRQWUDGLFWRU\ āQGLQJVÝS,QWKHLUUHYLHZWKHRQHLQVWLWXWLRQDO H[DPSOHRIDSROLF\OHYHOELGWRDGGUHVVWKHHWKLFDO concerns in learning analytics is that of The Open 8QLYHUVLW\8.,QWKH28SXEOLVKHGDÜ3ROLF\ RQHWKLFDOXVHRIVWXGHQWGDWDIRUOHDUQLQJDQDO\WLFVÝ delimiting the nature and scope of data collected and DQDO\]HGDQGDQH[SOLFLWVSHFLāFDWLRQRIGDWDWKDWZLOO

not be collected and used for learning analytics. The

SROLF\HVWDEOLVKHVWKHIROORZLQJHLJKWSULQFLSOHVS 1. Learning analytics is an ethical practice that

VKRXOGDOLJQZLWKFRUHRUJDQL]DWLRQDOSULQFLSOHV such as open entry to undergraduate level study. 2. The OU has a responsibility to all stakeholders to

XVHDQGH[WUDFWPHDQLQJIURPVWXGHQWGDWDIRU WKHEHQHāWRIVWXGHQWVZKHUHIHDVLEOH

3. 6WXGHQWVVKRXOGQRWEHZKROO\GHāQHGE\WKHLU

YLVLEOHGDWDRURXULQWHUSUHWDWLRQRIWKDWGDWD>7KLV principle furthermore warns against stereotyping students and acknowledges those individuals who GRQRWāWLQWRW\SLFDOSDWWHUQV7KHSULQFLSOHDOVR makes it clear that members of staff may not have access to the full data set, which can seriously LPSDFWWKHUHOLDELOLW\RIWKHDQDO\VLV@

4. The purpose and the boundaries regarding the XVHRIOHDUQLQJDQDO\WLFVVKRXOGEHZHOOGHāQHG and visible.

5. The University is transparent regarding data collection, and will provide students with the opportunity to update their own data at regular intervals.

 Students should be engaged as active agents in the implementation of learning analytics (e.g., LQIRUPHGFRQVHQWSHUVRQDOL]HGOHDUQLQJSDWKV interventions).

7. Modelling and interventions based on analysis of data should be sound and free from bias.

8. Adoption of learning analytics within the OU re-TXLUHVEURDGDFFHSWDQFHRIWKHYDOXHVDQGEHQHāWV RUJDQL]DWLRQDOFXOWXUHDQGWKHGHYHORSPHQWRI DSSURSULDWHVNLOOVDFURVVWKHRUJDQL]DWLRQ $VRQHRIWKHāUVWLQVWLWXWLRQDOUHVSRQVHVWRWKHHWKL-cal implications in the collection, analysis, and use of student data, this policy and its principles attempted WRPDSXQFKDUWHGWHUULWRU\2IVSHFLāFLQWHUHVWLV WKHGHāQLWLRQRIÜLQIRUPHGFRQVHQWÝDVUHIHUULQJWR ÜWKHSURFHVVZKHUHE\WKHVWXGHQWLVPDGHDZDUHRI the purposes to which some or all of their data may be used for learning analytics and provides consent. Informed consent applies at the point of reservation RUUHJLVWUDWLRQRQWRDPRGXOHRUTXDOLāFDWLRQÝ2SHQ University, 2014, p. 3). The policy does not address the possibility of students who prefer to opt out of the collection, analysis, and use of their data (as discussed E\(QJHOIULHW0DQGHUYHOG -HXQLQN6FODWHU DOVRVHH6KDFNORFN

In a recent overview of learning analytics practices in WKH$XVWUDOLDQFRQWH[W'DZVRQ*DÇHYLİDQG5RJHUV UHSRUWWKDWWKHÜUHODWLYHVLOHQFHDIIRUGHGWR HWKLFVDFURVVWKHVWXGLHVLVVLJQLāFDQWÝSDQGWKDW WKLVÜGRHVQRWUHĂHFWWKHVHULRXVQHVVZLWKZKLFKWKH VHFWRUVKRXOGFRQVLGHUWKHVHLVVXHVÝS7KHUHSRUW VXJJHVWVWKDWÜ,WLVOLNHO\WKDWWKHKLJKHUHGXFDWLRQ sector has not been ready for such a conversation previously, although it is argued that as institutions are maturing, ethical considerations take on a heightened VDOLHQFHÝS

$OVRLQWKH$XVWUDOLDQFRQWH[W:HOVKDQG0F.LQQH\

SRVLWLRQWKHQHHGIRUD&RGHRI3UDFWLFHLQOHDUQ-RECENT DEVELOPMENTS IN

ETHICAL FRAMEWORKS

LQJDQDO\WLFVLQWKHFRQWH[WRIWKHÜUHODWLYHLPPDWXULW\ of the discipline with institutions, practitioners and WHFKQRORJ\YHQGRUVVWLOOāJXULQJRXWZKDWZRUNVDQG āQGLQJWKHERXQGDULHVRIÙDFFHSWDEOHÚSUDFWLFHÝDQG the real potential for abuse/misuse and discrimination (p. 588). Of particular importance is the commitment WKDWÜ7KH8QLYHUVLW\ZLOOQRWHQJDJHLQ/HDUQLQJ$QD-lytics practices that use data sources: (a) not directly related to learning and teaching; and/or (b) where XVHUVPD\QRWUHDVRQDEO\H[SHFWVXFKGDWDFROOHFWLRQ E\WKH8QLYHUVLW\WRRFFXUÝS6WXGHQWGDWDZLOO RQO\EHXVHGLQWKHFRQWH[WRIWKHRULJLQDOSXUSRVHIRU which the data in question was collected; its use can continue under the following conditions:

([SOLFLWLQIRUPHGFRQVHQWLVJDWKHUHGIURP those who are the subject of measurement. Where informed consent means that: (a) clear and accurate information is provided about what data is or may be collected, why and how it is collected, how it is stored and how it is used; and (b) agreement is freely given to the practice(s) described. (p. 590)

The above principles should be read in conjunction with two remaining principles regarding how col-lected data should be used to enhance teaching and OHDUQLQJDQGWRJLYHVWXGHQWVÜJUHDWHUFRQWURORYHU DQGUHVSRQVLELOLW\IRUWKHLUOHDUQLQJÝSDQGRQH of transparency and informed participation. For a full discussion, see Welsh and McKinney (2015).

'UDFKVOHUDQG*UHOOHUSURYLGHDEURDGRYHUYLHZ of ethics, privacy, and respective legal frameworks, and highlight challenges such as the real possibility RIH[SORLWDWLRQLQOLJKWRIWKHDV\PPHWULFDOSRZHU relationship between data gatherer and data object, issues of ownership, anonymity and data security, privacy and data identity, as well as transparency and WUXVW7KH\SUHVHQWDFKHFNOLVW'(/,&$7(jWRHQVXUH that learning analytics proceeds in an acceptable and FRPSOLDQWZD\ÜWRRYHUFRPHWKHIHDUVFRQQHFWHGWR GDWDDJJUHJDWLRQDQGSURFHVVLQJSROLFLHVÝS 6FODWHUSURSRVHVDGUDIWWD[RQRP\RIHWKLFDO legal, and logistical issues in learning analytics with an overview of how a range of stakeholders, such as senior management, the analytics committee, data scientists, educational researchers, IT, and students are impacted and have responsibility in learning analytics. The draft covers a wide range of issues in-cluding, inter alia, consent; identity; potential impacts of opting out; the asymmetrical relationship between the institution and students; (boundaries around) the permissible uses of student data; transparency; GDWDLQFOXGHGDQGH[FOXGHGIURPXVHDQGVWXGHQW autonomy, amongst others. See Sclater (2015) for a full list of ethical concerns.

,QWKH'XWFKKLJKHUHGXFDWLRQFRQWH[W(QJHOIULHWHW al. (2015) consider the implications of the Law for the 3URWHFWLRQRI3HUVRQDO,QIRUPDWLRQIRUOHDUQLQJDQD-lytics. These include the need for permission (and the responsibility arising from receiving consent) and the implications of the consensual agreement between a service provider and recipient that the provider may use any personal information needed for the provision of the service. The law distinguishes between essential LQIRUPDWLRQDQGÜKDQG\ÝLQIRUPDWLRQ(QJHOIULHWHWDO (2015) take a contentious view that, given that learning analytics is seen as an emerging practice, it may safely EHUHJDUGHGDVFROOHFWLQJÜKDQG\ÝLQIRUPDWLRQDQG VRSHUKDSVH[FOXGHGIURPWKHQHHGIRUFRQVHQVXDO agreement between the institution and students. The authors suggest that these four principles should guide learning analytics:

• 3HUVRQDOLQIRUPDWLRQEHXVHGRQO\LQWKHFRQWH[W and purpose for which it was provided

• Subsequent use of such data should be reconcilable ZLWKWKHRULJLQDOFRQWH[WDQGSXUSRVH

• 'DWDVKRXOGEHFDUHIXOO\FROOHFWHGDQGDQDO\]HGDQG ÜVQHDN\ÝÜVWLHNHPHÝLQ'XWFKXVDJHRIDQDO\WLFVLV QRWSHUPLVVLEOHWKLVDSSHDUVWRHPSKDVL]HDQHHG for transparency, student consultation, and buy in • 'DWDPD\RQO\EHFROOHFWHGZKHQWKHSXUSRVH XVHRIWKHFROOHFWHGGDWDLVPDGHH[SOLFLWO\FOHDU (QJHOIULHWHWDOH[SORUHVWXGHQWULJKWVDURXQG the governance of their data, including the following: • Easy access to collected information

• The right to correct wrong information (or inter-pretations arising from it)

• The right to remove irrelevant information 2ISDUWLFXODULQWHUHVWLVDQH[SORUDWLRQRIWKHHWKLFDO implications for algorithmic decision making and the DXWKRUVĂDJH[DPSOHVWKDWOHDGWRSRWHQWLDOFRQĂLFW ZLWK'XWFKODZ7KHLPSOLFDWLRQLVWKDWKXPDQVQHHGWR take responsibility for and have oversight of algorith-mic decision making. Algorithms may, at most, signal particular behaviour for the attention of faculty or support staff. Further, students have a right to appeal decisions made based on analyses of their personal data. In cases where HEIs subcontract to software developers, the final responsibility and oversight remains securely with the institution and cannot be delegated (see Engelfriet et al., 2015).

It falls outside the scope of this chapter to map current DQGIXWXUHJDSVLQRXUXQGHUVWDQGLQJRIWKHFRPSOH[-ities and practicalDQGIXWXUHJDSVLQRXUXQGHUVWDQGLQJRIWKHFRPSOH[-ities at the intersections between

HANDBOOK OF LEARNING ANALYTICS PG 54

student data and advances in technology and methods of analysis. We would like to conclude, however, with some pointers for future consideration.

*LYHQWKHPDQGDWHRIKLJKHUHGXFDWLRQLQVWLWXWLRQVWR ensure effective, appropriate, cost-effective learning H[SHULHQFHVDQGWRVXSSRUWVWXGHQWVWREHVXFFHVVIXO there is broad agreement that institutions have a right to collect and use student information. However, there is no easily agreed upon position around consent, that is, in allowing students to opt out of the collection, analysis, and use of their data. Student positions DURXQGFRQVHQWPD\EHLQĂXHQFHGE\LVVXHVQRWZKROO\ logical or rational. The often-implicit calculation of EHQHāWVFRVWVDQGULVNVZLOOGHSHQGRQDUDQJHRI IDFWRUVVXFKDVLQWHUDOLDSUHYLRXVH[SHULHQFHVQHHG DQGSHUFHLYHGEHQHāWVVHHIRUH[DPSOH'DQLHO3LQN in O’Brien. 2010).

2QHUHFHQWH[DPSOHRIRSWRXWZDVOHGE\WKH1DWLRQDO Center for Fair and Open Testing in the US who en-couraged students to refuse to take government-man-GDWHGVWDQGDUGL]HGWHVWV$URXQGVWXGHQWV RSWHGRXWLQWKH×VFKRRO\HDU)DLU7HVWQG ZLWKWKH86'HSDUWPHQWRI(GXFDWLRQUHVSRQGLQJE\ WKUHDWHQLQJWRZLWKKROGIXQGLQJ6WUDXVVD )XUWKHUUHVHDUFKLVQHHGHGWRH[SORUHSRWHQWLDOFRQĂLFWV between students’ concerns, their right to opt-out, and the implications for the mandate of higher education to use student data to make interventions at an in-dividual level. Central to this issue is the question of ÜZKREHQHāWV"ÝVHH:DWWHUV$Q\FRQVLGHUDWLRQ of the ethics around the collection, analysis, and use of student data (whether in learning analytics or in IRUPDODVVHVVPHQWVVKRXOGDOVRUHFRJQL]HWKHFRQ-testing claims and vested interests.

,QWKHEURDGHUFRQWH[WRIRQOLQHUHVHDUFK9LWDN6KLO-WRQDQG$VKNWRUDESRLQWWRYDULRXVFKDOOHQJHV UHJDUGLQJHWKLFDOUHVHDUFKSUDFWLFHVLQRQOLQHFRQWH[WV such as the increasing and persisting concerns about UHLGHQWLāFDWLRQÜUHVHDUFKHUVVWLOOVWUXJJOHWREDODQFH research ethics considerations with the use of online GDWDVHWVÝS,QWHUHVWLQJO\WKHLUāQGLQJVDOVRVKRZ that many the researchers go beyond the Belmont principles (with the main emphasis on ensuring that outcomes outweigh potential harms caused by the UHVHDUFKE\UHIHUULQJWRÜWUDQVSDUHQF\ZLWKSDU-ticipants, (2) ethical deliberation with colleagues, and FDXWLRQLQVKDULQJUHVXOWVÝSDU

There is also increasing concern balancing optimism DURXQGDUWLāFLDOLQWHOOLJHQFH$,PDFKLQHOHDUQLQJ DQGELJGDWD)RUH[DPSOHWKH([HFXWLYH2IāFHRI WKH3UHVLGHQWRIWKH86UHOHDVHGDUHSRUW0XQR] 6PLWK 3DWLOWKDWKLJKOLJKWVEHQHāWVEXWDOVR addresses concerns regarding the potential harm LQKHUHQWLQWKHXVHRIELJGDWD7KHUHSRUWUHFRJQL]HV

WKDWLIÜWKHVHWHFKQRORJLHV>DOJRULWKPLFV\VWHPV@DUH not implemented with care, they can also perpetuate, H[DFHUEDWHRUPDVNKDUPIXOGLVFULPLQDWLRQÝS It makes a number of suggestions relating to invest-ment in research into the mitigation of algorithmic discrimination, encouraging the development and use of robust and transparent algorithms, algorithmic DXGLWLQJLPSURYHPHQWVLQGDWDVFLHQFHÜĂXHQF\ÝDQG the roles of the government and private sector in setting codes of practice around data use.

6LPLODUO\WKH8.*RYHUQPHQWUHFHQWO\UHOHDVHGD Ü'DWDVFLHQFHHWKLFDOIUDPHZRUNÝ&DELQHW2IāFH SURYLGLQJJXLGDQFHRQÜHWKLFDOLVVXHVZKLFKVLWRXWVLGH WKHODZÝS7KHIUDPHZRUNH[SORUHVLVVXHVVXFKDV WKHQDWXUHRIWKHEHQHāWVRIWKHFROOHFWLRQDQDO\VLV and use of personal data; the scope and nature of intrusion; the quality of the data and the automation of the decisions relating to the collected data; the risk of negative unintended consequences; whether the data objects agreed to the collection and analysis; the nature and scope of the oversight; and the security of the collected data. The framework also proposes a Ü3ULYDF\,PSDFW$VVHVVPHQWÝUHTXLULQJGDWDVFLHQWLVWV WRFODULI\ÜWULFN\LVVXHVÝSVXFKDVUHYLHZLQJWKH H[WHQWWRZKLFKWKHEHQHāWVRIWKHSURMHFWRXWZHLJK the risks to privacy and negative unintended con-VHTXHQFHVVWHSVXQGHUWDNHQWRPLQLPL]HULVNVDQG HQVXUHFRUUHFWLQWHUSUHWDWLRQDQGWKHH[WHQWWRZKLFK the opinions of the data objects/public regarding the SURMHFWZHUHFRQVLGHUHGVHH&DELQHW2IāFH ,QWKHFRQWH[WRIWKHDOJRULWKPLFWXUQLQKLJKHUHGX-cation, and the increasing blurring of the boundaries between broader developments in data and neurosci-ence, we need a critical approach to considering the HWKLFDOLPSOLFDWLRQVRIOHDUQLQJDQDO\WLFVDVZHāQGRXU ZD\WKURXJKWKHP\WKPHVVDQGPHWKRGV=LHZLW] RIVWXGHQWGDWD)RUH[DPSOH:LOOLDPVRQD FRQVLGHUVÜHGXFDWLRQDOGDWDVFLHQFHDVDbiopolitical

strategy focused on the evaluation and management

of the corporeal, emotional and embrained lives of FKLOGUHQÝSHPSKDVLVDGGHG$VVXFKZHKDYH to consider the basis and scope of authority of educa-WLRQDOGDWDVFLHQWLVWVZKRKDYHÜLQFUHDVLQJOHJLWLPDWH authority to produce systems of knowledge about FKLOGUHQDQGWRGHāQHWKHPDVVXEMHFWVDQGREMHFWV RILQWHUYHQWLRQÝ:LOOLDPVRQDS/HDUQLQJ analytics in future will be essentially based on and driven by algorithms and machine learning and we WKHUHIRUHKDYHWRFRQVLGHUKRZDOJRULWKPVÜUHLQIRUFH maintain, or even reshape visions of the social world, NQRZOHGJHDQGHQFRXQWHUVZLWKLQIRUPDWLRQÝ:LO-OLDPVRQES$FFRXQWDELOLW\WUDQVSDUHQF\DQG regulatory frameworks will be essential elements in the frameworks ensuring ethical learning analytics VHH3ULQVORR7DQHMD

While this chapter maps the progress in considering the ethical implications of the collection, analysis, and use of student data, it is clear that the poten-tial for harm will not be addressed without further consideration of institutional processes to ensure accountability and transparency. As Willis, Slade, DQG3ULQVORRLQGLFDWHOHDUQLQJDQDO\WLFVRIWHQ falls outside the processes and oversight provided by institutional review boards (IRBs). It is not clear at this stage by whom and how the ethical implications of learning analytics will be assured.

Since the emergence of learning analytics in 2011, WKHāHOGKDVQRWRQO\PDWXUHGEXWDOVREHFRPHPRUH nuanced in increasingly considering the fears and re-alities of ethical implications in the collection, analysis, and use of student data. In this chapter, we provide an overview of how our own thinking has developed DORQJVLGHEURDGHUGHYHORSPHQWVLQWKHāHOG$JDLQVW a backdrop of technological advances and increasing concerns around pervasive surveillance, and a growing

consensus that the future of higher education will be digital, distributed, and data-driven, this chapter maps how far the discourses surrounding the ethical implications of learning analytics have come, as well as some of the future considerations.

Each of the frameworks, code of practices, and conceptual mappings of the ethical implications in learning analytics discussed adds a further layer and a richer understanding of how we may move toward XVLQJVWXGHQWGDWDSUR[LHVWRLQFUHDVHWKHHIIHFWLYH-ness and appropriateXVLQJVWXGHQWGDWDSUR[LHVWRLQFUHDVHWKHHIIHFWLYH-ness of teaching, learning, and student support strategies in economically viable and ethical ways. The practical implementation of that understanding remains largely incomplete, but still wholly pertinent.

We would like to acknowledge the comments, critical inputs, and support received from the editorial team and specially the reviewers of this chapter.

(IN)CONCLUSIONS

ACKNOWLEDGEMENTS

REFERENCES

&DELQHW2IāFH0D\'DWDVFLHQFHHWKLFDOIUDPHZRUNKWWSVZZZJRYXNJRYHUQPHQWXSORDGV V\VWHPXSORDGVDWWDFKPHQWBGDWDāOH'DWDBVFLHQFHBHWKLFVBIUDPHZRUNBYBIRUBSXEOLFD-WLRQBBBSGI &DPSEHOO-3'H%ORLV3% 2EOLQJHU'*-XO\$XJXVW$FDGHPLFDQDO\WLFV$QHZWRRODQHZHUD

EDUCAUSE Review. http://net.educause.edu/ir/library/pdf/erm0742.pdf

&RXOGU\16HSWHPEHU7KHSULFHRIFRQQHFWLRQÜ6XUYHLOODQFHFDSLWDOLVPÝ>:HEORJSRVW@KWWSV WKHFRQYHUVDWLRQFRPWKHSULFHRIFRQQHFWLRQVXUYHLOODQFHFDSLWDOLVP 'DZVRQ6*DÇHYLİ' 5RJHUV76WXGHQWUHWHQWLRQDQGOHDUQLQJDQDO\WLFV$VQDSVKRWRI$XVWUDOLDQ SUDFWLFHVDQGDIUDPHZRUNIRUDGYDQFHPHQW$XVWUDOLDQ*RYHUQPHQWKWWSKHDQDO\WLFVFRPZSFRQ-WHQWXSORDGV63BB'DZVRQB5HSRUWBSGI 'UDFKVOHU+ *UHOOHU:7KHSXOVHRIOHDUQLQJDQDO\WLFVXQGHUVWDQGLQJVDQGH[SHFWDWLRQVIURPWKH stakeholders. Proceedings of the 2nd _{International Conference on Learning Analytics and Knowledge (LAK ’12),}

$SULO×0D\9DQFRXYHU%&&DQDGDSS×1HZ<RUN$&0

'UDFKVOHU+ *UHOOHU:$SULO3ULYDF\DQGDQDO\WLFV,WÚVD'(/,&$7(LVVXHØDFKHFNOLVWIRUWUXVWHG learning analytics. Proceedings of the 6th _{International Conference on Learning Analytics and Knowledge}

/$.Ú×$SULO(GLQEXUJK8.SS×1HZ<RUN$&0 (QJHOIULHW$0DQGHUYHOG- -HXQLQN(/HDUQLQJDQDO\WLFVRQGHUGH:HWEHVFKHUPLQJSHUVRRQV- JHJHYHQV685)QHWKWWSVZZZVXUIQOELQDULHVFRQWHQWDVVHWVVXUIQONHQQLVEDQNVXUIBOHDUQ-ing-analytics-onder-de-wet-wpb.pdf )DLU7HVWQG-XVWVD\QRWRWKHWHVWKWWSZZZIDLUWHVWRUJJHWLQYROYHGRSWLQJRXW *DÇHYLİ''DZVRQ6 -RYDQRYLİ-(WKLFVDQGSULYDF\DVHQDEOHUVRIOHDUQLQJDQDO\WLFVJournal of

Learning Analytics, 3×KWWSG[GRLRUJMOD

0XQR]&6PLWK0 3DWLO'-%LJGDWD$UHSRUWRQDOJRULWKPLFV\VWHPVRSSRUWXQLW\DQGFLYLO

ULJKWV([HFXWLYH2IāFHRIWKH3UHVLGHQW86$KWWSVZZZZKLWHKRXVHJRYVLWHVGHIDXOWāOHVPLFUR-HANDBOOK OF LEARNING ANALYTICS PG 56 VLWHVRVWSBBGDWDBGLVFULPLQDWLRQSGI 10&1HZ0HGLD&RQVRUWLXP7KH10&+RUL]RQ5HSRUWKWWSZZZHGXFDXVHHGX5H-VRXUFHV+RUL]RQ5HSRUW 10&1HZ0HGLD&RQVRUWLXP7KH10&+RUL]RQ5HSRUWKWWSZZZQPFRUJSXEOLFDWLRQ QPFKRUL]RQUHSRUWKLJKHUHGXFDWLRQHGLWLRQ 2Ú%ULHQ$6HSWHPEHU3UHGLFWDEO\LUUDWLRQDO$FRQYHUVDWLRQZLWKEHVWVHOOLQJDXWKRU'DQ$ULH- O\>:HEORJSRVW@KWWSZZZOHDUQLQJāUVWRUJSUHGLFWDEO\LUUDWLRQDOFRQYHUVDWLRQEHVWVHOOLQJDX-thor-dan-ariely 2SHQ8QLYHUVLW\3ROLF\RQHWKLFDOXVHRIVWXGHQWGDWDIRUOHDUQLQJDQDO\WLFVKWWSZZZRSHQ DFXNVWXGHQWVFKDUWHUVLWHVZZZRSHQDFXNVWXGHQWVFKDUWHUāOHVāOHVHFPVZHEFRQWHQWHWKL-cal-use-of-student-data-policy.pdf 3ULQVORR36HSWHPEHU)OHHLQJIURP)UDQNHQVWHLQDQGPHHWLQJ.DINDRQWKHZD\$OJRULWKPLFGH- FLVLRQPDNLQJLQKLJKHUHGXFDWLRQ3UHVHQWDWLRQDW18,*DOZD\KWWSZZZVOLGHVKDUHQHWSULQVSIHHO-ing-from-frankenstein-and-meeting-kafka-on-the-way-algorithmic-decisionmaking-in-higher-education 3ULQVORR36ODGH6 *DOSLQ)/HDUQLQJDQDO\WLFV&KDOOHQJHVSDUDGR[HVDQGRSSRUWXQLWLHVIRUPHJD

open distance learning institutions. Proceedings of the 2nd _{International Conference on Learning Analytics}

and Knowledge/$.Ú$SULO×0D\9DQFRXYHU%&&DQDGDSS×1HZ<RUN$&0

3ULQVORR3 6ODGH6$QHYDOXDWLRQRISROLF\IUDPHZRUNVIRUDGGUHVVLQJHWKLFDOFRQVLGHUDWLRQVLQ learning analytics. Proceedings of the 3rd _{International Conference on Learning Analytics and Knowledge}

/$.Ú×$SULO/HXYHQ%HOJLXPSS×1HZ<RUN$&0

3ULQVORR3 6ODGH6D(GXFDWLRQDOWULDJHLQKLJKHURQOLQHHGXFDWLRQ:DONLQJDPRUDOWLJKWURSHInter-national Review of Research in Open Distributed Learning ,552'/, 14×KWWSZZZLUURGORUJ

LQGH[SKSLUURGODUWLFOHYLHZ

3ULQVORR3 6ODGH6E6WXGHQWSULYDF\DQGLQVWLWXWLRQDODFFRXQWDELOLW\LQDQDJHRIVXUYHLOODQFH,Q0 (0HQRQ'*7HUNOD 3*LEEV(GVUsing data to improve higher education: Research, policy and

prac-ticeSS×*OREDO3HUVSHFWLYHVRQ+LJKHU(GXFDWLRQ5RWWHUGDP6HQVH3XEOLVKHUV

3ULQVORR3 6ODGH66WXGHQWSULYDF\VHOIPDQDJHPHQW,PSOLFDWLRQVIRUOHDUQLQJDQDO\WLFVProceed-ings of the 5th International Conference on Learning Analytics and Knowledge/$.Ú×0DUFK

3RXJKNHHSVLH1<86$SS×1HZ<RUN$&0

3ULQVORR3 6ODGH6D6WXGHQWYXOQHUDELOLW\DJHQF\DQGOHDUQLQJDQDO\WLFV$QH[SORUDWLRQJournal of

Learning Analytics, 3×

3ULQVORR3 6ODGH6E+HUHEHGUDJRQV0DSSLQJVWXGHQWUHVSRQVLELOLW\LQOHDUQLQJDQDO\WLFV,Q0 $QGHUVRQ &*DYDQ(GVDeveloping effective educational experiences through learning analytics (pp. ×+HUVKH\3$,*,*OREDO 5XJJLHUR'0D\:KDWPHWULFVGRQÚWWHOOXVDERXWWKHZD\VWXGHQWVOHDUQThe Conversation. http:// theconversation.com/what-metrics-dont-tell-us-about-the-way-students-learn-59271 6FODWHU10DUFK(IIHFWLYHOHDUQLQJDQDO\WLFV$WD[RQRP\RIHWKLFDOOHJDODQGORJLVWLFDOLVVXHVLQ OHDUQLQJDQDO\WLFVY-,6&KWWSVDQDO\WLFVMLVFLQYROYHRUJZSDWD[RQRP\RIHWKLFDOOH-gal-and-logistical-issues-of-learning-analytics-v1-0/ 6FODWHU13HDVJRRG$ 0XOODQ-/HDUQLQJDQDO\WLFVLQKLJKHUHGXFDWLRQ$UHYLHZRI8.DQGLQWHU-QDWLRQDOSUDFWLFH-,6&KWWSVZZZMLVFDFXNUHSRUWVOHDUQLQJDQDO\WLFVLQKLJKHUHGXFDWLRQ 6KDFNORFN;)URPEULFNVWRFOLFNV7KHSRWHQWLDORIGDWDDQGDQDO\WLFVLQKLJKHUHGXFDWLRQ+LJKHU (GXFDWLRQ&RPPLVVLRQKWWSZZZSROLF\FRQQHFWRUJXNKHFVLWHVVLWHBKHFāOHVUHSRUWāHOGUH-portdownload/frombrickstoclicks-hecreportforweb.pdf 6LHPHQV*$SULO5HĂHFWLQJRQOHDUQLQJDQDO\WLFVDQG6R/$5>:HEORJSRVW@KWWSZZZHOHDUQV-SDFHRUJEORJUHĂHFWLQJRQOHDUQLQJDQDO\WLFVDQGVRODU

6LHPHQV* %DNHU5$SULO/HDUQLQJDQDO\WLFVDQGHGXFDWLRQDOGDWDPLQLQJ7RZDUGVFRPPXQLFDWLRQ and collaboration. Proceedings of the 2nd _{International Conference on Learning Analytics and Knowledge}

/$.Ú$SULO×0D\9DQFRXYHU%&&DQDGDSS×1HZ<RUN$&0

6ODGH6 *DOSLQ)/HDUQLQJDQDO\WLFVDQGKLJKHUHGXFDWLRQ(WKLFDOSHUVSHFWLYHVZRUNVKRSPro-ceedings of the 2nd _{International Conference on Learning Analytics and Knowledge}/$.Ú$SULO×0D\

9DQFRXYHU%&&DQDGDSS×1HZ<RUN$&0

6ODGH6 3ULQVORR3/HDUQLQJDQDO\WLFV(WKLFDOLVVXHVDQGGLOHPPDVAmerican Behavioral Scientist,

57×

6ODGH6 3ULQVORR36WXGHQWSHUVSHFWLYHVRQWKHXVHRIWKHLUGDWD%HWZHHQLQWUXVLRQVXUYHLOODQFH and care. Proceedings of the European Distance and E-Learning Network 2014 Research Workshop('(1 ×2FWREHU2[IRUG8.SS×

6PLWK*-6XUYHLOODQFHGDWDDQGHPERGLPHQW2QWKHZRUNRIEHLQJZDWFKHGBody & Society× GRL; 6WUDXVV9D-DQXDU\86(GXFDWLRQ'HSDUWPHQWWKUHDWHQVWRVDQFWLRQVWDWHVRYHUWHVWRSWRXWV7KH :DVKLQJWRQ3RVWKWWSVZZZZDVKLQJWRQSRVWFRPQHZVDQVZHUVKHHWZSXVHGXFD-tion-department-threatens-to-sanction-states-over-test-opt-outs/ 6WUDXVV9E0D\Ü%LJGDWDÝZDVVXSSRVHGWRā[HGXFDWLRQ,WGLGQÚW,WÚVWLPHIRUÜVPDOOGDWDÝ7KH :DVKLQJWRQ3RVWKWWSVZZZZDVKLQJWRQSRVWFRPQHZVDQVZHUVKHHWZSELJGDWD ZDVVXSSRVHGWRā[HGXFDWLRQLWGLGQWLWVWLPHIRUVPDOOGDWD 7DQHMD+6HSWHPEHU7KHQHHGIRUDOJRULWKPLFDFFRXQWDELOLW\TechCrunch. https://techcrunch. FRPWKHQHHGIRUDOJRULWKPLFDFFRXQWDELOLW\ YDQ%DUQHYHOG$$UQROG. &DPSEHOO-$QDO\WLFVLQKLJKHUHGXFDWLRQ(VWDEOLVKLQJDFRPPRQODQ-guage. EDUCAUSE Learning Initiative, 1×

9LWDN-6KLOWRQ. $VKNWRUDE=%H\RQGWKH%HOPRQWSULQFLSOHV(WKLFDOFKDOOHQJHVSUDFWLFHV and beliefs in the online data research community. Proceedings of the 19th _{ACM Conference on Computer}

Supported Cooperative Work & Social Computing&6&:Ú)HEUXDU\×0DUFK6DQ)UDQFLVFR&$

86$1HZ<RUN$&0KWWSVWHUSFRQQHFWXPGHGXaNVKLOWRQSGI9LWDNHWDO&6&:SUHSULQWSGI

:DWWHUV$0D\,GHQWLW\SRZHUDQGHGXFDWLRQÚVDOJRULWKPV>:HEORJSRVW@KWWSKDFNHGXFDWLRQ FRPLGHQWLW\SRZHUDOJRULWKPV

Welsh, S., & McKinney, S. (2015). Clearing the fog: A learning analytics code of practice. In T. Reiners et al. (GV*OREDOO\FRQQHFWHGGLJLWDOO\HQDEOHGProceedings of the 32nd _{Annual Conference of the Australasian}

Society for Computers in Learning in Tertiary Education$6&,/,7(1RYHPEHU×'HFHPEHU

3HUWK:HVWHUQ$XVWUDOLDSS×KWWSUHVHDUFKPRRGOHQHW

:LOOLDPVRQ%D&RGLQJWKHELRGLJLWDOFKLOG7KHELRSROLWLFVDQGSHGDJRJLFVWUDWHJLHVRIHGXFDWLRQDOGDWD science. Pedagogy, Culture & Society, 24×GRL

:LOOLDPVRQ%E&RPSXWLQJEUDLQV/HDUQLQJDOJRULWKPVDQGQHXURFRPSXWDWLRQLQWKHVPDUWFLW\Infor-mation, Communication & Society, 20×GRL;

:LOOLV-6ODGH6 3ULQVORR3(WKLFDORYHUVLJKWRIVWXGHQWGDWDLQOHDUQLQJDQDO\WLFV$W\SRORJ\ derived from a cross-continental, cross-institutional perspective. Educational Technology Research and 'HYHORSPHQWGRLV

=KDQJ60D\6FLHQWLVWVDUHMXVWDVFRQIXVHGDERXWWKHHWKLFVRIELJGDWDUHVHDUFKDV\RX:LUHG KWWSZZZZLUHGFRPVFLHQWLVWVMXVWFRQIXVHGHWKLFVELJGDWDUHVHDUFK

=LHZLW]0*RYHUQLQJDOJRULWKPV0\WKPHVVDQGPHWKRGVScience, Technology & Human Values, 41(1), ×