Measuring impact of e-Learning on Quality and learning outcomes – a
proposal for a pHD project
Authors: Rita Falcão, Alfredo Soeiro
Affiliation: Universidade do Porto
Abstract
In Higher and Continuing Education, the use of eLearning methodologies is becoming a common resource. At this point, we have probably passed a first stage of eLearning implementation - experimentation. After a first stage where pilot projects were implemented and strategies were defined, we come to a phase where eLearning implementation is more mature. It should now be possible to find consistent examples of the use of eLearning in Engineering Education, with several years of application. It should be possible, at this time, to evaluate the impact of eLearning methodologies on the Learning Outcomes. The work here proposed intends to address the problem of measuring the impact of eLearning on Education, through the analysis of Learning Outcomes.
The scope of the study includes case-studies in Higher Education, pre and graduate courses. Also, cases of the use of eLearning in Continuing Education will be addressed. Authors intend to collect case-studies of courses that have been using eLearning technologies, in different contexts and Institutions. Data collected should include information about the implementation and Learning Outcomes of the course, before and after eLearning. Using a considerable amount of data, this study aims to identify a potential trend of alteration of results of Learning Outcomes. If possible, this study will try to find a relation between the increment of Learning Outcomes and the implementation of eLearning.
Keywords: evaluation, Learning Outcomes, quality
Introduction
E-learning is a relatively new field of research. Even though the use of technologies for learning has been implemented for several decades, the introduction of the Internet has promoted a considerable shift and innovation. Also, e-Learning is by its nature a multidisciplinary subject and, consequently, very complex. For these reasons, even though learning feeds from previous research in Technology and Education (among others), e-learning, as a research subject by itself, is still in a very early but exciting stage of development.
In the past years, e-learning strategies have been implemented in Institutions of Higher Education and others. Although these implementations have different objectives and follow different strategies, the importance of the use of e-learning technologies to improve the Quality of Education has been widely accepted. The e-learning Programme of the European Commission, promotes the “effective integration of Information and Communication Technologies (ICT) in education and training systems in Europe (2004 – 2006)”, and consolidates this perspective. As stated by the European Commission, ICT, properly used, should contribute to the quality of education and training and to Europe’s move to a knowledge-based society.
When analyzing the proceedings of most e-learning or e-learning related conferences, one finds that another trend is becoming frequent related to Quality and Quality Assurance of e-learning. The implementation of e-learning can be very demanding for an institution, since it introduces changes in many different processes. For this reason, concerns about the effectiveness of e-learning focus mainly on organizational and procedural factors, justifying the increased interest in Quality Assurance. However, quality issues should also address the impact of e-learning on learning results. When assessing the quality of e-learning or any education related process, the learning outcomes should be considered as an important indicator. Even though the implementation of e-learning can be very diverse, technically and pedagogically, an effort should be done to estimate if these technologies are affecting the results in a positive or negative way.
This proposal seeks to address these issues of quality in e-learning, and in particular the impact on the Learning Outcomes. What is the real added value of e-learning to the Education Process, and is this value dependent of the implementation strategies.
Objectives of the study
This study has two general aims related to the quality in e-Learning, in a Micro and Macro perspective:
• To measure the impact of the application of eLearning in Education, through the analysis of the Learning Outcomes.
• To identify success and unsuccess factors for the use of eLearning in Education of Engineering, based on the selected case-studies.
More specifically, the objectives are:
• To collect and analyze European and Worldwide initiatives and methodologies on the quality of e-learning, that address the issue of Learning Outcomes.
• To define and test a methodology to evaluate the general quality of e-learning trough the analysis of Learning Outcomes
• To collect and compile case-studies of e-learning at different institutions, including information about strategies and Learning Outcomes
• To identify factors of success or failure in e-learning, based on the analysis of provided case studies
• To develop indicators of quality in e-learning supported by case studies derived from best practice and taking account of definitions and practices in different countries
Detailed description
This research project intends to evaluate the real added value of e-learning to the Educational Process, from the perspective of the impact on the Learning Outcomes. To achieve this goal, a methodology of evaluation will be developed, based in case-study analysis, pre and post implementation and bibliographic analysis. The first stage of the study will consist on collecting information on existing methodologies for evaluating e-Learning, in particular from accreditation agencies and institutions. Based on the information collected, a methodology will be developed that will address the particular needs of this study.
On this stage, a structure for the case studies will be defined, including the type and amount of data to collect. Also, a matrix of existing quality indicators will be compiled and used as a complement to the evaluation of quality. Case studies of courses should include information about the Learning Outcomes before and after the e-Learning implementation, when possible. Evaluation will be based on results (outcomes) of similar courses and user assessment.
The second stage of this project consists on selecting case studies based on previously defined criteria and structure, and statistically analyze them. The case studies will focus on courses of cooperating Institutions that are involved in e-learning. This is a critical stage of the study and the longest. The statistic of analysis intended for this stage of the project is based on large numbers. However, the selection criteria for case studies impose some restrictions, increasing the difficulty of finding eligible courses and slowing down the pace of the project. The case study analysis, focusing mainly on the statistic overview of Learning Outcomes will try to measure the evolution after the e-Learning implementation.
The following stage consists on a deeper analysis. The previous results will be confronted with different context related factors. This stage aims to identify trends for failure or success factors in e-learning implementations. It is expected that the conclusions of the first stages of the project, will provide indicators of quality that support e-learning implementation. Also, these conclusions will feed a Guide for Good Practice on e-learning implementation, from the Learning Outcomes perspective.
For each stage of the research project, the following outputs are expected: Stage I - Preparation
• Report on State of the Art of e-learning Quality and Evaluation: Compilation of existing initiatives of quality evaluation in e-Learning, in particular the ones addressing Learning Outcomes. The goal of this activity is to identify trends and tools used in e-learning evaluation. Also, it will help to identify Institutions active in e-learning that can be asked to collaborate. This report will support the future decisions of the study.
• Evaluation methodology: Based on the previous outcome, a methodology will be defined to evaluate how e-learning is affecting the Learning Outcomes of students and trainees.
• Case-study Structure: The structure of the case-study that will be used for this project is very important and should reflect the specific needs of this study. However, it will benefit from the research done for the State of the Art report, since case-study analysis is the most common evaluation method for Quality in e-learning.
• Matrix of quality indicators: The third stage of the project consists on an analysis of the e-learning implementation based on context. Among other factors, a possible relation between existing quality indicators for e-learning and the impact of e-learning on the Outcomes should be addressed.
Stage II – Statistic Analysis
• Database of Institutions with e-Learning initiatives: This is a critical stage of the study, since the value of the statistical analysis will strongly depend on the amount and quality of the information gathered. Also, the diversity of institutions and contexts in which e-learning is being used will enrich the final conclusions and widen the applicability of results.
• Case-studies and statistical analysis of data collected: This analysis will look at the case-studies as a black-box, not taking in consideration the e-learning processes or methods used on the on-line courses. The main goal is to measure the alteration of Learning Outcomes on courses that moved from traditional teaching to e-learning.
Stage III – Context analysis
• Context analysis of data collected: From the previous stage, a selection of case studies will be analyzed more deeply, based on the results achieved. For this stage, is important to analyze case studies that represent diverse contexts of Education, a variety of e-learning implementations and also that presented different behaviors in relation with the Learning Outcomes.
• Success and failure factors: Based on the statistical and context analysis, an effort will be made to identify key factors in e-Learning that may affect the Learning Outcomes, in a positive or negative way.
• Good Practice Guide: this output will gather in a structured document, the findings of the previous activities. It is expected that this guide will help the stakeholders in future decisions concerning e-learning implementations
Finally, the results and conclusions of the study should be disseminated at different levels, to promote good practice.
This study intends to establish an innovative approach to e-Learning evaluation and quality. Focusing on the impact of e-Learning on the Learning Outcomes, at a larger scale, it intends to identify trends and success/failure factors in existing e-learning implementations. The major innovation consists on the statistical analysis of a large number of e-learning courses, ignoring at this stage, the methodologies and strategies used during the course.
Results and conclusions
At this point, results of several case studies from Universidade do Porto were collected. A preliminary analysis of results predicts a positive tendency of change after the use of e-Learning. In almost every case analysed, there is a tendency for increasing values of the general indicators of the course. The rates Passed/Registered, Passed/Evaluated and Evaluated/Registered increased after the introduction of e-Learning. Also, analysing the charts of grades distribution, we find that there is a slight positive deviation of the lines. Below, tables and charts of three case studies are presented.
Even though these results seem to indicate a positive change after e-Learning introduction, it is to soon to assume this conclusion. This is only one superficial analysis of course results, from a single institution. Also, teaching and learning are very complex processes, making very dangerous to relate this positive change with a single factor. A deeper and broader study is needed, including a context analysis.
Table 1 - Results obtained for a course in statistics, 1st Semester
Statistics (Y1/ S1) Before After
Registered 258 268 Evaluated 125 163 Passed 95 128 Pass/Register 37.0% 49.2% Pass/Evaluate 75.1% 78.5% Evaluate/Register 48.5% 62.0%
Figure 1 - General indicators for the course Statistics. P/R - passed over registered; P/E - passed over evaluated; E/R - evaluated over registered
Figure 2 - Average grade distribution for the course Statistics before and after e-Learning
Table 2 - Results obtained for a course in Economics, 7th Semester
Economics (Y4/1S) Before After
Registered 297 371 Evaluated 200 252 Passed 180 229 Pass/Register 60.6% 63% Pass/Evaluate 90.0% 91% Evaluate/Register 67.3% 69%
Figure 3 - General indicators for the course Economics. P/R - passed over registered; P/E - passed over evaluated; E/R - evaluated over registered
Figure 4 - Average grade distribution for the course Economics before and after e-Learning Table 3 - Results obtained for a course in English, 1st Semester
English (Y1/S1) Before After
Registered 187 76 Evaluated 125 50 Passed 85 32 Pass/Register 44% 42% Pass/Evaluate 67% 63% Evaluate/Register 66% 67%
Figure 5 - General indicators for the course English. P/R - passed over registered; P/E - passed over evaluated; E/R - evaluated over registered
Figure 6 - Average grade distribution for the course English before and after e-Learning
References
• CONOLE, Gráinne, OLIVER, Martin. Contemporary perspectives in E-learning Research: themes, methods and impact on practice. Oxon,UK: Routledge, 2007. • REIS LIMA, Jorge, CAPITÃO, Zélia. E-learning e e-conteúdos: Aplicação das teorias
e modernas de ensino e aprendizagem à organização e estruturação de e-cursos. Portugal: Centro Atlântico, 2003, Colecção: Sociedade de Informação
• KOPER, R., 2002, Improving Learning Through Technologies: Opportunities for All, www.prometeus.org
• Machado, J., E-learning em Portugal. 1 ed. Sociedade Digital. 2001, Lisboa: FCA. • KOMAROMi, László, MEDINA, José Collado, VAN DEN BRANDEN, Jef, SOEIRO,
Alfredo, PERRY, Brian, Quality guide of open and distance learning – Procedures and practices, SOCRATES ODL PROJECT (56591-CP-2-1999-1-HU-ODL-ODL), http://www.szamalk.hu/projects/P010/English.htm
• CARVALHO, C. V. (2003) Defining an evaluation methodology for blended learning in Higher Education, Evaluating e-learning – Galecia Project
