Lessons Learned in Evaluating e-Learning Products:
A Beginning Evaluator Checklist
Krista Galyen, University of Missouri,[email protected]
Camille Dickson-Deane, University of Missouri,[email protected] Abstract
The evaluation of e-learning products requires a level of preparation,
forethought, and skill to which most beginning evaluators are not accustomed. Like any other project management process, planning for an evaluation takes more than just technique and available skills; it requires understanding the client, their intentions, their political environment, as well as learning how to modify techniques and still deliver the required report. This presentation discusses lessons learned from an effectiveness evaluation project in order to better inform and prepare beginning evaluators of these aspects. Aspects such as proximity of team members, communication, political environment,
insufficient stakeholder management, and the availability of the specified
participants will be described. In addition, a beginning evaluator checklist will be shown which assists the evaluator in thinking about potential variables and processes.
INTRODUCTION
The process of conducting an evaluation requires lots of planning, management and
controlling of components. These components consist of human and technical resources as well as procedures which are used to provide a framework for conducting the evaluation. The different components need to be organized into some structure in order for the
evaluation to be successful. New evaluators would normally follow some existing procedure in an effort to copy success, but the components that existed for the procedure may not be the same or even similar to the [new] evaluators project. E-learning evaluation is even more complicated in that the e-learning product may have different characteristics or even, be based on different definitions of e-learning. Allen & Seaman (2007) defines four types of courses where at one extreme traditional courses have no online content that is delivered and the other extreme has no face to face instruction. E-learning can fall into either category of face to face or no face to face instruction as per its definition of being learning provided through some technological media. The media in this instance can span anywhere from being a CD-Rom to being strictly provided via the Internet (Nicols, 2008). The
medium of delivery is as important to the success of the evaluation as its evaluators; each piece of the puzzle must be carefully handled in order to produce a reasonable evaluation report.
THE PROJECT The product
OneTree Learning (OTL) supports students learning English as a second language (ESL). The company provides not only English learning services, but also acts as a guide to help
students become more prepared for academic classrooms. Their product, Language Arts for ESOL, is a CALP- (Cognitive Academic Language Proficiency) based program that teaches a core curriculum of language arts to English language learners. OTL's Language Arts for ESOL can be used either as a dedicated ESL curriculum, or as a supplementary program to assist ESL students within existing mainstream classes. No specialized ESL training is
necessary for non-ESL teachers to use the program. The program teaches the basics of language arts and provides an interactive environment of stories and games in which students practice speaking, listening, reading and writing. Native language support helps students follow the lessons.
Client/stakeholder and participants
The primary stakeholders were the owners/creators of the OneTree Learning product and the users/learners, represented by Mr. Christian Sykes and the participants/audience, respectively. The E-learning program was created for learners who are studying English as a Second Language (ESL) and was primarily gauged for Grade 3 learners but has since then been also used with adults. This also means that institutions where the product is or will be implemented are also primary stakeholders. For this evaluation, there were five
participants, two Latin American adult learners and three teachers/experts from varying elements associated with ESL.
Rationale and Research
Language Arts for ESOL is attempting to fill the need for ESL learners to gain competence in academic language, in particular Language Arts. Collier (1989) stated that rather than providing ESL instruction as a separate and distinct activity, it is more beneficial to integrate the ESL instruction with the academic content. Met (1991) also found that students learning their second language (L2) was more beneficial when it was presented within meaningful academic lessons rather than de-contextualized. In other words, it is better to assist students' language in reading and writing while they are actively engaged in the act of reading and writing rather than have a session merely discussing and practicing the
vocabulary associated with this area. Language Arts for ESOL is based upon these findings on how to most appropriately facilitate the learning of academic language skills.
The findings and recommendations stemming from this evaluation are intended to help the clients make the following decisions:
1. How should the product be implemented for best success?
2. Should any modifications be made to keep the package attractive as opposed to other similar packages on the market?
3. Should any modifications be made to make the product more effective for the target audience? For secondary audiences?
The evaluation team
The evaluation team comprised of four graduate student pursuing an e-learning evaluation course with the instructor posing as the supervisor. The team was dispersed, with two team members in one town and another two in another town; an approximate distance of 700 miles between both sub-teams. There was a varying levels of skills amongst the team members all of which did not include e-learning evaluation.
Methodology
The methods used to collect data included pre- and post-test for learners, focus group interviews with learners, and expert reviews. The data collected from the pre- and post-test was compared based on the length of time the software was used. The focus group data was summarized and coded.
There were a number of limitations including locating the appropriate participants for the evaluation, time constraints in getting the software sufficiently installed at the evaluating institutions and restrictions in collecting the data due to inclement weather.
• Communication
The team used Skype to have simultaneous conversations amongst themselves as well as with the client and primary stakeholder and GoogleDocs to collaborate on the evaluation report. These tools were ideal for the groups interaction but not everyone had the skill-set and level to use the tools to complete the report. This in essence resulted in additional time being taken to resolve such issues of missed
communications.
• Team formation and Motivation
The challenge of knowing each others' skill sets as well as the ability to be motivated to learn more/additional skills will require a certain type of individual characteristics. As difficult as this may seem, members of the evaluative team should move quickly from a group formation into a team formation in order to be aware of the team's strengths and weaknesses, creating solutions for all and improving the interactions within the team. This team had deficiencies in some skills that were required to complete their evaluation as best as possible. This did deter the team to some extent and could have improved the outcome of the evaluation report.
• Organization and management skills
Teams that are created for short periods initially operate in a self-directing mode; however, over time a leader emerges. In this case there were many roles that were needed to ensure that the project not only went smoothly, but that it also allowed for the team to operate at its maximum efficiency. The team had many missed/extended deadlines, loss of documents through varying versioning methods and difficulty in getting timely response from the key stakeholder. As such, this team also needed a person in charge of stakeholder management, time management and document management.
◦ Stakeholder management: The client needed communication with only one person and not several; this made for ease of understanding and quicker resolutions to any issues.
◦ Time management: Knowing the time allotted for the entire project makes it easier to manage the many parts and ensure that deadlines are maintained. ◦ Document management: This is one of the most important challenges this
team encountered. The versioning and formatting of the document/reports should be handled by a single individual. Even if a guide is provided for the versioning of the document, some team members may forget and having a person dedicated to such an effort can solve many issues.
• Participant management
Seeking and managing participants and the requirements associated with setting up the testing environment needs much more than just phone calls to decide how many participants and what technology/tools would be needed. The team dealt with public schools which added policy awareness to the table of decisions. As the team
progressed with the project, the lack of awareness of the policies that are attached with introducing and using technological tools/equipment in a public environment became evident. Teachers had misconceptions regarding the level of district-level "red tape" which prevented the approval of installing software. Whereas some thought approval could come within a day or two, district-level approval never arrived. As such, a deficient workaround was concocted. Appropriate bandwidth for internet use to download the software was a major challenge with this project. Another challenge was getting the level of commitment from the teachers who were interested in using the software in their classrooms. The teachers were very much interested but were seemingly overwhelmed when it came to actually implementing the e-learning
program. Hence when it came to the evaluation process, participants were not either prepared or present. Some of the teachers did not implement the software with their students or had them use it only once or twice total.
LESSONS LEARNED
The primary premise for the evaluative procedure is not the perceived outcomes but moreso what the evaluation report will suggest about the product (Alkin, 1980; Shulha & Cousins, 1997). The factors affecting an evaluative process (i.e. stakeholders interest, timeliness, user characteristics, etc...) color the quality of evaluation report and as such suggest that the method used should reflect the perceived result. The perceived result in this case was based on the evaluation of the effectiveness of the OneTree Learning Language Arts for ESOL program. Mr. Skykes, having tested the product with an audience for which it was not intended - adults - wanted the team to provide a report which demonstrated the results he had received informally. So in order to satisfy our client/stakeholder the team made changes. These changes and ultimately, the resulting report showed that there were a number of things that the team could have prepared to make for a successful evaluation project (see the Table 1 below). This checklist not only includes the identification of technological and human resources, but also identifies simple documents which will assist the evaluators with the progress of the evaluation.
Table 1 Checklist for a new evaluator
CHECKLIST
□
Access to collaborative technology/tools?□
Access to a text editor i.e. Google Docs, Zoho, etc..□
Audio/Video communication i.e. Skype, skype recorder□
A common area for sharing documentation□
A document explicitly identifying team members skills□
Identify a team leader [and a lead for stakeholder management]□
Identify a lead for time management□
Identify a lead for document management□
Identify a lead for participant management□
Required Documents□
A number of alternative approaches to your evaluation methodology□
Alternative sources of evaluation participants□
Approval documents allowing for▪ software/hardware use at public institutions (if required) ▪ installation of any type of media
N.B. Documents should state the period for which the approval is granted
□
A strict timeline for completing the evaluationN. B. Timeline should incorporate approvals required for the evaluation process (see above)
□
Methodology/Implementation Notes□
Troubleshooting - For general skills, team members who may have to install software should know how to troubleshoot installation challenges□
Implementer commitment - obtain a commitment from the person who you will be using to implement your solution for evaluation. This commitment should have a frequency and length of use clause□
Implementer meetings - schedule specifics times to meet him/her during the period of implementation to ensure appropriateness□
Length of implementation - Allow enough time for participants to establish a routine of use, especially for an effectiveness studyREFERENCES •
• Allen. I. E., & Seaman, J. (2007). Online Nation: Five years of Growth in Online Learning. The Sloan Consortium, Olin Way, Needham, MA. Retrieved on February 19th, 2009 fromhttp://wiseplus.exp.sis.pitt.edu:8080/dspace/bitstream/
123456789/47/1/online_nation.pdf
• Alkin, M. C. (1980). Naturalistic study of evaluation utilization. New directions for Program Evaluation, 5, 19-27.
• Collier, V. (1989). How long? A synthesis of research on academic achievement in a second language. TESOL Quarterly, 23(3), 509-531.
• Met, M. (1991). Learning language through content: Learning content through language. Foreign Language Annals, 24(4), 281-295.
• Nichols, M. (2008). E-Learning in context. Retrieved on 28th April, 2008 from
http://akoaotearoa.ac.nz/sites/default/files/ng/group-661/n877-1---e-learning-in-context.pdf
• Shulha, L. M. & Cousins, B. (1997). Evaluation use: Theory, research, and practice since 1986. Evaluation Practice, 18, 195-208.
