Indicators of pedagogical quality for the design of a Massive Open Online Course for teacher training

(1)



RUSC Vol. 12 No. 1 |UniversitatObertadeCatalunyaandUniversityofNewEngland|Barcelona, January2015 CC_{L. Y. Alemán de la Garza, T. Sancho-Vinuesa and M. G. Gómez Zermeño}_|CC_{by FUOC, 2015}_|_{Indicators of pedagogical quality for the design…} Recommended citation

Alemán, L. Y., Sancho-Vinuesa, T., &GómezZermeño, M. G. (2015). IndicatorsofpedagogicalqualityforthedesignofaMassiveOpenOnline Courseforteachertraining. RUSC. Universities and Knowledge Society Journal, 12(1). pp. 104-118. doihttp://dx.doi.org/10.7238/rusc.v12i1.2260

Abstract

MassiveOpenOnlineCourses(MOOCs)havegeneratedhighexpectationsandrevolutionizedsomeeducational practicesbyprovidingopeneducationalresourcesforreference, usageandadaptation;therefore, theirpedagogical qualityisoftenquestioned. Theobjectiveofthisstudyistoidentifyindicatorsrelatedtopedagogical, functional,

technologicalandtimefactorsinordertoassessthequalityoftheMOOCentitled“Liderazgo en gestión educativa

estratégica a través del uso de la tecnología”(Leadershipinstrategiceducationalmanagementthroughtheuseof

technology), offeredasateachertrainingprogramthroughCourserato10.161participants. ViatheDelphimethod,

agroupof55expertsagreedthattimeisakeyfactortobeconsideredinthedesignoflearningactivities. Itwas concludedthatwithoutmeasuringresults, thesuccessofaMOOCcouldnotbeevaluated;thus, institutionsand consortiamustestablishevaluationindicatorstofocustheireffortsontheenhancementofpedagogicalquality. By providingrelevantinformation, thelearningpotentialofeducationalresourcesbasedonconnectivismprinciples canbeevaluated, and socanthequalityofMOOCs. The goalistocontributetoavisionofafuture inwhich everyonehasaccesstoaworld-classeducation.

Keywords

MassiveOpenOnlineCourses, MOOC, evaluation, pedagogicalquality, teachertraining

Indicators of pedagogical quality for the design of

a Massive Open Online Course for teacher training

Lorena Yadira Alemán de la Garza

1

, Teresa Sancho-Vinuesa

2

and Marcela Georgina Gómez Zermeño

3

1. MonterreyInstituteof TechnologyandHigherEducation(ITESM), Mexico|

[email protected]

2. OpenUniversityofCatalonia(UOC), Spain|[email protected]

3. MonterreyInstituteof TechnologyandHigherEducation(ITESM), Mexico|

[email protected]

Submittedin:June2014

(2)



RUSC Vol. 12 No. 1 |UniversitatObertadeCatalunyaandUniversityofNewEngland|Barcelona, January2015 CC_{L. Y. Alemán de la Garza, T. Sancho-Vinuesa and M. G. Gómez Zermeño}_|CC_{by FUOC, 2015}_|_{Indicators of pedagogical quality for the design…}

Indicadoresdecalidadpedagógicaparael diseñodeuncursoen líneamasivoyabierto

deactualizacióndocente

Resumen

Los cursos en línea, masivos y abiertos (MOOC) han generado importantes expectativas y han revolucionado algunas prácticas educativas, al ofrecer recursos educativos abiertos para su consulta, uso y adaptación; sin embargo, con fre-cuencia se cuestiona su calidad pedagógica. El objetivo de este estudio es identificar indicadores relacionados con factores pedagógicos, funcionales, tecnológicos y de tiempo, para evaluar la calidad del MOOC Liderazgo en gestión educativa estratégica a través del uso de la tecnología, ofrecido como recurso de actualización docente en Coursera a 10.161 parti-cipantes. Mediante el método Delphi, un grupo de 55 expertos acordó que el tiempo es un factor clave a considerarse en el diseño de las actividades de aprendizaje. Se concluye que sin medición de los resultados no se puede valorar el éxito de un MOOC, por ello instituciones y consorcios deben establecer indicadores de evaluación para enfocar sus esfuerzos para la mejora de su calidad pedagógica. Si se proporciona información relevante se podrá evaluar el potencial de aprendizaje que poseen los recursos educativos basados en principios conectivistas y reconocer la calidad pedagógica de los MOOC, con el objetivo de coadyuvar a la visión de un futuro en el que todos tengan acceso a una educación de clase mundial.

Palabras clave

(3)



1. Introduction

Currently, Massive Open Online Courses, or MOOCs, have generated high expectations and revolutionized

pedagogicalpracticesbyprovidingopeneducationalresourcesforreference, usageandadaptation(UNESCO, 2012).

c-MOOCs, whichgaverisetothisphenomenon, adoptedapedagogicalstrategywithanepistemologicalbasis

groundedinconnectivism;theyalsopromotededucationalchange, notonlythroughtechnologicaladvances, but

alsothroughthetheoreticaldevelopmentsthatemergedfromthis fieldofstudy(Zapata-Ros, 2013). Theyprovided

accessopportunitiestothegeneralpublicandwerefacilitatedbyrenownedprofessionalswhotookavitalrole.

Thus, c-MOOCsmadeknowledgefromawiderangeofprestigiousuniversitiesavailabletothewholeworldata

speed, scope, scaleandpricethatnotraditionalcoursewouldbeinapositiontooffer(Bell, 2012). Justadecade

ago, thistypeoftrainingwouldhavebeenimpossiblewithoutthecurrentadvancesoftechnology(Friedman, 2013;

Skiba, 2013).

Siemens(2004)positsthatconnectivismisaneducationaltheorycharacterizedbyconsideringlearningasan

extensionofapersonalnetworkthroughwhichparticipantslearn, shareknowledgeandcomprehend. Heclaimed

thatthemost importantcontributionofMOOCsresidedintheirpotentialtochangetherelationshipbetween

studentsandteachers, academiaandthecommunityat large, byofferingabroadand diversevirtual space, a

meetingplacefortheexchangeofideas. HeemphasizedthatanyoneenrollingonaMOOCwouldprobably find

learninginitsmostexposedform, throughplatformsthatnotonlyinvitedparticipantstoseeandhear, butalsoto

participateandcollaborate.

Despiteitsrapidintegrationintotheeducationaloffering, theInstituteforProspective TechnologicalStudies

claimsthatthecriteriaforevaluatingtheeducationalqualityofaMOOClacktransparencyandwarnsthat, forthe

moment, theimplementededucationalmodels areevidentlynotsustainable(Aceto, Borotis, Devine, &Fischer,

2014).

ForSangràandWheeler(2013), themassiveaspectofthecourses, whichispromotedassomethingpositive,

hasneverbeenafeatureofsuccessfultraining. Theybelievethat, inMOOCs, informallearninghasfoundaperfect

allyinICT ingeneral, andinonlinelearninginparticular. ThisemphasizestheneedtoinvestigateMOOCsindepth

inordertoestablishwhethertheyrepresentrealopportunitiesforlearningininformalsettings, oriftheyaresimply

attemptstoformalizetheinformal.

ThisarticlepresentstheresultsofastudyaimedatidentifyingindicatorsfortheeducationalqualityofaMOOC,

whichwillgreatlyimprovethedesignofthecourseentitled“Liderazgo en gestión educativa estratégica a través del

uso de la tecnología”(Leadershipinstrategiceducationalmanagementthroughtheuseoftechnology), offeredasa

teachertrainingupdatestrategybyahighereducationinstitution. Intheanalysis, theassessmentoftheindicators

byagroupof55expertsispresented, andtheprofileandexpectationsof10,161participantsinthe firsteditionof

thecoursearedescribed.

1

.

1

.

Background

Although the history of MOOCs may seem short in absoluteterms, Boven (2013) locates its origins in open

anddistance education. Henotesthatmanyemergingmovementshaveadoptedtheprinciplesadvocatedby

(4)



havesooftenbeendescribedas“recurrentresourcesinthediscourseofeducationalopenness”(McAuley, Stewart,

Siemens, &Cormier, 2010, p. 46).

Asitsname suggests, aMOOCisanonlinecourseaccessibletovirtuallyanyone whowishes toparticipate

withunlimitedattendance(EDUCAUSE, 2011). For TschofenandMackness(2012), MOOCsareonlinecoursesthat

appealtoawidevarietyofparticipantsaroundtheworld;theyaremassivebecauseliterallythousandsofpeople

canparticipateinthem, andtheyareopenbecauseparticipantscanopenlyshareresources, ideasandexperiences

withoutanyrequirements. Theyconstituteacollectivecreationofknowledge, resultinginacompositiongreaterthan

thesumofitsparts. Inmanyways, theyareamicrocosmofanation(Liyanagunawardena, Adams, &Williams, 2013).

Whilereviewingtheeducationalresearchundertaken, ever-increasinginterestinthestudyofMOOCsinrecent

yearswasnoted. Thiscontemporaryinterestismanifestedintheevolutionoftheirconceptualreferencesandthe

definitionoftheirmainfeatures. AccordingtoRodriguez(2012), theterm‘MOOC’wascoinedbyDaveCormierwhen

thenumberofstudentsonthe courseentitled“Connectivism andConnectiveKnowledge(CCK08)”offeredby

GeorgeSiemensandStephenDownesreachedatotalof2,300. Afteranalyzingtheresultsoftheirexperience, they

couldseeMOOCsasanewmodalityofonlineeducationandstatedthatitsimplementationrequiredconceptual

changesintheprocessesofteachingandlearning, fromtheperspectiveofteachersandstudentsalike.

1

.

2

. T

heoretical

perspectives

on

learning

In pedagogy, learning theories are the basis of educational processes, and they often refer to principles of

behaviorism, cognitivismandconstructivismtocreateinstructionalenvironments. Addingtechnologytoexisting

learningtheoriesraisesmanyquestions andpromptstheorists toreviewthem continuously, inordertoadapt

them as learning conditions change. These three theories weredeveloped at a time when learning had not

beenimpactedbytheuseoftechnologyineducation. Atsomepoint, theunderlyingconditionshavebeenso

significantlyalteredthatadditionalmodificationsare nolongerfeasible. Aninterconnected worldallowsusto

explorehowinformationisacquired;connectivistenvironmentshavefacilitatedthecreationofnetworkstoshare

resources, ideasandexperiences, andtheymustbeevaluated. Facedwiththisnewreality, acompletelydifferent

approachisnecessary(Siemens, 2004).

The SCOPEO (2013) report confirms that the“first generation”of MOOCs was based on connectivism, a

pedagogicaltheorythatpositsthatpersonalknowledgeiscreatedfromanetworkthatprovidesitsmemberswith

information. Inturn, thesemembersprovidefeedbackwithadditional informationgeneratedwithin thesame

network. Theprocessendswhenthisinformation, whichmaycomefromdifferentnodes, transformsandaltersthe

knowledgebase, andgeneratesnewlearningforindividuals. ForChiecherandDonolo(2013), MOOCshavebroken

curricularrigidity, questioningtheabilityoftraditionalteachingtomeettrainingneedsinchangingconditions.

Training offerings in which teaching differs from the what is commonly referred toas “traditional classroom

education”arebeingoverwhelminglyadopted. Thishasgivenrisetocriticalreflectionontheperceptionoflearners

–andissomethingthatdemandsaparadigmshift.

AddressingthefutureofMOOCs, someexpertspredictthattheywillmutateintodifferentformsoflearningwith

agreateremphasisonparticipants’supportandadecreaseontheirnumbertoallowadequateattention. Experts

emphasizetheneedtoimplementevaluationsystemstoestablishpedagogicalqualitycriteriathatgobeyond

(5)



1

.

3

.

Indicators

of

the

educational

quality

of

a

MOOC

Fortheeducational community, itis arealitythat the useoftechnology in teachingpractices has ledtothe

emergenceofmodernsocialstructuresandorganizationalforms, inwhichthetraditionalspaceandtimereferents

arenolongervalid(Garrido, 2003). Giventhelackofspaceandthegrowingdemandforadmissiontoeducational

institutions, onlineeducationisrenownedforofferingopportunitiestoexpandtheeducationalrangeandcoverage.

ForMoore(2013), distanceeducationisapsychologicalconstructthatdependsonmacrofactorssuchasdialogue,

structureandautonomy.

Inrelationtotheirpedagogicaldesignandtothelearningexpectationsofstudents, MOOCsimplyachangeof

instructionalschemes. At first, theirstructurewasthoughtofinaminimalistwaytoallowparticipantstodesigntheir

ownlearning, butresearch findingshavesincerevealedthat, inordertoimprovetheirpedagogicalquality, multiple

factorsrelatedtohow, whereandwhenparticipantslearnmustbeconsidered(TheNewMediaConsortium&

UniversitatObertadeCatalunya, 2012).

Although MOOCs imply the design of new schemes that recognize studies outside the classroom, the lack

ofstandardsforevaluatingtheirpedagogicqualityisoftenquestioned(Bernal, Molina, &Perez, 2013). ForGómez

-Zermeñoand Alemán(2012), the integrationoftechnologyintoeducationalprocessesrequiresthe establishment

ofboththeoreticalfoundationsand evaluationmechanisms toidentifytheirnumeroussourcesofinfluence. They

pointoutthatthedesignoftechnology-basedstrategiesthatseektostrengthenaccesstoqualityeducationposes

significantchallengestoeducationalresearchersyetoffersattractiveadvantagesthatcouldencouragetheiradoption.

IntheMOOCQualityProjectbytheEuropeanFoundationforQualityeLearning, itbecameapparentthatthe

evaluationofMOOCswasmorecomplicatedthaninotheronlineofferingsduetothelackofanagreeddefinition

onwhattheyare, totheinvolvementofperceivedexternalfactorsandtothelackofconsensusontheirpurpose

(EFQUEL, 2013). Bernal etal. (2013)recommendthat MOOCsshouldapplythesamequalitystandardsused in

formalopenanddistancecourses. Thefactthattheyaremassive, openandonlinecallsforgreaterrigorintheir

qualityinordertoovercomedissimilarprofilesandalackofmonitoringofresultsandobjectiveattainment.

Inassessingtheeducationalqualityofdistanceeducationandopenlearningresources, Arias(2007), Cabero

and Romero (2007), Gómez-Zermeño (2012), Gómez-Zermeño, Rodríguez, and Márquez (2013), Domingo and

Marquès(2011), andRoigetal. (2013)usedindicatorsrelatedtopedagogical, functionalandtechnologicalfactors.

ForBarbera, Gros, andKirschner(2012), timeisacriticalfactorthathasalsobeenusedasaqualitymeasure, since

itisrelatedtotheamountandthesequenceinwhichpeoplelearnthroughtheaccumulationofexperiences. In

collaborativelearningenvironments, theimplementationofstrategiesthatpromoteparticipants’self-regulationis

recommended(Franco-Casamitjana, Barbera, &Romero, 2013).

2. Method

Theobjectives ofthisresearchcanbeaddressed viadifferentmethodologies. Given theobject ofstudy, aquasi

-experimental design was adopted. Cross (2013) claims that educational research into MOOCs generates large

methodologicalandinterpretivechallenges, asitposesnewdynamicsintheteaching-learningprocess. Strengthening

(6)



researchintoaneducationalinnovationmayhaveaquasi-experimentaldesignthatdoesnotrequireevaluationofa

randomsample. Moreover, theresultsaremeasurablewithonepost-testgroup(Shadish, Cook, &Campbell, 2002).

Regardingqualityindicatorstostrengthenthedesignofacourseofthisnature, asetofindicatorswasselected

fromstudiesby Arias(2007), Barberaetal. (2012), CaberoandRomero (2007), Franco-Casamitjanaetal. (2013),

Gómez-Zermeño(2012), Gómez-Zermeñoetal. (2013), Domingo andMarquès(2011), andRoigetal. (2013). A

groupofexpertsreviewedandvalidatedtheseindicatorsusingtheDelphimethod. Thismethodinvolvesselecting

a groupof experts, who are asked theiropinion on issues relating to the future, implementing consecutive

anonymousroundstoensuretheautonomyofparticipants. Thepredictivepowerofthismethodisbasedonthe

systematicuseofintuitivejudgmentbyallexperts(Astigarraga, 2003).

Oncethesetofselectedindicatorshadbeenidentified, aquestionnaireentitled“MOOC-I-QualityIndicators”

wasdesigned,consistingofclosedquestionsthatassess50indicatorsona4-pointLikertscale. Fortheanalysis, the

indicatorswereclassifiedinto15subcategoriesrelatedtoPedagogical, Functional, Technologicaland Timefactors

(Table1). Thisinstrumentwasappliedtoagroupof55expertsinvolvedinthedesign, developmentanddeliveryof

MOOCs, aswellasindistanceeducationcoursesandopenlearningresourcesofferedbytheinstitution. Thisgroup

wasformedby14lecturersresponsiblefordesignandcontentgeneration, andatotalof41tutors, instructional

designers, graphicdesigners, programmersandaudiovisualproducers.

Based on studies by Breslow, Pritchard, DeBoer, Stump, Ho, and Seaton (2013), the“MOOC-I-Participants’

Information”questionnaire, madeupof30questionsgatheringdemographic, employment, educationalupdate,

expectationandopiniondatafromthoseonthecourse, wasdesigned. Takingintoconsiderationtheassessment

madeby the experts for indicatorsof pedagogicalquality, the MOOC entitled“Liderazgo en gestión educativa

estratégica a través del uso de la tecnología”(Leadershipinstrategiceducationalmanagementthroughtheuseof

technology)wasdesignedandofferedasateachertrainingupdatestrategy;thisquestionnairewasadministered

to10,161participantsonthe firsteditionoftheMOOC.

3. Results Analysis

Alignedwiththeobjectivesofthisresearch, theanalysisoftwodifferentissuesispresented. First, theassessment

ofqualityindicatorsforthedesignofMOOCsand, second, aftertheMOOCinquestionhadbeendesignedand

implemented, thestudents’profilesandexpectations.

3

.

1

.

Indicators

for

assessing

the

pedagogical

quality

of

a

MOOC

TheresultsoftheDelphimethodshowthatwhileexpertsratedtheindicatorsrelatedtothe Timefactorhigher, they

alsorecognizedthatthequalityofaMOOCmusttaketheotherfactorsintoaccount(Figure1). Itisworthnotingthat

theindicatorforthetimetoTake examswasratedthehighest. StudiesbyBarberaetal. (2012)corroboratethattime

remainsacrucialstrategicissueinonlineteaching, requiringexplicitattentionfromteachersanddesigners, since

itaffectsstudents’learning. ForFranco-Casamitjanaetal. (2013), timemanagementskillsdetermineself-regulation

ofstudentsandmembersofagroupalike(Guitert, 2011). Indistanceeducation, autonomyreferstotheextentto

(7)



Figure 1.Indicator categories

Pedagogical Functional Technological Time

3.25 3.3 3.35 3.4 3.45 3.5 3.55

%

When analyzingthe results by subcategory (Table 1), the expertsconsidered the pedagogical quality of

thecontentofaMOOCtobearelevantfactor. Theyalsorecommendedareviewofthepedagogicalapproach, the

provisionoftutorialsandthespecificationoftheevaluationprocessofeducationalactivities. ForMoore(1983),

the contentortopicofstudy determinesthedialoguebetweenteachersandlearners;italsoconstitutesone

ofthemaincharacteristicsofopeneducationalresources. Thus, excellentqualityshouldbeensured(UNESCO,

2012).

Table 1.Results of indicator assessment by subcategory

Category Subcategory Result

Pedagogical 3.37

Contents

Pedagogical approach

Tutorial andevaluation

Adequacyandadaptationtousers

Motivational capacity

Resources 3.60 3.47 3.44 3.29 3.27 3.15 Functional 3.4

Ease of use

Autonomyandusercontrol

Functionalityofthedocumentation

3.72 3.44 3.03

Technological 3.43

Interaction and dialogues Navigation

Visual environment

Designandtechnology

Versatility

3.61 3.52 3.52 3.44 2.97 Time 3.53 Take exams Performactivities Performexercises

Studythetopics

Calendar / Schedule

Participateindiscussionboards

(8)



Regardingthesubcategoriesoffunctionalfactors, expertsbelievethatEase of useisanimportantfactor, followed

byAutonomy and user control, andFunctionality of the documentation. UnderthesubcategoryEase of use, theindicator

“Exerciseinstructionsareclearandeasytounderstand”waspositivelyvalued, while“Theactivitiessuggesttheuse

of additional documents (blogs, wikis, etc.)” was considered less important. Among the Technological factors,

the subcategories Interaction and dialogues, Navigation and Visual environmentwere rated highly, while Versatility

was not considered relevant. The experts mentioned, in relation to the indicators of Interaction and dialogues,

that communication is one ofthe mostimportant elements, emphasizing the importance of makingmeans of

communicationavailabletofosterexchangesamongandbetweenparticipants(discussionboards, news, etc.)andtutors.

Intheexperts’opinion, MOOCsenablethecomprehensivedevelopmentofcompetitivecitizenswhoareable

topositionthemselvesasagentsofchange. Expertsrecognizethattomeetthegrowingdemandforeducation,

theadoptionofinnovativemodelsthatdevelopcommitmentandlinkagebetweenparticipantsisrequired(Brown,

2013). Suchmodelsshouldalsoemphasizetheneedtoimplementanevaluationsysteminordertoimprovethe

educationalqualityofresourceslikeMOOCs.

3

.

2

.

Participants’

pro

fi

les

and

expectations

Basedonthequalityindicatorsassessedbyexperts, theMOOCentitled“Liderazgo en gestión educativa estratégica

a través del uso de la tecnología”(Leadershipinstrategiceducationalmanagementthroughtheuseoftechnology)

wasdesignedandimplementedusingtheCoursera’®platform. Forthreeweeks, thecallformassparticipationin

thecoursewasdisseminatedviainstitutionalelectronicmediaandsocialnetworks. A totalof10,161participants

enrolledontheMOOCinquestion. Theparticipantswerefrom79countrieslocatedinallcontinents: The Americas

90.0%, Europe9.0%, Asia0.6%, Oceania0.1%and Africa0.1%. AccordingtoCoursera, 85%ofthestudy’spopulation

camefrom emergingeconomies. OneofthemainfeaturesofaMOOCisits opennesstoenrolment;soallthe

participantswhoconfirmedtheirinterestinvoluntarilyregisteringforfreebyansweringthequestionnaire“MOOC-I-

Participants’Information”formedthepopulationofthisstudy. Table2describestheirmainfeatures.

Table 2: Participants’ profiles on the MOOC entitled “Liderazgo en gestión educativa estratégica a través del uso de la tecnología” (Leadership

in strategic educational management through the use of technology)

Data Characteristics Responses

S

O

C

IODE

M

OGR

AP

HI

C

Country of residence

Mexico57%,Colombia 7%,Peru6%,Argentina3%,Chile3%, Ecuador2%,UnitedStates2%,Venezuela

2%,DominicanRepublic2%,Guatemala1%,Brazil 1%, Honduras0.7 %, El Salvador0.7%,CostaRica

0.6%,Bolivia0.5%,Uruguay0.5%,PuertoRico0.3%,Nicaragua0.3%,Paraguay0.3%,Panama0.2%,

Canada0.2%,andwith0.1% Trinidadand Tobago, Haiti, FrenchGuyana

Spain 7%,UnitedKingdom0.3%, Italy0.2%,Russian Federation0.2%,Portugal 0.2%,Germany0.2%,

France0.2%,andwith0.1%Greece,Poland,Switzerland,Belgium, Ireland,Denmark,Ukraine, Turkey,

Norway, TheNetherlands,CzechRepublic, Hungary,Austria,Sweden,Serbia,Romania,Moldavia,Malta,

Macedonia,Lithuania,Latvia,Cyprus,Croatia,Aruba,Andorra

China0.3%,andwith0.1% HongKong,Korea, India,Vietnam, Taiwan,Australia,Philippines, Thailand,

Singapore,New Zealand,NewCaledonia,Mauritius,Malaysia, Japan,Morocco,SouthAfrica,Angola,

Algeria,Pakistan,Kazakhstan, Israel, Iran, IslamicRepublic

Gender Female59.4%andMale40.6%

Age Averageof37 years9monthsold,highestfrequency34yearsold,rangefrom14to 76; 75.0%is45or

(9)



RUSC Vol. 12 No. 1 |UniversitatObertadeCatalunyaandUniversityofNewEngland|Barcelona, January2015 CC_{L. Y. Alemán de la Garza, T. Sancho-Vinuesa and M. G. Gómez Zermeño}_|CC_{by FUOC, 2015}_|_{Indicators of pedagogical quality for the design…} Data Characteristics Responses

O

CCUP

A

TIO

NAL

Occupation Teacher60.9%,Principal 11.0%,Pedagogical advisor10.9%, Higher2.5%, Inspector0.4%,Student14.2%

Educational level Preschool 8.5%, ElementarySchool 17.6%,SecondarySchool 18.7%, HighSchool 21.4%,Superior33.8%

Geographical zone Urban 76.0%,Rural 11.3%andUrban/Rural 12.7%

Type of school Public56.2%,Private32.8%andPublic/private11.0%

School equipment Mediaroom20.1%, Internetinmediaroom18.8%,Classroom10.4%, Internetinclassroom12.7%,

Principal’soffice17.5%, InternetinPrincipal’soffice17.8%andnone2.7%

TE AC HER TR A I N I N G UP D A TE

Educational level Undergraduate52%,Graduate37%, HighSchool 7%,PhD4%

Years of Service 5yearsor less28.6%,6to10years20.6%,11to15years14.6%,16to20years11.7%,21to25years

7.4%,26to30years5.9%,morethan31years3.5%,Noyearsofservice 7.7%

Frequency of teacher training update

courses

1to2courses53.7%,3to4courses22.0%,5to6courses4.5%,morethan 7 courses2.0%,doesnot

participateinteachertrainingupdatecourses17.7%

Types of teacher training update

courses

National CatalogueofContinuing Education11.3%,Centrefor Trainingand TeachertrainingUpdate9.7%,

National Programof TeachingCareer 7.4%,Requestedcourses21.6%,coursesfrompublicinstitutions

16.5%andcoursesfromprivateinstitutions18.7%andothercourses14.8%

Modality Classroom35.8%,Online19.4%,Blended40.7%andother4.1%

Use of ICT level None0.6%,Basic20.5%, Intermediate38.5%,Advanced32.7%, Expert 7.7%

Development of ICT

level None3.0%,Basic30.2%, Intermediate39.3%,Advanced22.7%, Expert4.9%

EX P E C T A TIVE S Reason for participating

TakeaMOOC15.4%,Model ofStrategic Educational Management31.1%, Technological tools25.4%,

TécnologicodeMonterreycourse17.7%,LearnwhataMOOCis10.1%andOther0.3%

Main expectation

Teacherprofessional development46.5%,Pointsfor TeachingCareer4.8%,KnowaMOOC14.7%,

EvidenceofMOOCparticipation13.6%, Evidenceof TecnológicodeMonterrey18.2%,

Economicstimulus2.3%

Workspace School 26.9%, Home67.7%andCybercafé 5.4%

Hours participation Nospecifichour27.1%,8:00am-10:59am9.1%,11:00am-02:59pm 7.3%,03:00pm-05:59pm8.3%,

06:00pm-08:59pm22.0%,09:00pm-12:00am26.2%

Intention to complete the course

I intendtofinishthecourse96.03%, I justwanttoknowtheagenda2.01%, I justwanttoparticipatein

someactivities1.52%and I donotintendtofinishthecourse0.45%

Inrelationtotheirexpectations, 43.3%oftheparticipantsbelievedthatthroughMOOCstheycouldacquire

enrichedlearningand23.6%mentionedthattheywouldexperiencehigherlearningproficiencycomparedtoa

classroomcourse, while29.8%thoughtthattheywouldbegettingthesamelearningexperience. Itisrelevantthat

96%oftheparticipantsexpressedcommitmenttothesuccessfulcompletionofthecourse, and68.99%reaffirmed

theirassurancetostrengthentheirprofessionaldevelopmentthroughparticipationinotherMOOCsofferedaspart

ofateachertrainingupdatestrategy.

Oncompletion, theMOOCachievedaterminalefficiencyrateexceeding22%(seeMOOC12inFigure2), which

isconsidered“atypical”, aswellasahighercommitmentratefromparticipants, whichreached52.15%. Itisnoted

thattheaverageterminalefficiencyrateofallMOOCsthisinstitutionhasofferedis4%, whichcoincideswiththe

(10)



Figure 2.Terminal efficiency rate with highest number of enrolments

0% 5% 10% 15% 20% 25%

%

MOOC 1 MOOC 2 MOOC 3 MOOC 4 MOOC 5 MOOC 6 MOOC 7 MOOC 8 MOOC 9 MOOC 10 MOOC 11 MOOC 12

4. Conclusions

Inthispaper, qualityindicatorsendorsingthedesignofaMOOConLeadershipinstrategiceducationalmanagement

havebeenidentified, ashavetheprofilesofparticipantsinits firsteditionandtherateofterminalefficiency.

ThemainconclusionfromthisresearchisthatthesuccessofaMOOCcannotbeevaluatedwithoutmeasuring

results. Thus, institutionsandconsortiashouldestablishindicatorstofocuseffortsonimprovingtheirpedagogical

quality. Designchangesanddevelopmentcannotbeplannedeither, astheseactionsrequiretheimplementation

of appropriate mechanisms to measure the participants’performance, which should be integrated into the

standardsoftheireducationalmodel. Whenapplyingqualitycriteria, itshouldbeconsideredthattheindicators

cannotoperateinisolation, soitisnecessarytodefinetheminordertobuildacompletepictureoftheeducational

systemthatwillreceivefeedback.

Accordingto55experts, timeisakeyfactorthatimpactsthepedagogicalqualityofaMOOC;however, they

emphasizethattheresultsarealogicalconsequenceoftheinterrelationshipbetweenthePedagogical, Functional,

Technologicaland Timefactors. WhendesigningaMOOC, thetimeitwilltakeparticipantstoreviewthecontent,

videos, resources, exercisesandtests, andtotakeanactivepartinactivitiesandcollaborativelearningdiscussion

boardsshouldbeconsidered(Gros, Barbera, & Kirschner, 2010). The designedsysteminvolves promotingself

-regulationskills, andmustincludetheelementsandapproachesthatsupportthetheoryofconnectivisminorder

tobeabletoofferopeneducationalresourcesthatprovidearealstrategicopportunitytoimprovethequalityof

education(UNESCO, 2012).

As the use of technology in education advances, the understanding of the elements and principles of

connectivismwillbechallenged, andeducationalresearchwillprovideevidenceonthenetworkcapacitiesand

possibilitiesintertwinedwiththe variouslearningstyles ofindividuals. By providingpertinent information, the

(11)



ofMOOCsofferedbyeducationalinstitutionsberecognizedandcontributetothevisionofafuture inwhich

everyonehasaccesstoaworld-classeducation.

4

.

1

.

Future

research

There are significant areas of opportunity related to the evaluation of MOOCs, as standards regarding their

pedagogicalqualityhaveyettobeagreedupon. Littleisknownabouttheirusesandscope, orabouttheways

inwhichtheyareintegratedintotheeducationalmodelsofdifferentinstitutionsandconsortia. Separatelinesof

researcharethereforeproposedtoaddressthisissue. Ontheonehand, itiscrucialtocarryonstudyingindetailthe

resultsofeachoftheindicatorsthatmakeupthedifferentsubcategoriesinordertounderstandtheinterplayof

allfactorsandtheirlevelofimpactonpedagogicalquality. Ontheotherhand, thedefinitionofsuccessofasimilar

courseshouldgofurtherthantherateofterminalefficiencyanddelvedeeperintotheanalysisoftrafficpatternsto

understandtheinterestsandbehaviorofitsparticipants.

5. References

Aceto, S., Borotis, S., Devine, J., &Fischer T. (2014). Mapping and Analysing Prospective Technologies for Learning.Seville,

Spain:JointResearchCentre, InstituteforProspective TechnologicalStudies.

Arias, J. (2007). Evaluación de la calidad de Cursos Virtuales: Indicadores de Calidad y construcción de un cuestionario de

medida (Doctoraldissertation).UniversityofExtremadura, Extremadura, Spain.

Astigarraga, E. (2003). El método delphi.SanSebastián, Spain:UniversidaddeDeusto.

Barbera, E., Gros, B., &Kirschner, P. (2012). Temporalissuesine-learningresearch: A literaturereview. British Journal of

Educational Technology, 43(2), 53-55. doi:10.1111/j.1467-8535.2011.01255.x

Bell, M. (2012). MassiveopenonlinecoursesmovingaheadwithMOOCs. Internet@Schools, 19(5). Retrievedfrom

http://www.internetatschools.com/Articles/Column/Belltones/BELLTONES-Massive-Open-Online-Courses

--Moving-Ahead-With-MOOCs-85936.aspx

Bernal, Y., Molina, M., & Pérez, M. (2013). LaCalidad de laEducación aDistancia: Elcaso de losMOOC. Revista

Iberoamericana para la Investigación y el Desarrollo Educativo, 3(10), 1-13. Retrievedfrom http://www.ride.org.mx/

docs/publicaciones/10/psicologia_y_educacion/E06_.pdf

Boven, D. (2013). The Next GameChanger: The Historical Antecedentsofthe MOOC Movementin Education.

eLearning Papers, 33, 1-7. Retrievedfrom http://www.openeducationeuropa.eu/en/download/file/fid/26967

Breslow, L., Pritchard, D., DeBoer, J., Stump, G., Ho, A., & Seaton, D. (2013). Studyinglearning inthe worldwide

classroom:ResearchintoedX’s firstMOOC. Research & Practice in Assessment Journal, 8, 13-25. Retrievedfrom

http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF2.pdf

Brown, S. (2013). BacktothefuturewithMOOCs. ICICTE 2013Proceedings, 237-246. Retrievedfrom http://www.icicte.

org/Proceedings2013/Papers%202013/06-3-Brown.pdf

Cabero, J., &Romero, R. (2007). Diseño y producción de TIC para la formación.Barcelona, Spain:UOC.

Chiecher, A., &Donolo, D. (2013). Trabajogrupalmediadoporforos. Aportesparaelanálisisdelapresenciasocial,

(12)



virtuales y aprendizaje. Nuevas perspectivas de estudio e investigaciones(pp. 151-198). Mendoza, Argentina:Editorial

Virtual Argentina.

Cross, S. (2013). Evaluation of the OLDS MOOC curriculum design course: participant perspectives, expectations and

experiences.OLDSMOOCProject, MiltonKeynes, UnitedKingdom.

Domingo, M., &Marquès, P. (2011). Classroom2.0ExperiencesandBuildingontheUseofICT in Teaching. Comunicar,

18(37), 169-174. http://dx.doi.org/10.3916/C37-2011-03-09

EDUCAUSE(2011). 7 things you should know about MOOCs. EDUCAUSELearningInitiative.

EuropeanFoundationforQualityine-Learning[EFQUEL](2013). The MOOC Quality Project. Retrievedfrom http://

mooc.efquel.org/the-mooc-quality-project/

Franco-Casamitjana, M., Barbera, E., &Romero, M. (2013). A MethodologicalDefinitionfor TimeRegulationPatterns

andLearningEfficiencyinCollaborativeLearningContexts. eLC Research Paper Series, 6, 52-62. Retrievedfrom

http://journals.uoc.edu/index.php/elcrps/article/download/1871/n6-franco-casamitjana

Friedman, T. (2013, January26). Revolutionhitstheuniversities. The New York Times, The opinion pages. Retrieved

from http://www.nytimes.com/

Garrido, A. (2003). El aprendizaje como identidad de participación en la práctica de una comunidad virtual(Doctoral

dissertation). UniversitatObertadeCatalunya, Barcelona.

Gómez-Zermeño, M. G. (2012). Bibliotecas digitales: recursos bibliográficos electrónicos en educación básica.

Comunicar, 20(39), 119-128. Retrievedfrom http://www.revistacomunicar.com/index.php?contenido=detalles

&numero=39&articulo=39-2012-14

Gómez-Zermeño, M. G., & Alemán, L. Y. (2012). Administración de proyectos de capacitación basados en tecnología.

Monterrey, Mexico:EditorialDigitaldel TecnológicodeMonterrey.

Gómez-Zermeño, M. G., Rodríguez, J. A., & Márquez, S. (2013). Estudio Exploratorio-Descriptivo“CursoHíbrido:

ContabilidadV”. Revista de Investigación Educativa de la Escuela de Graduados en Educación, 4(7), 70-79. Retrieved

from http://rieege.tecvirtual.mx/index.php/rieege/article/view/126

Gros, B., Barbera, E., &Kirschner, P. (2010). Timefactorine-Learning:impactliteraturereview. eLC Research Paper

Series, 0, 16-31. Retrievedfrom http://journals.uoc.edu/index.php/elcrps/article/download/issue0-gros-barbera

-kirshner/issue0-gros-barbera-kirshner

Guitert, M. (2011). Time management in virtual collaborative learning: the case of the Universitat Oberta de

Catalunya(UOC). eLC Research Paper Series, 2, 5-16. Retrievedfrom http://journals.uoc.edu/index.php/elcrps/

article/download/n2-guitert-catasus/n2-guitert-catasus

ITESM(2014). Formación que transforma vidas.Monterrey, Mexico: TecnológicodeMonterrey.

Liyanagunawardena, T., Adams, A., & Williams, S. (2013). MOOCs: A systematicstudy ofthepublishedliterature

2008-2012. The International Review of Research in Open and Distance Learning, 14(3), 202-227. Retrievedfrom

http://www.irrodl.org/index.php/irrodl/article/download/1455/2602

McAuley, A., Stewart, B., Siemens, G., &Cormier, D. (2010). The MOOC model for digital practice. Digital ways of knowing

and learning.Charlottetown, Canada:UniversityofPrinceEdward.

Menéndez, L. (2013). Estudiar‘online’ygratis. Escritura Pública, 80, 18-21. Retrievedfrom http://www.notariado.org/

liferay/c/document_library/get_file?folderId=12092&name=DLFE-89971.pdf

Moore, M. G. (1983). Theoryoftransactionaldistance. InD. Keegan(Ed.), Theoretical Principles of Distance Education

(13)



Moore, M. G. (2013). Thetheory oftransactional distance. InM. G. Moore (Ed.), Handbook of distance education

(3rded., pp. 66-85). NewYork, USA:Routledge.

ObservatoriodeInnovaciónEducativa(May, 2014). Reporte Edu Trends. Monterrey, Mexico:TecnológicodeMonterrey.

Retrievedfrom http://www.observatorioedu.com/redutrends

PennGSE(2013). Study shows MOOCs have relatively few active users, with only a few persisting to course end.University

ofPennsylvania:Graduate SchoolofEducation. Retrievedfrom http://www.gse.upenn.edu/pressroom/press

-releases/2013/12/penn-gse-study-shows-moocs-have-relatively-few-active-users-only-few-persisti

Rodriguez, O. (2012). MOOCs and the AI-Stanfordlike Courses: twosuccessful and distinct courseformats for

massiveopenonlinecourses. European Journal of Open, Distance, and E-Learning, 2012(2), 1-13. Retrievedfrom

http://www.eurodl.org/materials/contrib/2012/Rodriguez.pdf

Roig, R., Flores, C., Álvarez, J., Blasco, J., Grau, S., Guarinos, I., … Tortosa, M. (2013). Características de los ambientes de

aprendizaje on-line para una práctica docente de calidad. Indicadores de evaluación. Alicante, Spain:Universidad

de Alicante.

Romero, M. (2011). Thetimefactorinanonlinegroupcoursefromthepointofviewofitsstudents. eLC Research

Paper Series, 2,17-28. Retrievedfrom http://journals.uoc.edu/index.php/elcrps/article/download/n2-romero/n2

-romero

Sangrà, A., &Wheeler, S. (2013). NewInformalWaysofLearning:Or AreWeFormalisingtheInformal?RUSC. Universities

and Knowledge Society Journal, 10(1), 286-293. http://dx.doi.org/10.7238/rusc.v10i1.1689

Schmelkes, S. (2001). La investigación en la innovación educativa.Mexico:CINVESTAV.

SCOPEO(2013). MOOC: Estado de la situación actual, posibilidades, retos y futuro. SCOPEOReportNo2, June2013.

Retrievedfrom http://scopeo.usal.es/wp-content/uploads/2013/06/scopeoi002.pdf

Shadish, W. R., Cook, T. D., &Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal

inference.Boston, USA:HoughtonMifflin.

Siemens, G. (2004). Connectivism. A learningtheoryforthedigitalage. International Journal of Instructional Technology

and Distance learning, 2(1), 3-10. Retrievedfrom http://www.itdl.org/journal/jan_05/Jan_05.pdf

Skiba, D. (2013). MOOCsandtheFutureofNursing. Nursing Education Perspectives, 34(3), 202-204. doi: 10.5480/1536

-5026-34.3.202

TheNewMediaConsortium&UniversitatObertadeCatalunya(2012). Perspectivas tecnológicas: educación superior

en Iberoamérica 2012-2017: Un Análisis Regional del Informe Horizon del NMC y la UOC. Barcelona, Spain:UOC.

Tschofen, C., &Mackness, J. (2012). Connectivismanddimensionsofindividualexperience. The International Review

of Research in Open and Distance Learning, 13(1), 124-143. Retrievedfrom http://www.irrodl.org/index.php/irrodl/

article/download/1143/2117

UNESCO(2012). 2012 Paris OER Declaration. 2012 World Open Educational Resources (OER) Congress.Paris, France:

UNESCO.

Zapata-Ros, M. (2013). MOOCs, una visión crítica y una alternativa complementaria: La individualización del aprendizaje y

de la ayuda pedagógica. Universidadde AlcaládeHenares, Spain. Retrievedfrom http://eprints.rclis.org/18658/7/

(14)



About the authors

Lorena Yadira Alemán de la Garza

[email protected]

CoordinatorofSpecialProjectsandContinuingEducation, MonterreyInstituteof Technology

andHigherEducation(ITESM), Mexico

DoctoralcandidateontheEducationandICT (e-learning)doctoralprogramattheOpenUniversityofCatalonia(UOC),

Spain. Sheholdsamaster’sdegree, withhonors, inEducationalInstitution AdministrationfromITESM, andabachelor’s degree, withhonors, inBusiness Administrationfrom TecMilenioUniversity, Mexico. Shehasworkedasapostgraduate lecturer in the Graduate School of Education at ITESM on master’s degree programs in Educational Institution

Administration, inEducationandinEducational Technology. SheisamemberoftheResearchChairin TheSchoolasa KnowledgeOrganization. SheisthetechnicalmanageroneducationalresearchprojectsoftheMexicanNationalCouncil ofScienceand Technology(CONACYT). Sheisthecoordinatorofuniversityextensionprogramsandspecialprojects.

OfnoteamongtheseistheHigherMiddleEducation Teacher Trainingprogram(PROFORDEMS)andtheNew Teaching Profilediploma.

Instituto TecnológicoydeEstudiosSuperioresdeMonterrey(ITESM)

Avda. EugenioGarzaSada, 2501 Sur EdificioCEDES, pisoSS1

64849 Monterrey(NuevoLeón)

Mexico

Teresa Sancho-Vinuesa

[email protected]

LecturerandIN3 Researcher, OpenUniversityofCatalonia(UOC), Spain

Vice-rectorforResearchandInnovationattheUOC. SheholdsadoctorateinElectronicEngineeringandabachelor’s degreeinMathematics. SheistheprincipalresearcheroftheCIMANET (OnlineScienceandMathematics)RD&Igroupat theUOC. Shehaslecturedinthe fieldofnumericalanalysis, probabilitytheoryandstochasticprocessesatLaSalleSchool ofEngineeringand Architecture, Spain. Shehasbeenamemberofthe TEXTOpedagogicalandeditorialteam, andhas coordinatedMathematicsonInformation TechnologyStudiesattheUOC. Shehasbeentheacademiccoordinatorand directorofthedoctoralprograminInformationandKnowledgeSocietyattheUOC. Sheiscurrentlyresponsibleforthe Mathematicssubjectsonthebachelor’sdegreeprogramin TechnicalEngineeringand Telecommunications. Shehasbeen aresearcherontheinterdisciplinaryresearchprogramontheInformationSocietyinCatalonia(ProjectInternetCatalonia),

jointlyledbyManuelCastellsandImma Tubella;withinthisframework, sheheadedastudyonchangesoccurringin CatalanuniversitiesbyanalyzingInternetuse. Shehasnowresumedheractivitiesinthe fieldofMathematicsteachingfor Network-basedEngineering.

UniversitatObertadeCatalunya(UOC) RambladelPoblenou, 156

(15)



Marcela Georgina Gómez Zermeño

[email protected]

Researcher-Lecturer, DirectoroftheEducationResearchCenter, MonterreyInstituteof Technology

andHigherEducation(ITESM), Mexico

SheholdsadoctorateinEducationalInnovationfromtheGraduateSchoolofEducationatITESM, andamaster’sdegree inInformationandCommunication TechnologyEngineeringSciences. INT-CITCOM, France TélécomsHigherEducation.

Shealsoholdsabachelor’sdegreeinComputerand AdministrationSystemsfromITESM. Sheisatenuredlecturerat theGraduateSchoolofEducationatITESMonmaster’sdegreeprogramsinEducationandinEducational Technology,

andonthedoctoralprogram inEducationalInnovation. SheisamemberoftheResearchChairsin TheSchoolasa KnowledgeOrganizationandinInnovationin TechnologyandEducation, andoftheMexicanEducationResearchCouncil (COMIE). SheisthetechnicalmanageroneducationalresearchprojectsoftheMexicanNationalCouncilofScienceand

Technology(CONACYT)andonthe ALFA programoftheEuropeanCommission. SheformspartoftheNationalSystem ofResearchers(SNI)Level 1.

Instituto TecnológicoydeEstudiosSuperioresdeMonterrey(ITESM)

Avda. EugenioGarzaSada, 2501 Sur EdificioCEDES, pisoSS1

64849 Monterrey(NuevoLeón)

Mexico

Thetextspublishedinthisjournalare–unlessindicatedotherwise–coveredbytheCreativeCommons Spain Attribution 3.0 licence. Youmaycopy, distribute, transmitandadaptthework, providedyouattribute it(authorship, journalname, publisher)inthemannerspecifiedbytheauthor(s)orlicensor(s). Thefulltext ofthelicencecanbeconsultedhere:<http://creativecommons.org/licenses/by/3.0/es/deed.en><http:// creativecommons.org/licenses/by/3.0/es/deed.es>