Does technology improve reading outcomes? Comparing the effectiveness and cost-effectiveness of ICT interventions for early grade reading in Kenya

Does

technology

improve

reading

outcomes?

Comparing

the

effectiveness

and

cost-effectiveness

of

ICT

interventions

for

early

grade

reading

in

Kenya

Benjamin

Piper

*

,

Stephanie

Simmons

Zuilkowski

,

Dunston

Kwayumba

,

Carmen

Strigel

RTIInternational,NairobiRegionalOffice,TheWestwood,5thFloor,ValeClose,RingRoadParklands,P.O.Box1181VillageMarket,00621Nairobi,Kenya

b

FloridaStateUniversity,CenterforInternationalStudiesinEducationalResearchandDevelopment,LearningSystemsInstitute,UniversityCenterC4600,P.O.

Box3062540,Tallahassee,Florida32306-2540,UnitedStates

c

RTIInternational,NairobiRegionalOffice,TheWestwood,5thFloor,ValeClose,RingRoadParklands,P.O.Box1181VillageMarket,00621Nairobi,Kenya

d

RTIInternational,3040EastCornwallisRoad,P.O.Box12194,ResearchTrianglePark,NC27709-2194,UnitedStates

ARTICLE INFO

Articlehistory:

Received17March2015

Receivedinrevisedform14January2016 Accepted14March2016

Availableonline4April2016

Keywords: Literacy Internationaleducation Technology Reading Educationpolicy Kenya ABSTRACT

Educationpolicymakersareinvestingininformationandcommunicationstechnology(ICT)withouta researchbaseonhowICTimprovesoutcomes.Thereislimitedresearchontheeffectsofdifferenttypesof ICTinvestmentsonoutcomes.TheKenyaPrimaryMathandReading(PRIMR)studyimplementeda randomized controlledtrial comparing theeffects and costof three interventions– e-readersfor students,tabletsforteachers,andthebasePRIMRprogramwithtabletsforinstructionalsupervisors.The resultsshowthattheICTinvestmentsdonotimproveliteracyoutcomessignificantlymorethanthebase non-ICTinstructionalprogram.Ourfindingsshowthatcostconsiderationsshouldbe paramountin selectingICTinvestmentsintheeducationsector.

ã2016TheAuthors.PublishedbyElsevierLtd.ThisisanopenaccessarticleundertheCCBY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1.Introduction

TheglobalEducationforAllmovementhasbeensuccessfulby someindicators–improvingaccessforprimary-agechildren,for example– andafailurebyothers.Inmanycountries,including Mali,Uganda,andtheGambia,morethanhalfofchildrenarestill nonreadersattheendofsecondgrade(DubeckandGove,2015; GoveandCvelich,2011).Thisisacrisisatboththeindividualand nationallevels.Lowliteracyoutcomesareassociatedwithschool dropoutovertime(Marteletoetal.,2008),andyouthwithlimited schoolingarelesslikelytohaveoccupationaloutcomesotherthan farming occupations as adults (Winters et al., 2009). Recent research suggests that literacy skills are related to economic growthatthenationallevel(HanushekandWoessmann,2012). Keepingchildreninschoolswheretheydonotlearnliteracyskills–

or anything of pedagogical value – is, therefore, a waste of

resourcesfromanationaldevelopmentperspective.Accordingly, manyinternationaleducationalinitiativeshaveshiftedtheirfocus fromprogramsand policies relatedtoeducation accesstoward thosethatalsoaimtoimprovequality.

As mobile devices become cheaperand access more wide-spread, many policy makers, educators, and researchers have lookedtotechnologyasameansofaddressingthesequalityissues. Inlow-resourcesettings,mobiledevicescanincreasetheamount of text children are exposed to and supportthe scaffolding of poorlytrainedoruntrainedteachers.Mobiletechnologyallowsfor individualizedlearning,evenincontextswherelargeclassesare thenorm(GraceandKenny,2003).Contentcaneasilybeprovided digitally in multiple languages, whethernational languages or mother tongues (Wagner, 2014), thereby avoiding the costs associatedwithprintingsmallrunsoflanguage-specifictextbooks. With appropriate software, teachers can carry out continuous assessmentstoimproveinstructionandgiveimmediatefeedback onstudentperformance,providingtailoredsupportwhereneeded. Inthelonger run,exposuretotechnologycancontributetothe skills that children willneed for jobs in thefuture (Graceand Kenny, 2003).Evenchildren withoutpreviousexperience using mobile devices can quickly pick up the skills needed to play

*Correspondingauthor.

E-mailaddresses:[email protected](B.Piper),[email protected]

(S.S.Zuilkowski),[email protected](D.Kwayumba),[email protected] (C.Strigel).

http://dx.doi.org/10.1016/j.ijedudev.2016.03.006

0738-0593/ã2016TheAuthors.PublishedbyElsevierLtd.ThisisanopenaccessarticleundertheCCBY-NC-NDlicense(http://creativecommons.org/licenses/by-nc-nd/4.0/).

ContentslistsavailableatScienceDirect

International

Journal

of

Educational

Development

learninggames(Kim etal.,2012).Thesepotential benefitshave resultedinmanypilotprojectsindevelopingcountries,butlittle robust research on the effects of these projects on learning outcomes to guide stakeholders in determining policies or designinglarge-scaleprograms.

This paper aims to begin to fill this gap by examining the potential of ICT interventions to improve literacy within early gradeclassroomsinKenya.WithinthecontextofthePrimaryMath andReading(PRIMR)Initiative,arandomizedcontrolledtrial,1_we

compare the effects on learning outcomes – in one particular county–ofthreePRIMRinterventions:studente-readers,teacher tablets, and the base PRIMR intervention with tablets for government-funded instructional supervisors, who are called TAC tutors in Kenya.2 _{Wethen use a cost-effectiveness lens to}

examine those outcomes. Finally, using data fromassessments measuring Grade 2 students’ skills in literacy at baseline and endline assessments, we compare the impact of the three ICT interventiongroupswiththeresultsofthebasePRIMRprogramas implemented in other parts of Kenya. This approach provides evidencenotonly ofvalue-addedeffects ofan ICTintervention builtonto a successful literacy intervention, but also of which levelsofinterventionarelikelytohavethegreatesteffectin low-resourcesettingssuchasKenya.

2.Background

2.1.ICTliteracyinterventions

As access to mobile phones and other portable electronic devicesspreadsthroughouttheworld,thenumberofinterventions attemptingtousethesedevicestoenhancelearningforpeopleof all ages is increasing dramatically. Ten years ago, examining programsusingformsoftechnologythatwerecurrentatthetime, aWorldBankreportnotedthattherewaslittleevidenceonthe effectivenessandcost-effectivenessofICTinterventions,andthe fewstudiesthatexistedfoundmixedresults(Trucano,2005).In thedecadesince,thecostoflaptops,tablets,andsmartphoneshas decreased sharply,increasing theirviabilityasscalable toolsin low-resourcesettings,andtheevidencebaseontheireffectiveness inclassroomsandschoolsisbeginningtodevelop.Arecentreview ofabroadrangeofinterventionstoimprovelearningindeveloping countriesfoundthat, across15technology-basedinterventions, theaverageeffectsizewasamodest0.15standarddeviations(SD) (McEwan, 2014). Because not all of these interventions were successful,moredetailedcomparativeresearchintothefeaturesof suchprograms–whethertheysucceededor failed– maybea usefulnextstep.

BeforemovingtotheanalysisofthePRIMRresearch,webegin withaliteraturereviewfocusingonanarrowsetofstudies–those examiningtheuseofICTapplicationsinearlygradeclassroomsin developingcountriesfortheenhancementofliteracyinstruction andoutcomes.Twopossibilitiesexistthatwouldsupporttheuseof technologyinclassrooms:(1)thatitischeaperthanalternative approachesofteachingthematerial,or(2)thatitismoreeffective, withthebenefitsresultingingreatercosteffectiveness.However, littlerobustevidencehasbeenproducedthatspeakstothesetwo points (Wagner, 2014). Among the 44 mobile literacy projects reviewedbyWagner,onlyeight(i.e.,18%)hadconductedaformal evaluationofanytype,andjustonearandomizedcontrolledtrial.

Thecurrentbodyofresearchunderlinesthattechnologymust beintegratedintoaneffectivelearningprograminorderforitto haveanimpactonstudents.InPeru,arandomizedcontrolledtrial oftheOneLaptopPerChildprogramfoundthat,whilechildren’s access to and usage of computers increased, there were no statisticallysignificanteffectsonmathorlanguagescores(Cristia et al., 2012).The authors concluded that “governments should consideralternativeusesofpublicfundsbeforeimplementing large-scaletechnologyineducationprograms”(p.20).However,inthe Peruintervention,thesoftwarewasnottailoredtopedagogicalor curricularneeds,andstudentsspentmuchoftheirtimeusingword processingandotherbasictools,playinggames,andlisteningtoor recordingmusic.Thatstudy,therefore,doesnotdirectlyanswerthe questionofwhetheratechnologyprogrammightbeeffectivewhen builtontop of,and in directsupport of, a high-qualityliteracy program.ThismisalignmentbetweenthegoalsofICTprogramsand thetools’actualuseintheclassroomiscommon(Trucano,2005).

InurbanZambia,researcherstestedaNyanjaliteracygamewith urban first-graders using a randomized controlled trial ( Jere-Folotiya et al., 2014).The authorsfoundthe greatesteffects in spelling in the intervention groups in which the students and teachers all played the game and teachers were trained in a compatible phonics-basedapproach.ThisstudysupportsthehypothesisthatICT applicationscanbeeffectiveforliteracyiftheyarewell-alignedwith an underlying pedagogic intervention. But the study did not comparetherelativeeffectivenessofvariousICTapplicationsand hardwarechoicesthatmightbeavailabletopolicymakers.

A study using e-readers with 481 students in Ghana had positiveeffectsonfourth-gradestudents’readingscores, particu-larlywhenthechildrenreceivedsupportthroughanafter-school program(ILCAfricaandWorldreader,2012).However,40%ofthe e-readers broke during a seven-month period, which the implementersbelievedwasduetothedevicesnotbeingdurable enoughtohandlethedustandotherenvironmentalissues.3_The

programwas popularwithstudents andteachers despitethese implementationissues,andstudentswhoreceivedthee-readers often shared them with family members and friends, thereby creatingamultipliereffect.

ICT interventions may have differential effects for certain groups of students. For example, there is some evidence that mobileliteracyapplicationsworkbetterforchildrenwhoalready havestrongerliteracyskillsbeforetheintervention.Asanexample, the Mobile and Immersive Learning for Literacy in Emerging Economies(MILLEE)programdevelopedliteracygamesforuseon mobilephonesinthecontextofanEnglishafter-schoolprogramin Uttar Pradesh, India (Kam et al., 2009). Participants were prescreenedwithasimpleEnglishtestrequiringthemtoexhibit theirabilitytowritesimplesentences. Therewas astatistically significantassociationbetweenpretestscoreandgainscore,with thestudentswithstrongerskillsgainingmoreoverthecourseof the program. Children may also face differences in ICT device accessandusagebygenderorsocioeconomicstatus,althoughthe limited amount of rigorous data available makes conclusions regarding this suppositionimpossible atthis point. Though ICT applicationshavethepotentialtobeadaptiveforstudentswith visual, hearing,physical,or developmental disabilities(Wagner etal.,2005),theyhavenottypicallybeendesignedforsuchusage ininterventionsinsub-SaharanAfrica.

Even when technology use is relatively intensive, teachers remain the core of the education system. Teachers must be

1 _{PRIMR’speriodofperformancewas2011–2014;additionalprogramdetailsare} presentedinalatersubsection.

2

TACtutorsoperatefromwithintheTeachers’AdvisoryCentre(TAC),aunitofthe Teachers’ Service Commission. In 2016, the title was formally changed to “CurriculumSupportOfficer.”

3

Worldreader,oneoftheimplementersofthestudy,hassinceincreasedusage andcareeducationinitsprograms,aswellasworkedwithhardwaresuppliersto increasethedurabilityofitse-readers.Arecentreportindicatedthatbreakagerates hadfallenbelow10%(Worldreader,2013).

supportedinplanninglessonsthatmakeuseofthetechnologyina productiveway (Trucano, 2005); the availability of technology doesnotnecessarilyleadtochangesinteacherbehaviorwithout accompanying training and an overall theoretical approach designedtoimprovelearning(Cristiaetal.,2012;Wagneretal., 2005).WhileICTscanenhanceteachermotivation,theycanalsobe a source of stress for those who are new to the technology (Trucano, 2005), and this stress may result in less effective teachingoverall.However,inGhana,teachersreportedthathaving currenttextbooksandmaterialsonane-readermadeiteasierfor themtoplan(ILCAfricaandWorldreader,2012),suggestingthat suf_ficient training canhelptoovercome initialteacher reserva-tions.

Thestudiesdiscussedabove,includingthebroadreviewcarried outbyWagner(2014)oftherangeofprogramsbeingimplemented inAfricaandAsia,makeclearthatICTscannotstandaloneasthe solutiontothedevelopingworld’schallengesineducationquality. Itisimportanttoacknowledgethat“socialpractices”arepartofthe equation (Roschelle, 2003). That is, technology may change relationshipsbetweenandamongstudentsandteachersinways that are unintended. As proposed by Wagner (2014), mobile learningapproacheswillbemostsuccessfulwhentheycombine appropriatedevices,aredesignedwiththeendusersinmind,and arefocusedonaspecificpurpose.Thesefactorsareallinfluenced by the context in which the program is situated (see Fig.1).

Wagner’s (2014) model extends previous research because it requiresresearcherstoexaminetheinterplayamongthedevices chosen, theend users to be affected, and the purposes of the program,withinaparticularcontext.Ouranalysisoftheliterature foundveryfewrigorousstudiesthatexaminedallthreeofthese factors,andevenfewerthatdidsowithinthesub-SaharanAfrican context.

2.2.Context

Kenya’sFreePrimaryEducationinitiativehasbeensuccessfulin ensuringthatchildrenhavetheopportunitytoattendschool.Gross primaryenrollment rates arenow above 100%(UnitedNations Educational,ScientificandCulturalOrganization(UNESCO),2014). Studentperformancehasbeenpoor,however,especiallyontestsof KiswahiliandEnglishliteracyintheearlygrades.TheMinistryof Education, Science, and Technology (MoEST) has set national readingfluencybenchmarksforGrade2of65correctwordsper minute(cwpm)for Englishand 45 cwpmforKiswahili.Similar benchmarksforemergentfluencyweresetat30cwpmforEnglish

and17cwpmforKiswahili.InthebaselineassessmentfortheICT study,theresearchersfoundthatGrade2studentsreadonaverage only9.3cwpminEnglishand8.9cwpminKiswahili.Theseresults alignedwiththosefroma numberofpreviousstudiesinwhich Kenyanstudentsfailedtodemonstrategrade-appropriatereading skills(Mugoetal.,2011;Onsomuetal.,2005;Wasangaetal.,2010). PRIMR’s ICT study was implemented in Kisumu County, a diversecountyinthewesternpartofKenya.Fishing,agriculture, andlightindustryaretheprimaryoccupations.TheLuoarethe major ethnic group, and the predominant mother tongue is Dholuo.Thismeansthatchildreninprimaryschoolsarelearning in, at best, their second and third languages (Kiswahili and English), as the language-of-instruction policyof using mother tongueisveryseldomfollowed(PiperandMiksic,2011).

2.3.ThePrimaryMathandReadingInitiative(PRIMR)

Asnoted,theICTinterventionevaluatedherewasbuiltonthe basePRIMRprogramfundedbyUSAID,whichwasimplementedin 370 schools and had four central components, none focused specificallyonICT.AnexpansionofthePRIMRprogram,fundedby the UKDepartment for InternationalDevelopment (DFID), was implementedin834publicprimaryschoolsinrurallocations.The PRIMR program, whether funded by DFID or USAID, provided teacherand headteachertraining inliteracyand numeracyfor 10daysper year, including aninitial five-daytraining and two refreshersessions;low-cost,full-colorbooksforeverychildata 1:1ratio;teachers’guidesthatmatchedcloselywiththecontentin thestudentbook;andongoinginstructionalsupportforteachers onthekeyelementsofinstructionmoststronglycorrelatedwith learning achievement.In additiontothesekeycomponents, an engaged and active MoEST encouraged its instructional super-visors,theTACtutors,toconsistentlyvisitclassroomsandprovide instructional feedback through a coaching model (Piper and Zuilkowski,2015).Thisfullpackageofinterventionsledtoeffect sizesrangingfrom0.2to0.5SDforliteracyoutcomesinEnglish andKiswahiliinthebase,non-ICTPRIMRprogram(Piper,Jepkemei et al., 2015; Piper et al., 2014). The base PRIMR program had alreadybeeninplaceforoneyearatthecommencementoftheICT trialinKisumuCounty.

This randomized controlled trial design compared the out-comesfromacontrolgroupwiththerelativeeffectivenessofthe basePRIMRprogram(whichincludedamodestTACtutortablet component)aswellastwomoreintensiveICTinterventions,oneof which added tablets for teachers and the other of which incorporatede-readersforstudents.Thethreeinterventionsare discussedindetailbelow.

The PRIMRICT program in KisumuCounty was designed to capitalizeontheinterestandengagementoftheMoESTaboutICT. The2013academicyearcoincidedwiththeelectoralpromiseof thenewJubileecoalitiongovernmenttoprovidelaptopsata1:1 ratioforallGrade1childreninthecountry.Duringtheperiodin question,MoESTpolicymakershadheightenedinterestinICT,and they perceivedthe PRIMR ICT program as being an important precursortothelarger-scalelaptopprogram.Thethreetechnology optionstestedinthisstudywereproposedratherthanlaptopsdue totheirlowercostandeaseofuse.

3.Researchquestions

Basedonthegapsintheliteraturerelatingtoourunderstanding ofwhatimpactICTcanhaveonlearningoutcomes,howchoicesin hardwareandsoftwareinfluencelearningoutcomes,andhowthe costofICTinterventionsrelatestolearningoutcomes,wechoseto addressthefollowingresearchquestions:

RQ1:Whichofthethreetreatmentswasthemosteffective?

Fig.1.ICTprogramdesignsolutionasafunctionofpurposes,devices,andend users.

RQ2: Which of the three treatments was the most cost effective?

RQ3:HowdidthetreatmenteffectscomparetothebasePRIMR programimplementedelsewhereinKenya?

4.Researchdesign

Inthissectionwepresentdetailsabouttheresearchdesignof thePRIMRICTstudy.

4.1.Treatmentgroups

Thethreetreatmentgroupswereasfollows.

InthePRIMR+TACtutortabletgroup,theinterventionwas essentiallythesameasthebasePRIMRprogram.Itincludedthe same amount of teacher training, instructional support, and studentlearningmaterialsata1:1ratio.Inaddition,thistreatment gavetwoofficerstabletstosupportteachersintheirinstruction. Thatis,theTACtutorsintwozones,oneperi-urbanandonerural, weregivenatabletthatcontainedmaterialstoimprovethequality oftheirinstructionalsupporttoteachers.Thisincludedelectronic versionsoftheteachers’guidesfortheTACtutorstofollowasthey conducted classroom observations; electronic versions of the studentbooks,tocutdownontheheavyloadthatmanyTACtutors physically carried during the implementation of the PRIMR program;andinstructionalsupporttoolsthatsupplementedthose PRIMR-specificitems.Thecontentincludedallstudentandteacher materialsthatwereontheteachertablet(seenextparagraph)in additiontomaterialsthatweredesignedexclusivelyfortheTAC tutors.Suchmaterialsincludedguidanceonspecificpedagogical challenges in Kenyan classrooms. The TAC tutors’ ability to consistentlyusethetabletsforinstructionalsupportwasuneven, andoneofthetwoTACtutorsassignedtothePRIMR+TACtutor treatmentgroupstruggledtousethetechnologyeffectively.This treatment group was, in short, the base PRIMR program implementedintwozonesofKisumuCounty.

Intwootherzones,oneperi-urbanandonerural,allGrade2 teachersinthePRIMR+teachertabletgroupwereprovidedwith atablettoscaffoldtheirliteracyinstruction.Thetabletcontained multimedia lesson plans that matched the base PRIMR paper-basedlessons,butalsohadembeddedaudiofilesforlettersounds and word reading; supplementary pedagogical aids, including letterflashcardsandanapplicationcalledPapayaTM_,designedfor

letter-soundpracticeforteachersstrugglingtodifferentiateletter sounds in English and Kiswahili; and the Tangerine:ClassTM

application, which was a sophisticated classroom feedback mechanism.Teachers wereguided ontablet-basedassessments togive theirstudents,basedonwherethey werein thePRIMR materials, and then the Tangerine:Class program analyzed the resultsandgaveteacherssuggestedfeedbackonwhattoreteach

and which pupils needed more help. A recent study on the Tangerine:ClassprogramshowedthatthePRIMR+teachertablet intervention had carefully scaffolded training and follow-up support,andthatteacherspreferredtoteachfromthetabletthan from the typical book-based lesson plans(Strigel et al., 2015). Successin thisICTcomponent reliedontheteachers’abilityto manipulate thetabletaswellastheirskillinusingthecontent therein.Teachers variedwidely in theirfrequency ofusing the tabletsforlessons,withsometeachersimplementingthelessons with fidelity, and others only infrequentlyutilizing the PRIMR methods.WithrespecttotheuseoftheTangerine:Classsoftware, fewoftheteachersusedthetoolconsistently(Strigeletal.,2015), although the teachers spoke positivelyaboutthe powerof the tool’sabilitytoprovidenuancedfeedbackonlearning(Piperand Kwayumba,2014).

Lastly,inthePRIMR+pupile-readersgroup,Grade2students and teachersin two zones,oneperi-urbanand onerural,were providedwithindividuale-readers.Thesee-readerscontainedthe basePRIMRreadingtextbooksinEnglishandKiswahiliinelectronic format; relevant textbooks from Kenyan publishers; reading materialsinDholuo,thelocallanguageofmanyinKisumuCounty; andhundredsofadditionalgrade-appropriatereadersinEnglish, Kiswahili,andDholuo.Thenumber ofbooksavailable wasexpanded duringtermbreaks,suchthatduringthe2013academicyear,each studenthadaccesstomorethan160additionalreaders.Duringthe initial implementation of the program, pupils were allocated afternoonsessionsforreading,withteachersnearbyavailablefor consultation.Bythesecondterm,pupilswereallowedtoreadfrom home.Thee-readergroupwassupportedbyWorldreader,which hadtwofull-timestaffwhoshowedtheteachershowtousethe e-readerseffectively.WithoversightfromtheWorldreaderteam,the e-readerschoolcommunitiesmanagedthee-readersexceedingly well,aslessthan2%ofe-readerswerelostorseverelydamaged duringtheentireyearofthePRIMRICTprogram.

Allthree treatmentdesignsinvolvedtraining theTAC tutors supportingtheinterventionpriortotheteachertraining.TheTAC tutors were shown how to give teachers instructional support using the base PRIMR program or the ICT-enhanced PRIMR program. TheTACtutors ineach ofthethree treatmentgroups received 15days of teacher training spread out over the three terms.Teachersparticipatedinatotalof10daysoftrainingledby the TAC tutors, supported by the PRIMR technical teams. The trainingwasdistributedoverthethreetermssothatteachershad initialtrainingsandthenrefreshersbeforethesecondandthird termsoftheyear.For thetwoICTtreatment groups,thePRIMR teacherandTACtutortrainingsincludedsessionsontheusageof thedevicesandthesoftwareprovided,withparticularemphasis onthePRIMRmethodologyinEnglishandKiswahili.Duringthe refresher trainings, the topics discussed were tailored to each interventiongroupanddesignedtoaddressissuesobservedand

Table1

StudentdatacollectedatPRIMRICTprogrambaselineandendline.

Treatmentgroup Sampledzonesandschools Pupilsassessed

Zonetype Schoolsperzone Sampledschools January2013baseline October2013endline

PRIMR+tutortablet Peri-urban 16 10 198 198

Rural 13 10 197 188

PRIMR+teachertablet Peri-urban 15 10 200 196

Rural 18 10 199 199

PRIMR+e-reader Peri-urban 15 10 200 200

Rural 15 10 190 193

Control Peri-urban 15 10 199 200

Rural 18 10 197 186

reported during the previous term, to improve English and Kiswahiliinstruction.

4.2.Sample

As noted, the ICT study used a randomized controlled trial design.ThePRIMRteamrandomlyselected8zonesfromthetotal of34zonesinKisumuCounty.These8zoneswerethenrandomly assignedtooneofthethreetreatmentconditionsorthecontrol group,stratified by ruraland peri-urban location. Within each zone,10primaryschoolswererandomlyselectedtoimplementa particulartreatment.Thisresultedinatotalof80schoolsinvolved intheICTprogram:60interventionschoolsand20controlschools. WithrespecttothebaselineandendlineassessmentsinJanuary 2013 and October 2013, respectively, the PRIMR researchers randomlyselected20Grade2pupilsfromeachofthe80primary schools.Simplerandomsampling,stratifiedbysex,wasusedto sampleanequal numberofboys andgirlsin thestudyatboth baseline and endline. The baseline literacydata collection was completedin January2013,at thestart of theschool yearand beforethestartofimplementation;datacollectionfortheendline studywascompletedinOctober2013.Theassessedsamplewas 1580studentsinJanuary2013and1560studentsinOctober2013 (seeTable1).

4.3.Measures

The instrument used to measure student performance at baselineandendline,forallthreeICTtreatments,wastheEarly GradeReading Assessment (Dubeckand Gove,2015; Gove and Wetterberg,2011),adaptedforuseinKenya.Asintheevaluations ofthebasePRIMRprogram,thisanalysisoftheICTstudyfocuseson the key outcomes of oral reading fluency (ORF), expressed in correctwordsperminute;andthepercentagesofpupilsreaching theMoEST’sbenchmarksforreadingfluencyandcomprehension (studentswereassessedonmanyotherlanguagetasksthatarenot presented,forparsimony).TheinstrumentsusedintheICTimpact evaluation had internal consistency results similar to those establishedforthebasePRIMRprogramimplementedelsewhere inKenya(PiperandKwayumba,2014;PiperandMugenda,2012). Data were collectedusingTangerine1,an open-source early learningassessment softwaredevelopedbyRTI Internationalto enhance data collection efficiency, assure data quality, and acceleratedataavailabilityforanalysis.Thisapplicationhadbeen pilotedinKenyaandhadsincebeendeployedmorethan20times in 20 different countries by 10 different organizations (RTIInternational,2013).Tangerineutilizedanautomatictiming system,auto-stopfunctionalitieswhenevertheallocatedtimeran outontheinstruments’timedsubtasks,andastopfunctionthat

allowed assessorstomarkthelast itemsread.Additionally, the applicationallowedthechildtobeaskedonlythequestionsthat correspondedtothepercentageofthepassagethechildhadread. TheEGRAwasconductedbyatrainedteamofassessors,somewith more than five years of literacy assessment experience. The average interraterreliability scores forthe fielddata collection teamwere96%duringbaselineand97%atendline.

Table2presentsdescriptivestatisticsforEnglishoralreading

fluency and Kiswahili oral reading fluency at baseline (January 2013)andendline(October2013).Theresultsshowlowfluency rates in both English and Kiswahili at thebaseline for all four treatment groups. The resultsat the baseline showed that the lowest mean fluency rates were found in the teacher tablet treatmentgroup,withthehighestintheTACtutortablettreatment group. Atthe endline, thecontrol group had the lowest mean

fluencyratesinKiswahili,whiletheteachertabletgrouphadthe lowestfluencyratesinEnglish.

4.4.Data-analyticplan

Despiterandomassignmentofthezonestotreatmentgroups, analysis of the baseline data revealed statistically significant differencesinperformanceamongthefourgroups.Specifically,as notedabove,theteachertabletgroup’sbaselineresultswerelower thanthoseofothergroups,includingthecontrolgroup.Inorderto determine theimpact of thetreatment, we therefore utilizeda difference-in-differences (DID) identification strategy using ordinary least squares(OLS)regressionmodels.DIDmodelscompare changes in a program’s outcome variables at two different assessmentpointsfortreatmentand controlgroups.Thisallows the analysis to separate program impact from changes in the populationnotduetoprogramimpact(MurnaneandWillett,2011). AkeyassumptionoftheDIDmodelisthatthetrendsinthe covariatesnotrelatedtotheinterventionwillbesimilarforthe treatmentandcontrolgroups.Table3presentsthekeycovariates at the student, teacher, classroom, and school levels. We used regression analysistoestimatewhethertherewerestatistically significantdifferencesinthesecovariatesbetweentreatmentand controlgroups. Covariates arepresentedby treatmentgroup at baseline, and then the differences between the baseline and endlineresultsappearinthecolumnforendline.Thecovariates presented are the percentage of students who were female; studentage;astudentwealthindex(thetotalofninehousehold items that the students identified as having in their home); whetherthestudentattendedkindergarten,pre-kindergarten,or otherpreschool;whetherthepupilhadtheEnglishorKiswahili books;andwhetherthechildreadathome.Attheteacherlevel,we presenttheaverageageoftheteachers,andtheaverageclasssize. At the school level, we share head teachers’ average years of

Table2

Grade2EnglishandKiswahiliORFatbaselineandendline.

Group Averagecorrectwordsperminute,bylanguageandassessmentperiod (standarderrorsinparentheses)

English Kiswahili

Baseline Endline Baseline Endline

Control 8.6 (0.4) 25.3 (1.1) 7.7 (0.4) 16.4 (0.7) Pupile-reader 10.1 (0.6) 36.0 (1.1) 10.0 (0.5) 24.9 (0.7) Teachertablet 5.7 (0.3) 23.9 (0.8) 6.3 (0.3) 18.9 (0.6) TACtutortablet 15.5

(0.8) 39.7 (1.0) 13.0 (0.6) 26.6 (0.6)

experience,andthepercentageofschoolsreportinghavingaccess toregularwatersourcesandtoelectricity.

Table 3 shows that manyitems differedvery little between treatmentgroups.Allfourtreatment groupshadbetween47.6% and51.3% femalestudents, andtheaveragestudentageranged between8.0and8.3yearsold.Studentshadbetween3.0and3.4of thehouseholditems named inthe questionnaire,and between 86.7%and96.0%ofpupilsreportedattendingkindergartenorother preschool.Theageandexperienceofteachersalsowereminimally different,withaverageteacheragesbetween41.2and45.3years old, and head teachers having between 7.6 and 8.8 years of experience. Differences existed in students’ access to English books,withthecontrolgrouphavingthehighestrate(39.7%)and theTACtutortabletgrouphavingthelowestrate(30.4%).Similarly, the control group had the highest access to Kiswahili books (35.2%), with the teacher tablet group having the lowest rate (24.3%).Thecontrolgrouphadthehighestpercentageofstudents readingathome(52.2%)andtheTACtutortabletgrouphadthe lowestpercentage(35.7%).Classsizesdifferedbytreatmentgroup, withthecontrolgroup(53.2students)andTACtutortabletgroup (52.5students)havinglargerclassesthanthee-readergroup(41.5 students)ortheteachertabletgroup(42.6students).TheTACtutor tablet grouphad thehighest percentage of schools withwater

(75.9%) while the control group had the lowest (55.0%). The e-reader group had the highest percentage of schools with electricity (36.3%), with the control group having the lowest (16.1%).Thebaselinecomparisonsonthesecovariatesrevealedno largesystematicdifferencesadvantagingoneparticulartreatment groupoveranother.

InorderfortheDIDestimatetobevalid,whatisimportantis notsimplywhetherthereweredifferencesamongthegroupsat thebaseline,butwhetherthetrendsinthecovariatesweresimilar. Theendlinecolumnsshowthedifferencesinthecovariatefromthe baseline.Our interestwasinsubstantivelydifferenttrendsover time between thetreatmentgroups, becauseif they existed, it wouldinvalidatetheassumptionsoftheDIDanalysis.Forstudent gender,age,wealthindex,andkindergartenattendance,theresults showthatnoneofthechangesbetweenbaselineandendlinewas largerthan1%for anyof thetreatment groups.Similarly, small changes existedfor teacherage,class size,andyears asa head teacher.Largerdifferencesfrombaselinetoendlineexistedforthe variablesrelatedtohaving anEnglishbook,Kiswahilibook, and readingathome,particularlyforthethreetreatmentgroups.This islogical giventhatthesethreetreatmentsincreasedaccessto learningmaterials.Largerdifferencesexistedfortheschoolhaving accesstowater,withthecontrolgroupincreasingby2.4%whilethe

Table3

Covariatesattheschool,teacher,classroom,andteacherlevelatbaseline;anddifferencesatendline,bytreatmentgroup(standarderrorsinparentheses).

Covariate Control Pupile-reader Teachertablet TACtutortablet

Baseline Endline Baseline Endline Baseline Endline Baseline Endline Studentfemale(%) 48.5 (0.0) 0.6*** (0.0) 51.3 (0.0) 0.0*** (0.0) 47.6 (0.0) 0.3*** (0.0) 48.3 (0.0) 0.1*** (0.0) Studentage(years) 8.0

(0.1) 0.0* (0.0) 8.2 (0.1) 0.1*** (0.0) 8.1 (0.0) 0.1*** (0.0) 8.3 (0.1) 0.0** (0.0) Studentwealthindex 3.4

(0.1) 0.1* (0.0) 3.4 (1.0) 0.1** (0.0) 3.0 (0.1) 0.1*** (0.0) 3.4 (0.1) 0.1*** (0.0) Attendedkindergartenorpre-K(%) 89.3

(1.3) 1.0** (0.3) 94.0 (1.3) 0.2 (0.3) 86.7 (1.4) 0.8* (0.3) 96.0 (1.2) 0.9** (0.3) HadEnglishbook(%) 39.7

(2.7) 1.1 (0.6) 39.4 (2.9) 1.0 (0.6) 32.3 (2.3) 7.0*** (0.5) 30.4 (2.5) 7.0*** (0.5) HadKiswahilibook(%) 35.2

(2.6) 0.8 (0.6) 30.7 (2.7) 1.9** (0.6) 24.3 (2.1) 7.5*** (0.4) 31.0 (2.7) 6.6*** (0.6) Readsathome(%) 52.2 (2.8) 1.1 (0.6) 48.5 (2.8) 2.2** (0.6) 42.5 (2.2) 3.4*** (0.5) 35.7 (2.8) 7.2*** (0.6) Teacherage(years) 45.3

(0.0) 0.0*** (0.0) 41.2 (0.0) 0.1*** (0.0) 41.8 (0.0) 0.2*** (0.0) 43.8 (0.0) 0.1*** (0.0) Classsize(numberofstudents) 53.2

(0.0) 0.2*** (0.0) 41.5 (0.0) 0.8*** (0.0) 42.6 (0.0) 0.1*** (0.0) 52.5 (0.0) 0.6*** (0.0) Yearsasaheadteacher(years) 7.6

(0.0) 0.1*** (0.0) 8.8 (0.0) 0.3*** (0.0) 8.1 (0.0) 0.2*** (0.0) 7.7 (0.0) 0.1*** (0.0) Schoolhadwater(%) 55.0

(0.0) 2.4*** (0.0) 57.5 (0.0) 0.9*** (0.0) 64.0 (0.0) 0.9*** (0.0) 75.9 (0.0) 0.9*** (0.0) Schoolhadelectricity(%) 16.1

(0.0) 4.1*** (0.0) 36.3 (0.0) 1.5*** (0.0) 34.3 (0.0) 0.6*** (0.0) 24.4 (0.0) 3.8*** (0.0) * p<0.05. ** p<0.01. *** p<0.001. Table4

Differences-in-differencesestimatesoftreatmenteffectsonoutcomemeasures.

Treatmentgroup Averagecorrectwordsperminute,bylanguage(standarderrorsinparentheses) Percentageofchildrenatbenchmark

English Kiswahili English Kiswahili

Studente-reader 6.1*** (1.4) 5.5*** (1.0) 12.2*** (2.9) 16.5*** (3.3) Teachertablet 8.5*** (1.5) 5.8*** (1.0) 16.2*** (3.0) 15.9*** (3.3) BasePRIMRTACtutortablet 9.9***

(1.7) 6.7*** (1.1) 18.3*** (3.4) 17.2*** (4.0)

Note:TheTaylorserieslinearizationmethodwasusedincalculatingthestandarderrors. ***

other treatment groups changed by less than 1%. Access to electricityalsoincreased,withthelargestgainsexperiencedbythe controlgroup.Bothoftheseindicatorscouldhavebeenaffectedby theruralelectrificationprogramimplementedbytheGovernment ofKenyain2013.Thesemodestdifferencesinthecovariatetrends suggestthattheassumptionsoftheDIDmodelhold.

5.Findings

5.1.RQ1:Whichofthethreetreatmentswasthemosteffective?

To answer ourfirst researchquestion, we fitDIDmodels to comparetheORFresultsofstudentsineachtreatmentgroupwith thoseofstudents in thecontrol group.We foundthat allthree treatmentgroupsimprovedstatisticallysignificantlymoreinboth EnglishandKiswahilithanthecontrolgroup.Theeffectsidentified overthe controlgroup in Englishwere 9.9 wordsfor thebase PRIMRTACtutortabletgroup(p-value<0.001),8.5cwpmforthe teachertabletgroup(p-value<0.001),and6.1cwpmforthepupil e-readergroup(p-value_<0.001),asshown inTable 4. Compar-isonsofKiswahiliscoresbetweenthethreetreatmentgroupsand thecontrolgroupalsofollowedthesamepattern.Thelargesteffect wasidenti_fiedinthebasePRIMRTACtutortabletgroup(6.7cwpm;

p-value<0.001); the teacher tablet effect was 5.8 cwpm (p-value<0.001), and thee-reader groupeffect was 5.5 cwpm (p-value<0.001).

Basedontheresults,wecalculatedCohen’sdeffectsizes.The effectsizeswerelargestforthebasePRIMRandTACtutortablet group(0.29SDinEnglishand0.32inKiswahili)andlowestforthe e-readergroup(0.17SDinEnglishand0.26inKiswahili).Theeffect sizesfortheteachertablettreatmentgroupwere0.26SDinEnglish and0.28SDinKiswahili.

We examined whether the PRIMR programs increased the percentages of students meeting the MoEST benchmarks for emergent reading fluency and comprehension. Recall that the emergent-readerbenchmarksforGrade2weresetat30cwpmin Englishand17cwpminKiswahili.Table4presentsourfindingson theimpactofthethree treatmentgroupsonthepercentage of pupilsreadingattheemergentreaderbenchmark.ForEnglish,the three groups increased the percentage of pupils reaching this benchmarkby18.3percentagepoints(p-value<0.001)inthebase PRIMR TAC tutortablet group,16.2percentage points (p-value

<0.001)intheteachertabletgroup,and12.2percentagepoints (p-value<0.001) in the pupil e-readergroup. In Kiswahili,the

treatmentgroupsincreasedthepercentagesreachingthe bench-mark(17cwpm)by17.2(p-value<0.001),15.9(p-value<0.001), and 16.5percentage points (p<0.001) for thebasePRIMR TAC tutor tablet group, the teacher tablet group, and the pupil e-readergroup,respectively.

5.2.RQ2:Whichofthethreetreatmentswasthemostcosteffective?

Thefirstresearchquestionaskedwhetherthethreetreatments had statistically significant impacts on English and Kiswahili literacyoutcomes.Forthesecondresearchquestion,weexamined thecost effectivenessof eachtreatment.Examining costin the contextofeffectivenessisessentialforthefieldofICTineducation, giventhewide variationin costsassociatedwiththemanyICT investmentsavailableinthesector.Tomeasurethecost effective-nessofthethreeinterventions,wefirstcomputedthecostofthe basePRIMRprogramandthenaddedtheunitcostoftherespective ICTcomponent.ThecostsofthebasePRIMRinterventionwerethe sameforallthreetreatmentgroups,astheyincludedthecostof pupilbooks,teachers_’guides, teachertraining,classroom obser-vations,andTACtutortraining.Thebaseunitcostwasthereforea ratioofthetotalcostforprogrammaterialsandtrainingdividedby thenumberofpupils.ThisbasecostwasUS$4.56forEnglishand Kiswahili,orUS$2.28perpupilpersubject,andwasthesamefor thethreeinterventiongroupscomparedhere.Thecostofasingle tabletatthetimeofpurchasewasUS$200,buttoaccountforthe rapidly reducingcost oftechnology,we utilizedUS$150for the purposeofthiscostanalysis.Ontheotherhand,thee-readerused costUS$70pere-reader,includingaccessoriessuchasaprotective case. To account for therapidly reducing cost of e-readers, we utilized $40asthecost ofthee-reader.Todetermine perpupil costsoftheICThardwarefortheTACtutortabletgroup,wedivided theICTportionofthecostsbythenumberofschoolsinazone(10) andbytheestimatedaveragenumberofpupilsinaclass(50).For the teacher tablet group, we divided the cost by the average numberofpupilsinaclass(50).IntheTACtutortabletgroup,the technologycomponent added US$0.10perpupil;in theteacher tabletgroup,thetechnologycomponentaddedUS$3.00perpupil. The e-readers added US$40 per pupil since the devices were providedtopupilsata1:1ratio.ThetotalcostsofthethreePRIMR Kisumu interventionsare shown in Fig. 2, which differentiates betweenthebasecostofthePRIMRprogramandtheadditionalICT costs. The unit cost of the pupil e-reader intervention was

significantlyhigherthanthoseoftheothertreatments,evenusing thelowercurrentestimatedcostofbasice-readers.

Costeffectivenessforthetreatmentgroupswasmeasuredby dividingtheeffectforeachtreatmentbytherespectiveper-pupil per-subjectunitcost,computingthegainsinORFoverbaselineper dollar.Controlgroupcosts werecalculated usingtheestimates identifiedinPiperetal.(2014).Fig.3showsthatthebasePRIMR TACtutortablettreatmentgrouphadthehighestimpactonoral readingfluency perdollar spent.ForthebasePRIMRTAC tutor tabletintervention,aninvestmentofonedollartranslatedtoan additional fluency of 11.6 cwpm for English and 6.7cwpm for Kiswahili.Theteachertabletgroupwassimilarincost effective-nesstothecontrolgroup.Costeffectivenesswasthelowestforthe e-readergroup,withthecontrolgroupaswellasthetwoother interventionsshowingmuchlargerORFgainsperdollarthanthe e-reader group. Stated another way, it was much more cost-effective toimplement theexisting control conditionthan the pupile-readerintervention.Thiswilllikelyremainthecaseevenas e-readerhardwarecostsdrop,as ourestimates useda reduced e-readercost.

5.3.RQ3:HowdidthetreatmenteffectscomparetothebasePRIMR programimplementedelsewhereinKenya?

To address this research question, we compared the EGRA resultsoftheKisumu PRIMRinterventiongroups withfindings fromthreedifferentversionsofthePRIMRproject.Thefirstwas the(non-ICT) PRIMR resultscollectedat theend of 2012 after

slightlylessthanoneyearofinterventioninperi-urbanlocations (Piperetal.,2014).Thesecondwasthe(non-ICT)PRIMRresults collected after 2 years of intervention in the same peri-urban locations(Piper,Jepkemeietal.,2015),andthethirdwasthe (non-ICT)PRIMRresultscollectedafter2yearsofinterventioninrural locations(Piper,Kwayumbaetal.,2015).Fig.4presentstheeffects onORFinEnglishandKiswahiliforthethreetreatmentgroupsof theICTPRIMRprogramcomparedwiththeeffectsonORFinthe samesubjectsfromtheotherthreestudies.

While the fact that the studies took place in different geographiclocationsatdifferenttimespreventscausalinference fromthesecomparisons,itprovidesevidenceastothepotential value-addedoftechnologyascomparedtothenon-ICTversionof thesameprogram.Ouranalysisshowedthattheimpactsofthe threetreatmentgroupsinKisumuwerewithintherangeofthe basePRIMRprogrameffects intheotherthreeevaluations. The largesteffectswereidentifiedafterthefirstyearofthebasePRIMR initiative. The smallest effect for English was in the student e-readerprogram,andforKiswahilithesmallestwasthe2-year effectofthebasenon-ICTPRIMRprogram.Inotherwords,there doesnotappeartohavebeenasystematicadvantagetoanyofthe ICT interventions over the non-ICT PRIMR programs, either in KisumuCountyorelsewhereinKenya.Someofthecomparisons actuallysuggestastatisticallysignificantadvantageforthebase PRIMR TACtutortablet program overthee-readerand teacher tabletprograms.Thissuggeststhatthere maybelimited value-added to an ICT intervention, and that it may in fact distract somewhatfromtheimplementationofthebaseprogram.

Fig.3.ORFgainsoverbaselineperpupildollarspent.

6.Discussion

ThisstudyaimedtodeterminewhetherICTinterventionscan produceliteracylearninggainsforstudentsintheearlygradesof primaryschool.PreviousICTliteracyprogramseitherhavebeen poorlyevaluatedor havefailedtoproduceimpacts onlearning outcomes. With the PRIMR ICT programs, we found that interventionsusingtechnologycanbeeffectiveiftheyareused ontopofawell-designedliteracyprogram.However,ourresults alsosuggest that there may belimited value-added tothe ICT component;the impact of the low-technology base version of PRIMRwassimilartothatofallthreeoftheICTgroups,intermsof improvementsin students’oral reading fluency in English and Kiswahili or gains in the proportion of pupils reading at the benchmarklevel.GiventhecostsoftheICTadd-onsinthiscase, particularlythee-reader,thecost-effectivenessofsuchprograms canbemuchlowerthanforthebaseprogram.

Despite thefindings showingthat studente-readersdidnot produce learning gains superior to the cheaper ICT options, e-readersarepromisingformanyotherreasons.Smallstudiesin the United States and Canada have found that children enjoy readingone-readers(Ciampa,2012;JonesandBrown,2011).They allowstudentsaccesstomanydifferentbooks,bothtextbooksand pleasurereadingbooks,atalowcost.IntheGhanastudydiscussed above, students had on average 107 books on their e-readers, including86thatwereautomaticallyloadedontothee-readerand 21 the students had selected themselves (ILC Africa and Worldreader,2012).InpartsofruralKenyawherebook distribu-tionisa logistical challenge,thisapproach hasthepotential to dramaticallyreducedistributioncosts.IntheKisumuICTstudy,the researchersalsofound that students in thee-reader treatment groupread18booksoverthecourseofthe2013academicyearand 70%consistentlyparticipatedinregularfreereadingsessionsinthe afternoons(PiperandKwayumba,2014).However,theseapparent successesin access to reading materials did not translate into significantlyimprovedstudentperformanceoverlessintensiveor costlyICT interventions.Thismaybe inpart becausestudents’

initialliteracy skills were sopoor that they did not get much benefitfromunscaffoldedreadingtime.Itispossiblethate-readers maybebettersuitedtoolderchildrenwhoaremorecapableof independentreading.Thisalignswithfindingsfromthestudycited aboveshowingthatchildrenwithstrongerskillsatthebeginning oftheprogramgainedmore(Kametal.,2009),andwithothers indicating that mobile phone e-reading is most popular with peoplewho aremore educated thanaverage in theircountries (West and Chew, 2014). It is possible that a more structured programusingthee-readers,givingthestudentsspecificactivities andsupport,wouldimproveresults(Wijekumaretal.,2012).

Aversionofthee-readerprogramevaluatedherewaspilotedin Ghana(ILCAfrica and Worldreader,2012)prior to theKisumu Countyexperiment,buttherelativelycomplexTACtutortabletand teachertabletinterventionswerenotpilotedbeforethis evalua-tion.Thetechnicaltoolsdeveloped fortheTACtutortabletand teacher tablet interventions were created for the _first time specificallyfor the Kisumu study. Largeroutcomes might have beenidentifiedifalltheICTinterventionshadreceivedathorough pilottrialbeforetheevaluationperiodbegan.TheTACtutortablet programwasutilizedatalargerscaleinthebasePRIMRprogram startingin2014(Piper,Kwayumbaetal.,2015).Resultsfromthat study,witharandomizedcontrolledtrialresearchdesign,showed largeeffectsonlearningoutcomes.Withayearofpiloting,anyof thetablet-based interventionsmight have shown largereffects duringthesubsequentevaluationphase.

The technology-intensive features of the teacher tablet programmayhavebeenadistractionforstudentsandteachers. Inthisstudy,extensivetrainingwasrequiredtoenableteachers

simplytolearntoturnthetabletsonandoff,andtologin.One technicaladvisorspent90minhelpinga teacherintheteacher tablet group to consistently remember her username and password.While this teacherdramaticallyimproved herability toutilize thetechnologyover the8 monthsof theKisumuICT intervention,therewasanopportunitycost:Thetimeandenergy spent to learn the technology could have been spent on instructional improvement activities. Finally, the teachertablet programwastheinitialtrialoftheTangerine:Classsoftware,which gaveteachersasignificantamountoffeedbackontheirstudents’

performance.Anecdotalevidenceshowedthatteacherswhodid usethe_findingsfoundthemuseful.However,asnoted,theremight beanopportunitycostintermsofinstructionalqualitywhensuch technologyrequiressignificanteffort.

Alackoftechnologicalexposure,includingunfamiliaritywith thecapabilitiesofthesetypesofdevices,mightalsohavelimited thevalueaddedbytheTACtutortabletprogram.Forexample,the TACtutorsdidnotappeartousethemoreadvancedassessment anddatacollectionfeaturesofthetabletfrequently.Oneofthetwo TACtutorsintheTACtutortabletgroupstruggledtoutilizethe applications on the tablets for the duration of the program, althoughtheeffectoftheprograminhiszoneremainedlarge.This challengemighthavebeenspecifictothatTACtutor,asthetablet programwasscaledupin2014tothebasePRIMRprogramandthe PRIMRtechnicalteamwasabletoconfirmthatTACtutorsusedthe tools consistently (Piper, Kwayumba et al., 2015). Anecdotal evidence fromthescale-up of theTAC tutortablet program as an addition tothe basePRIMR program in Kenyasuggests the potentialoftechnologytargetedatinstructionalimprovementto easetheloadofinstructionalsupervisorsandthereforeprovidea cost-effectivecontributionofICTtoimprovinglearningoutcomes (Piper,Kwayumbaetal.,2015).TheMoESTdecidedtoscaleupthe TAC tutortablet programbeyond the1384 PRIMRschools, and during2015,eachTACtutorinKenyawasprovidedatabletand training to implement the tablet-based instructional support system. Thiswas donetosupporttheTusomenational literacy program,implementedin eachpublicprimaryschooland 1000 low-costprivateschoolsfrom2015to2018.

ThePRIMRendlineevaluationincludedaquestionnaire admin-isteredtotheTACtutorsinthestudy.TheTACtutorswereaskedhow manytimestheyvisitedschoolsandclassroomsinthepastmonth. ThecontrolTACtutorsreported9.2visitstoschoolsand9.2visitsto classrooms. This contrasted with the responses of the PRIMR treatmentTACtutors,whoreported24.6visitstoschoolsand41.0 classroomvisitsinthepastmonth.Theseareself-reportedstatistics, andthePRIMRtreatmentTACtutorswouldhavebeenawareofthe social desirability of reporting that they visited schools and classroomsinPRIMR.However,itdidappearthatthethreePRIMR treatmentshadasignificantimpactonclassroomsupporttime.A simpleOLSregressionmodelrevealedthateveryadditionalreported classroom visitwasassociatedwithameandifferenceof1.0cwpmin English oral readingfluency, evencontrolling for assignmentto treatmentgroup.Itappearsthatthisincreasedattentiontoschools andclassroomswasanimportantmoderatorforthePRIMRICTand basePRIMReffectonlearningoutcomes.

ThefieldofICTineducationisexpandingitsfocusonmeasuring theimpactof theICT interventionsonlearning outcomes.This papershouldprovideevidenceforpolicymakersastheyconsider further ICT investments of various types. If any of the three programs analyzed here had been evaluated against a control groupalone,theresultswouldhaveindicatedanimpressiveeffect on learning outcomes that might have made them eligible for scale-uporincreasedfunding.Forexample,asimplerstudythat measuredonlytheeffectofthee-readertreatmentgroupwould havehadconvincingresultsoftheimpactofthee-readerprogram, larger than in previous e-reader evaluations (ILC Africa and

Worldreader,2012;Wagner, 2014).Countries havetaken multi-million-dollarprogramstoscalewithlessevidence.Thespecific contribution of this paper is to examine not only whether particularICToptionswereeffective,butwhethertheyweremore effectivethanotheroptionsthatwerelessexpensive.Thisisan essential question for this subfield of education, and one that requiresadditionalresearchbeyondthemodestcontributionsof thispaper.More importantly,giventhefailureofearlier ICT-in-education studies to link the ICT investments to instructional improvement processes, this paper presents the results of the additionalimpactofICTontopofanalreadyeffectiveinstructional improvementprogram.Estimatingnotonlywhethertheprogram worked,butwhetheritwouldworkbetterthanalessICT-intensive alternative,isanimportantcomparisoninthepolicyandfunding environmentsofmanysub-SaharanAfricancountries.

TheKenyaneducationsectorisrelativelywellestablished,and providesanannualper-childcapitationgrantofUS$14toschools everyyear.Eachschool’sBoardofManagementspendsthismoney asitseesfittoimproveeducationquality.Thecapitationgrantis often late and sometimes insufficient, but it gives individual schools some flexibility in how they allocate funding. Fig. 5

providessomecontextfortheICTexpendituresthatschoolscould consider in such an environment. A simple and enormously effectivechangewouldbetoassignthebestteacherstoGrade1, ratherthantotheendofprimaryschoolasmanyschoolsdo.This wouldhavenocost,butsignificanteffect.TheTACtutortabletcost perpupilforthePRIMRstudieswasmodest,atUS$0.10perchild peryear.ProvidingtheTACtutorwithsufficienttransporttovisit every teacher every month cost US$0.53. The purchase and transportcostsofaPRIMR(printed)bookweresomewhathigher, at US$0.75. An amountof US$3.00 per pupil was sufficient to provideateacheratablet,assumingthefullcostwasallocatedin oneyear,aswasdoneinthisstudy;spreadingthecostovertwo yearswouldreducetheannualcosttoUS$1.50.Studenttextbooks approved by the Kenya Institute of Curriculum Development (KICD)andalreadyinthemarketcostmorethanthis,atsomewhat lessthanUS$4perbook.Asanothercomparison,payingthesalary (butnotallowances)ofanadditionalteacherwouldcostUS$37per studentperyear,whichisslightlylessexpensivethanprovidingan e-readertoeachchild.ThisfigureshowsthatsomeICTcostscanbe minimal (the TAC tutor tablet, for example) while others are extremelyexpensiverelativetoschools’otherspendingoptions. This is highly relevant to the Kenyan government’s debate on whethertomoveforwardwiththeelectionpromisetoprovide laptopstoeachGrade1child,whichwillcostatleastUS$200per pupil,ascurrentlyconceived.Thisanalysisshowsclearlythatitis

essentialtoexamineactualcostsandopportunitycostsanduse themindecisionsrelatedtotheprovisionofICTineducation.

7.Limitations

The findings for researchquestions1 and2 aredrawnfrom differences-in-differences results, allowing an estimate that attemptstoshowthecausaleffectofthethreetreatmentgroups. Toanswerresearchquestion3,wecomparedtheimpactsofPRIMR inKisumuCountytothoseinothercountiesinKenya,includingthe basePRIMRprogramprovidingtabletsforTACtutors.Wecannot infercausalityinthecomparisonofthePRIMRstudiesoutsideof Kisumu County and the treatment groups in Kisumu County. However,astheKenyaPRIMRstudieshaveexpanded geographi-cally andover time,theresultshavebeenrelativelyconsistent: improvementsof4–12cwpmperacademicyearonassessmentsof ORFandprogrameffectsof0.2–0.5standarddeviationsonarange of learning outcomes (Piper, Jepkemei et al., 2015; Piper, Kwayumba et al., 2015; Piper et al., 2014). These effects are remarkably similarto what was identified in the one-year ICT study in Kisumu County, and make it more likely that the noncausalcomparisonisreasonable.

FewofthecomparisonsbetweentheKisumutreatmentgroups showedstatisticallysignificantdifferences.Thisisinpartbecause theimpactsonlearningoutcomesweresimilar,butalsobecause thestudywasunderpoweredtoundertakepairwisecomparisons andidentifystatisticallysignificantdifferenceswithsmalleffects. Additionalresearchinthisareaofcomparisonbetweentreatment groups will require a larger sample to have enough statistical powertoestimatetherelativeimpactoftheinterventions.

8.Conclusion

TheKisumuICTstudyallowedforacomparativeanalysisofthe impactofICTonlearningoutcomesinKenyawheninterventions took place at three different levels in the education system: student,teacher,andinstructionalcoach.Theresultsofallthreeof theKisumuCountyinterventionsshowedthatliteracyoutcomes canbeimproved.DespitegreatexcitementoverICTinterventions tosupportliteracy,however,technologyisnotacure-allforthe poor literacy outcomes in Kenya and other countries in sub-Saharan Africa. In this case, whereas technology mayhave helped TAC tutors to better provide instructional support, we foundnoevidencethatprovidingtabletstoteachersore-readersto studentswasmoreeffectivethanthebasePRIMRliteracyprogram that was implemented without expensive ICT.When costs are

considered,therearenon-ICTinterventionsthatcouldhavelarger impacts on learning outcomes with lower cost. Our analysis suggests that, for future interventions, tablets and e-readers shouldprimarilybeaconduitsupportingimprovedinstructional interventions.Asthebodyofknowledgecreatedbythisandother studiescontinuestogrow,we recommendthat future research focusonthenuancesofICTforclassroomliteracyinterventions (Wagner, 2014)—that is, what works best, in the most cost-effectivedesigns,forwhom,andinwhichcontexts.

Fundingsource

Implementation of the Primary Math and Reading (PRIMR) InitiativeinKenya(2011–2014)wasledbyRTIInternational.The programwasfundedbytheUnitedStatesAgencyforInternational Development (USAID/Kenya), under the Education Data for DecisionMaking (EdData II) Blanket Purchase Agreement, Task Order 13, Contract No. AID-623-M-11-00001. USAID/Kenya participatedinPRIMRdesignandimplementationdecisions,and also granted the authors permission to use the data set for additional analysis and dissemination. The lead author, Dr. BenjaminPiper,servedas USAID’sChiefof PartyonthePRIMR Initiative.Costsforpreparingthearticlewerecoveredjointlyby PRIMRandbyaninternalProfessionalDevelopmentAwardissued byRTIInternational.Thecontentsofthearticleanddecisionsabout publication were the sole responsibility of the authors. The authors’ views expressedin this publication do not necessarily reflecttheviewsofUSAID,theUnitedStatesGovernment,orRTI International.

References

Ciampa,K.,2012.Readinginthedigitalage:usingelectronicbooksasateachingtool forbeginningreaders.Can.J.Learn.Technol.38(2).http://www.editlib.org/p/ 42890/.

Cristia,J.P.,Ibarrarán,P.,Cueto,S.,Santiago,A.,Severín,E.,2012.Technologyand childdevelopment:evidencefromtheonelaptopperchildprogram.IDB WorkingPaperSeriesNo.IDB-WP-304.Inter-AmericanDevelopmentBank, Washington,DC.https://www.aeaweb.org/aea/2013conference/program/ meetingpapers.php.

Dubeck,M.M.,Gove,A.,2015.Theearlygradereadingassessment(EGRA):its theoreticalfoundation,purpose,andlimitations.Int.J.Educ.Dev.40,315–322. http://dx.doi.org/10.1016/j.ijedudev.2014.11.004.

Gove,A.,Cvelich,P.,2011.EarlyReading:IgnitingEducationforAll.AReportbyThe EarlyGradeLearningCommunityofPractice(Rev.Ed.).RTIInternational, ResearchTrianglePark,NC.http://dx.doi.org/10.3768/rtipress.2011. bk.0007.1109.

TheEarlyGradeReadingAssessment:ApplicationsandInterventionstoImprove BasicLiteracy.In:Gove,A.,Wetterberg,A.(Eds.),RTIPress,ResearchTriangle Park,NC.http://www.rti.org/pubs/bk-0007-1109-wetterberg.pdf.

Grace,J.,Kenny,C.,2003.Ashortreviewofinformationandcommunication technologiesandbasiceducationinLDCs—whatisuseful,whatissustainable? Int.J.Educ.Dev. 23(6),627–636.http://dx.doi.org/10.1016/S0738-0593(03) 00062-2.

Hanushek,E.A.,Woessmann,L.,2012.Dobetterschoolsleadtomoregrowth? Cognitiveskills,economicoutcomes,andcausation.J.Econ.Growth 17(4), 267–321.http://dx.doi.org/10.1007/s10887-012-9081-x.

ILCAfricaandWorldreader,2012.iREAD[ImpactonReadingofE-Readersand Digitalcontent]GhanaStudy:FinalEvaluationReport.USAID,Washington,DC. http://pdf.usaid.gov/pdf_docs/pnadz402.pdf.

Jere-Folotiya,J.,Chansa-Kabali,T.,Munachaka,J.,Sampa,F.,Yalukanda,C., Westerholm,J.,Lyytinen,H.,2014.Theeffectofusingamobileliteracygameto improveliteracylevelsofgradeonestudentsinZambianschools.Educ.Technol. Res.Dev. 62(4),417–436.http:/dx.doi.org/0.1007/s11423-014-9342-9. Jones,T.,Brown,C.,2011.Readingengagement:acomparisonbetweene-booksand

traditionalprintbooksinanelementaryclassroom.Int.J.Instr. 4(2),5–22. http://www.e-iji.net/dosyalar/iji_2011_2_1.pdf.

Kam,M.,Kumar,A.,Jain,S.,Mathur,A.,Canny,J.,2009.Improvingliteracyinrural India:cellphonegamesinanafter-schoolprogram.PaperpresentedattheThird InternationalConferenceonInformationandCommunicationTechnologiesand Development.

Kim,P.,Buckner,E.,Kim,H.,Makany,T.,Taleja,N.,Parikh,V.,2012.Acomparative analysisofagame-basedmobilelearningmodelinlow-socioeconomic communitiesofIndia.Int.J.Educ.Dev. 32(2),329–340.http://dx.doi.org/ 10.1016/j.ijedudev.2011.05.008.

Marteleto,L.,Lam,D.,Ranchhod,V.,2008.Sexualbehavior,pregnancy,and schoolingamongyoungpeopleinurbanSouthAfrica.Stud.Fam.Plann.39(4), 351–368.http://dx.doi.org/10.1111/j.1728-4465.2008.00180.x.

McEwan,P.J.,2014.Improvinglearninginprimaryschoolsofdevelopingcountries: ameta-analysisofrandomizedexperiments.Rev.Educ.Res..

Mugo,J.,Kaburu,A.,Limboro,C.,Kimutai,A.,2011.AreOurChildrenLearning? AnnualLearningAssessmentReport.Uwezo,Nairobi.http://www.uwezo.net/ wp-content/uploads/2012/08/KE_2011_AnnualAssessmentReport.pdf. Murnane,R.J.,Willett,J.B.,2011.MethodsMatter:ImprovingCausalInferencein

EducationalandSocialScienceResearch.OxfordUniversityPress,NewYork. Onsomu,E.,Nzomo,J.,Obiero,C.,2005.TheSACMEQIIProjectinKenya:AStudyof

theConditionsofSchoolingandtheQualityofEducation.SouthernandEastern AfricaConsortiumforMonitoringEducationalQuality(SACMEQ)andKenya MinistryofEducation,ScienceandTechnology,HarareandNairobi. Piper,B.,Miksic,E.,2011.Mothertongueandreading:usingearlygradereading

assessmenttoinvestigatelanguage-of-instructionpolicyinEastAfrica.In: Gove,A.,Wetterberg,A.(Eds.),TheEarlyGradeReadingAssessment: ApplicationsandInterventionstoImproveBasicLiteracy.RTIPress,Research TrianglePark,NC,pp.139–182.http://dx.doi.org/10.3768/rtipress.2011. bk.0007.1109.

Piper,B.,Mugenda,A.,2012.ThePrimaryMathandReading(PRIMR)Initiative: BaselineReport.PreparedUndertheUSAIDEducationDataforDecisionMaking (EdDataII)Project,TaskOrderNo.AID-623-M-11-00001(RTITask13).RTI International,ResearchTrianglePark,NC.http://pdf.usaid.gov/pdf_docs/ pa00hx75.pdf.

Piper,B.,Kwayumba,D.,2014.ThePrimaryMathandReading(PRIMR)Initiative. KisumuInformationandCommunicationTechnology(ICT)Intervention: EndlineReport.PreparedUndertheUSAIDEducationDataforDecisionMaking (EdDataII)Project,TaskOrderNo.AID-623-M-11-00001(RTITask13).RTI International,ResearchTrianglePark,NC.http://pdf.usaid.gov/pdf_docs/ pa00k285.pdf.

Piper,B.,Zuilkowski,S.S.,Mugenda,A.,2014.ImprovingreadingoutcomesinKenya:

first-yeareffectsofthePRIMRinitiative.Int.J.Educ.Dev. 37,11–21.http://dx. doi.org/10.1016/j.ijedudev.2014.02.006.

Piper,B.,Zuilkowski,S.S.,2015.TeachercoachinginKenya:examininginstructional supportinpublicandnonformalschools.Teach.Teach.Educ.47,173–183. Piper,B.,Jepkemei,E.,Kibukho,K.,2015a.Pro-poorPRIMR:improvingearlyliteracy

skillsforchildrenfromlow-incomefamiliesinKenya.Afr.Educ.Rev. 12(1). http://dx.doi.org/10.1080/18146627.2015.1036566.

Piper,B.,Kwayumba,D.,Oyanga,A.,2015b.ThePrimaryMathandReading(PRIMR) InitiativeDFID/KenyaRuralExpansionProgramme,BungomaandMachakos EndlineStudy.PreparedforDFID/KenyaUnderContract202657-108.RTI International,ResearchTrianglePark,NC.https://www.eddataglobal.org/ countries/index.cfm?fuseaction=pubDetail&ID=806.

RTIInternational,2013.MobileTechnologyforLearningAssessments.http://www. tangerinecentral.org/aboutsite(Webpage).

Roschelle,J.,2003.Keynotepaper:unlockingthelearningvalueofwirelessmobile devices.J.Comput.AssistedLearn.19(3),260–272.http://dx.doi.org/10.1046/ j.0266-4909.2003.00028.x.

Strigel,C.,Lang’o,M.,Kwayumba,D.,Koko,S.,2015.Casestudy:tangerine:classfor data-informedinstructionaldecisionmakinginKenya.In:Crompton,H., Traxler,J.(Eds.),MobileLearningAndMathematics:Foundations,Design,and CaseStudies.Routledge,NewYork,pp.198–212.

Trucano,M.,2005.KnowledgeMaps:ICTsinEducation.infoDev/WorldBank, Washington,DC.http://www.infodev.org/infodev-files/resource/ InfodevDocuments_8.pdf.

UnitedNationsEducational,ScientificandCulturalOrganization(UNESCO),2014. TeachingandLearning:AchievingQualityforAll.EducationforAllGlobal MonitoringReport.UNESCO,Paris.http://www.unesco.org/new/en/education/ themes/leading-the-international-agenda/efareport/reports/2013/.

Wagner,D.,Day,B.,James,T.,Kozma,R.B.,Miller,J.,Unwin,T.,2005.Monitoringand EvaluationofICTinEducationProjects:AHandbookforDevelopingCountries. infoDev/WorldBank,Washington,DC.http://www.infodev.org/articles/ monitoring-and-evaluation-ict-education-projects.

Wagner,D.,2014.MobilesforReading:ALandscapeResearchReview.USAID, Washington,DC.http://pdf.usaid.gov/pdf_docs/pa00jtgt.pdf.

Wasanga,P.M.,Ogle,A.M.,Wambua,R.M.,2010.ReportonMonitoringofLearner AchievementforClass3inLiteracyandNumeracy.KenyaNational ExaminationsCouncil,Nairobi.http://www.knec.ac.ke/main/index.php? option=com_phocadownload&view=category&download=182:std-3-abridged-report&id=62:std-3-abridged-report.

West,M.,Chew,H.E.,2014.ReadingintheMobileEra:AStudyofMobileReadingin DevelopingCountries.UNESCO,Paris.http://unesdoc.unesco.org/images/0022/ 002274/227436e.pdf.

Wijekumar,K.K.,Meyer,B.J.F.,Lei,P.,2012.Large-scalerandomizedcontrolledtrial with4thgradersusingintelligenttutoringofthestructurestrategytoimprove nonfictionreadingcomprehension.Educ.Technol.Res.Dev. 60(6),987–1013. http://dx.doi.org/10.1007/s11423-012-9263-4.

Winters,P.,Davis,B.,Carletto,G.,Covarrubias,K.,Quiñones,E.J.,Zezza,A.,Azzarri,C., Stamoulis,K.,2009.Assets,activitiesandruralincomegeneration:evidence fromamulticountryanalysis.WorldDev.37(9),1435–1452.http://dx.doi.org/ 10.1016/j.worlddev.2009.01.010.

Worldreader,2013.ImprovingE-ReaderDurability.Worldreader,SanFrancisco,CA. http://cdn.worldreader.org/wp-content/uploads/2013/11/Improving-E-reader-Durability.pdf.