Original citation:
Rawles, S., Joy, Mike and Evans, M. (2002) Computer-assisted assessment in computer
science : issues and software. University of Warwick. Department of Computer Science.
(Department of Computer Science Research Report). (Unpublished) CS-RR-387
Permanent WRAP url:
http://wrap.warwick.ac.uk/61239
Copyright and reuse:
The Warwick Research Archive Portal (WRAP) makes this work by researchers of the
University of Warwick available open access under the following conditions. Copyright ©
and all moral rights to the version of the paper presented here belong to the individual
author(s) and/or other copyright owners. To the extent reasonable and practicable the
material made available in WRAP has been checked for eligibility before being made
available.
Copies of full items can be used for personal research or study, educational, or
not-for-profit purposes without prior permission or charge. Provided that the authors, title and
full bibliographic details are credited, a hyperlink and/or URL is given for the original
metadata page and the content is not changed in any way.
A note on versions:
The version presented in WRAP is the published version or, version of record, and may
be cited as it appears here.For more information, please contact the WRAP Team at:
Software
SimonRawles
rawlesds.warwik.a.uk Department of Computer Siene
University of Warwik Coventry CV4 7AL
United Kingdom
Mike Joy
m.s.joywarwik.a.uk Department of Computer Siene
University of Warwik Coventry CV4 7AL
United Kingdom
Mihael Evans
Departmentof Computer Siene University of Warwik
Coventry CV47AL United Kingdom
February 4,2002
Abstrat
1 Introdution 3
2 About thereviews 3
2.1 TheWebsite anditsmaintenane . . . 3
2.2 Review methodology . . . 4
3 Tehnial issues 5 3.1 Installation . . . 5
3.2 Tehnologies. . . 5
3.3 Seurity . . . 5
3.4 Standards . . . 6
3.4.1 TheIMSStandards . . . 6
3.5 Strutureofmaterials,reuseandintegration. . . 7
3.6 Question banks . . . 7
4 Pedagogial issues 7 4.1 Sopeoffuntionality . . . 7
4.2 Waysoftakingtests . . . 8
4.2.1 Adaptivetesting . . . 8
4.3 Summativetestingand plagiarism . . . 8
4.4 Markingshemes . . . 9
4.5 Feedbakto students . . . 9
4.6 Feedbakto sta . . . 9
4.7 Question types . . . 10
4.8 RelevanetoComputerSiene . . . 10
4.9 DeeperlearningandBloom'staxonomy . . . 10
4.10 Studentautonomy . . . 11
5 Developer and Institution 11 5.1 CAAtoolproduers . . . 11
5.2 Qualityofinterfae . . . 11
5.3 Support . . . 12
5.4 Timeandmoneyosts . . . 12
6 Ideas forfurther work 12 6.1 Review exerise . . . 12
6.2 Topisin CAAresearh . . . 13
7 Conlusions 13
8 Dislaimer 14
9 Appendix: Summariesof the reviews 14
Onedenitionofomputer-assistedassessment(CAA)is`theuseofomputersin assessment',inluding examinationsandassignments. CAAsoftwareanbeusedtoautomatemanyoftheaspetsofassessment, whihanbroadlybedividedinto delivery,markingandanalysis.
ThoughCAA is oftenpromoted as anantidote to risingleturer workloadasa resultof inreasing student numbers, there are many reasons to use CAA tehnology beyond this. A key appliation of CAAis in formativeassessment,and in thissetting CAAsoftwareis apableof failitatinglearningby deliveringtests, automatially markingand providing appropriatefeedbak. In addition: progressmay bemoreeasilymonitoredthoughtheappliationoffrequenttests;studentself-evaluationandautonomy is enouraged; and the interative nature of the omputer may be employed to ondut new types of assessment,for example adaptivetesting. Theadministrativebenets inlude the redutionin human error,integrationwithinstitutionaldatabasesandthetimesavingsonethesoftwarehasbeenestablished andthematerialsprepared.
ThisreportsummarisesworkdoneinollaborationwiththeLearningandTeahingSupportNetwork fortheInformationandComputerSienes[18℄,oneof24subjetentressetupto oersubjet-spei expertise, best pratie and information on learning and teahing in the Information and Computer Sienes. The LTSN-ICS CAA review exerise [15℄ is an ongoing projet, the purpose of whih is to investigate a range of software urrently available whih have apabilities of assisting the assessment proess.
Inthis report, wesummarise this work by examiningboththe urrentCAA software available and going further to onsider its appliation in institutions of higher eduation. The software hosen is intendedtorepresentawiderangeofprodutsavailable,fromfreeaademisoftwarewrittenas proof-of-oneptineduationalresearhto ommerially-writtenvirtuallearningenvironmentsoftwareintended formassdeploymentin largeuniversities. Theemphasisin this exerisehasbeenonassessmentin the information and omputersienes, though manyof theproduts reviewed have been general-purpose question-and-answertools. We disuss software available in generalterms; individual software reviews areavailablefromtheprojet'sWebsite [15℄.
TheworkpresentedheredisussestheCAAsoftwareavailableattheendof2001. TheprojetWebsite [15℄iskeptup-to-date,andthereaderisinvitedtoonsultthematerialthereforfurther information.
2 About the reviews
2.1 The Website and its maintenane
The Website is seen as the main produt of the researh, and is intended to represent the urrent situation in CAA software through its reviews and assoiated information. The site an be aessed throughthe URLhttp://www.ds.warwik.a.uk/ltsn-is/resoures/aa/reviews/. Sine there aremanywaysin whih theinformationonthesite anbeomeout-of-dateorsuperseded,theWebsite ismaintainedaordingtoadened `updatingmethodology' whih ensurestheinformationis kept up-to-date.
Measuresemployedaspartofthismethodologyinlude:
generalinvolvementwiththeCAAommunity;
systematiaurayheksforeahrevieweverythreemonths;
ativeinterestin,anduseof,feedbakprovidedfromtheWebsite;
provisionof,andpartiipationin,moderateddisussionforumsontheWebsite;
amonthlyInternetsearhativityfornewCAAproduts;
Eah of the reviews are written using a standard struture whih aims to over most aspets of the software's tnessforuse. The strutureis onlyusedasaguideline, andis adapted whereneessary. It shouldbenotedthat,although there areobjetiveomparisonsof thefailitiesavailable in CAA tools, manyof the riteriaagainst whih thetools havebeenmeasured are subjetive. On this aount, the reviewsshould be treatedwithaution. Inpartiular,omparisons in thereviewsand theomparative summarytablesreettheauthors'opinions.
Thereviewstruturemaybesummarisedasfollows:
Basiprodut information. Thetitleandpublisherofaprodut. TheURLoftheprodut'sWebsite. A short2{3line desriptionoftheprodut,overingitsmain features orappliations. The avail-abilityandlieningsituationfortheprodut. Whih platformsthesoftwarerunson|inluding lientandserverrequirementsorplugininformationwhereappliable.
Overview. A breakdownoftheomponentsin thesoftware. Alist ofweak pointsandstrong features ofthesoftware.
Capabilities. Thetypesofassessmentsupported withreasonswhytheyaresuitableorunsuitable.
Studentexperienes. Theease of useof the softwareand howmuh priorIT knowledgeis required touse it. Stabilityofthesoftware. Whetherthesoftwarehasbeenonlytrialledorfullydeployed. Eetonstudyorlearning. Formof feedbakanditsusefulness. Degreeofremoteaess. Results oftestsforHTMLorretness. Generalfeelingsabouttheassessment.
Staexperienes. Nature of leturer feedbak. Usefulness and appropriateness of the statistis and reporting. Degreetowhihanup-to-dateviewoftheassessmentsituationanbepresented. Extent towhihthedeliveryoftestsanbeontrolled. Robustnessofthesoftware. Integrationwithother toolstheleturermightuse. Easeofuse,partiularlyin questiondesign. Previewingofquestions. Degreeofpriorknowledgeneessarytoauthor,suhasspeial markupsorlanguages. Supportfor reuse of materials and question banks. Flexible speiation of question forms. Eort required for upkeep of the software ompared with traditional methods. General feelings on authoring, distribution,delivery,markingandadministration.
Appropriatenessfor deployment withininstitutions. Upkeep requirements for sta and depart-ments. Installationproess. Eet onstudentreruitmentand distane learning. Quality ontrol issues. Possibilitiesof entral olletionand proessing of assessmentdata. Whether the deploy-mentofthesoftwareislikelytobesmoothandeÆient. Salabilityofthetehnologyandthekind ofsalethatthesoftwareis best usedat.
Integration with related methods. Thissetionisabouthowwellthe softwareanbeusedin dier-ent learning situations. Integration withomputer-aided learningsoftware, optial mark reogni-tion. Appliationtogroup-basedwork. TypesoftasksthesoftwaresupportsonBloom'staxonomy. Supportforstagedquestions|anlaterquestionsbeseletedfromearlierones? Randomisationof questions. Support forfreetextquestions andtheextenttowhihtheyanautomatiallymarked. Ease of adding new question or assessment types. Ability to dene and exibility of marking shemes. Abilityforstudentsto submitbinarylesasresponses.
Support from the originatingorganisation. Quality of doumentation. Whether the software is ativelymaintained and frequently upgraded. Extentof maintenane ommitment. Existene of on-siteexperts andtheireetiveness.
judgementforthoseomparingproduts.
Additionally, thereviewsare summarisedin aseries oftables, with rowsshowingdierentproduts andolumnsshowingfeatures ofproduts. Thetablesomewith theirownexplanations,butgenerally, featuresareshownbyonelled oremptyirle,intheaseoffeaturesthat areeither presentorabsent, orbythreeirlesrepresentingaratingfrom0to3,rangingfromabseneorunsuitabilityforapartiular featuretototaloverageorexelleneinthatfeature. Tablesareusuallylledinafterareviewisomplete andshouldreetthejudgementsmadein thereview.
3 Tehnial issues
Thissetiondisussesthemoreoperationalandtehnial aspetsofthetools.
3.1 Installation
Most of the produts were relatively straightforwardto install, though aommon problem, espeially in respet of software developed at aademi institutions, is that software is typially not pakaged ompletely. In partiular, software whih is intended to be ross-platform is toospei to partiular mahine ongurationsoroperatingsystems, andrequired smalledits to getworking. Inthease ofa fewproduts,installation wasratheraninvolvedproess, takinghoursfor usersskilledwith omputers toaomplish. Those produtsmaythereforerequireinstallationbysystem administratorsratherthan theleturersthemselves.
3.2 Tehnologies
Manyofthenon-ommerialprodutsreviewedusedestablishedandoftenopentehnologieslikeJava[27℄, Perl[21℄ortheApaheWebserver[1℄. Asnewtehnologiesemergeandexistingtehnologydevelops,CAA software isoftennotupdatedto takeadvantageofthese, andCAA thereforeoftenlags tehnologially. Themostommonly-supported operatingsystemsonwhih theprodutsrun are thenewerversions of Windowsand the various avoursof Unix. However,asigniantnumber of theommerial produts operateaspaid-for serviesontheWeb.
Deliverytehnologiesvary,thoughunsurprisinglytheWorldWideWebis apopularhoie,allowing assessmentstobeeasilydistributedthroughitsalmostuniversalpreseneontheusers'desktops. However, otherprodutsuseustom-implementedprograms,oftenaspartofalient-serverapproah[16,17℄. Other produtsmakediret use of third-party software. For instane, the Authorwaremultimedia authoring tool[20℄is usedto drivetheTRIADS[19℄ assessmentengine,and isdiretlyused toeditquestions and otherwisesetupassessments.
3.3 Seurity
Seurityisanaspetoftheprodutswhihhasgenerallynotbeenonsideredindesign. Forsomeproduts, thisis simplybeause theyarenotintendedfor anythingother thanfor purposesofself-testing. Many produts, however, laim that they an be used for diagnosti oreven summative testing. Sine this requires theauthentiationof students andthe storageof their results, aswell asmeasures to prevent onestudentfromintereptinganother'ssubmissionand otherommuniation,thisouldunderminethe integrity of the assessment proess. Enryption and other seurity measures are not ommonly used. Provision of adequateauthentiation is somewhat easier, though in some ases passwordswere not as loselyguardedbytheprodutsasmightbeexpeted.
problems.
Somesoftwaredoesaddressseuritywell,notablyNottinghamUniversity'sCoursemastersystem[16℄, whih enrypts traÆ betweenthe administrator and theserver, and uses aross-hekingsession key systemtoprotetstudents'ommuniationswiththeserverfrompaketsniÆng. Othersoftwareappears to reah a given level of seurity, for example aepting passwords as part of the login proess, but ompromisestheseurity byinorporatingthepasswordsin theHTML soure ofeah page served. A similarproblemexistsinsomeprodutsinwhihtheanswerstothequestionsthemselvesanbereadby simplyviewingthesoure,but thisisnottypiallyanissuewithmoreapable,ommerialsolutions.
3.4 Standards
Adoptionof standards,suh astheIMS's Questionand Test Interoperabilitystandard(IMS-QTI)[12℄, does not appear to be a priority for most of produers of the CAA produts reviewed, and there is littleinformation in mostasesto suggestthey areonsidering onformane, thoughafew ommerial produersare reognisingthe standard and suggesting future software will use the standards to some extent. It appears thatthe rangeofquestionformatsisbeoming morediverse. Indeed, insomeases, IMS standardshavebeen extendedin anad-homanner, suggestingthebasiIMS standardis neither omplete enough nor exible enough for the needs of developersof the moreomplex produts. This trendseemslikelytoontinue,asIMSstandardslagbehindtheproduts'apabilitieswhihtheyaimto over.
However, markup has been used to provide a strutured and delivery-independent approah to a numberof projets. In addition to the use of IMS standards [12℄, the WebMCQ [7℄ system is partly motivatedbyawideragendafortheorganisationofteahingmaterialsbasedonamarkuplanguage,and BristolUniversity'sTMLsystem[13℄hasbeenestablishedforseveralyearsasausablemarkuplanguage forquiz-likeassessments,andissupported bytheNetquestdeliverysystem.
3.4.1 The IMS Standards
An industry standard struture [12℄ has been developed for the representation of question and test data, failitating the use of these data between dierent produts. The standardhas beendened by theIMSGlobalLearningConsortium,omprisingeduational,ommerialandgovernmentorganisation members. Thisstandard,theQuestionandTest Interoperability(QTI)speiation,isdenedin XML, theextensible markuplanguage.
Thestandarditself,atthetimeofwriting,isatversion1.2,apublidraftspeiation. Itsstrutures fordeliveryand response allowthe onstrutionofquestions fromaset ofstandardquestiontypesand theolletionandsoringof responses. Thestandardisrelativelylarge,with welloverahundredXML elementsoveringmainly thefollowing:
speiationsforquestion-and-answerdataandthepakagingofsuhdata;
informationaboutproessingresponsesforassessment(soring,feedbak,et.) andreportingthese results;
othermeta-data;
high-levelpresentationalinstrutions(thoughnotspeiationsforaspeiuser-interfae);
thedenitionofexternalAPIsforproduts.
IMS have also developed QTILite, aut-down versionof the main speiationwhih, amongst other things,onlydealswithmultiple-hoiequestions.
intheseareassessmentmethodssuhasomputeradaptive testing,in whihdiÆultyisadjustedbased onthe urrentestimate of theexaminee's ability, orsimulated ases in whih, for example, a medial situationisdesribedandquestionsleadexamineesthroughdierentsituationsthatmayourbasedon theirresponses. However,itisstilltheasethat theonditionalseletionofquestionsbasedonoutome responseisomittedfromthestandard. Theabilitytodenepreonditionsandpostonditionsforsetions anditemsisalso leftforfurtherstudyandinlusionin futureversionsofthestandards.
3.5 Struture of materials, reuse and integration
Thestruture, orotherwise, of learningandassessment materialhas animpat onhowit is used, and perhapsmoreimportantly,reused. Thedevelopmentofstruturesandmarkuplanguagesforsuh mate-rialshasbeenthesubjetof severalprojets,eahwith divergentresults[28℄. Theinterestin struture andreuse prompts thequestionof to what extentorganisationof materials exists in CAAsoftware. A relatedissueistowhatextentareassessmentandlearningmaterialsintegrableintopresentationmedia, suhastheWeb.
There appearsto belittleprovision forreuse andintegration in mostofthesoftware, theexeption beingtheVirtualLearningEnvironments(VLE),whereitisoftentheasethatlearningandassessment materialshavesomeoneptofstrutureinthattheyarepresentedonseparatepagesormodules. Most produts whih provide a basi level of material organisation allow questions to be stored and reused innewassessmentsthroughrefereneto alloated IDnumbers. TheIMS-QTI standardsandstandards assoiatedwith WebMCQ[7℄detailamultilevelhierarhy,in whih there arelevelsforourses,bloks, assessments,setionsandquestions. Meta-dataisalsowell-developedinthestandards,thoughboththe oneptofhierarhyandmeta-dataisofteninadequatelyrepresentedexpliitlyin produtsthemselves.
3.6 Question banks
There has been muh interest in aquestion-bank approah to assessment for CAA software. That is, software whih failitates thereationand useof aset of questions,either sharedbetween assessments or between leturers. This onept does not feature muh in the tools reviewed, perhaps beause of therelativelylowpenetrationofstandardsinto theproduts. A signiantproblemwiththe proessof testingusingCAAisthebuildingofquestions,whihistime-onsumingandsurprisinglyhardtodowell, and thesope to organise questions,and ultimatelyto share them, would enourage the reationof a resouretobeusedyear-on-yearandtoshare witholleagues.
4 Pedagogial issues
Thissetiononsiders thetoolswithrespettohowtheyaredeployedforusein eduation.
4.1 Sope of funtionality
The intended apabilities and settings for the produts vary signiantly. Some produts, for exam-ple theTACO system [25℄, are intended only assimple multiple hoie Web interfaesfor inlusion in Web-deliveredourseworkmaterial. Others areintended asfully-featuredquestion-and-answersystems addressingmanytypesoftesting,andthereis awidespetrumoffuntionalitybetweenthese.
Mostprodutsreviewedarereasonablyfoussedonomputeraidedassessment,butsomealsoaddress otherareasthatanbeoveredbythetools. Other produtsfeatureomputer-assistedassessmentonly aspart of a wider produt. Commerial produts seem to be gravitating towards a Virtual Learning Environment, intended as a `virtual university in a box'. Consequently the fous shifts from CAA towardsa broad general-purpose eduationalsolution, losingmany of the advaned apabilities whih ourseorganisersmayrequire. SomeCAAprodutsarefoussedaroundpartiularaademigoalsrather thansimplyassessment;forexampletheTMLsystem[13℄attempts toestablishamarkupfor question-and-answerdatafrom thedaysbeforeIMS standardshadbeomewell-dened.
aademispin-oompanies,andto someextenthavetheadvantagesofbothommerialandaademi pratie.
4.2 Ways of taking tests
Thesearerelativelybasi,andgenerallylakanyformofonditionalquestioning. Thissigniantlylimits theirusefulness forformativetesting, anarea in whih theuse of aomputer would seem partiularly appropriate. Someproduts deliverusing asinglepage withasequeneof questionsand onlyallowfor settingsimplesetsofquestions.
4.2.1 Adaptive testing
Thoughtheprovisionof feedbakguides thelearner to someextent, itdoes notexploit theinterative natureoftheomputertothesameextentasdoesadaptivetesting. Adaptivetestinghasbeendenedas amethodofassessmentwheretheseletionoflaterquestionsdependontheresponsestoearlierquestions [22℄. It an arguablybe used asan aidto aurate assessment, asin the Amerian GraduateReords Examination(GRE)[9℄,whereorretanswersleadto progressivelyharderquestions andinorretones toeasierquestions. Adaptivetesting anbeapplied in theassessment-as-learningsetting asapowerful feedbak reinforement mehanism. One way in whih this ould beemployed might be to ategorise questions andthen trakthose subjetsor questiontypesin whih astudentis repeatedly failing, and present more of those questions during formative or pratie tests. Another way might be to employ GRE-stylediÆultyadjustmenttoensurethateverystudentisstrethed, irrespetiveoftheirability.
Some aspets of non-adaptive testing are lost, for example the ability to step bak through a test andhangeanswersto previousquestions,sinetheyhavebeenusedtodeterminefutureones. Another onsiderationisthatthequestion-settermusttrynottosetquestionswithanydeliverysequeneinmind, sine|thoughsequenes arepossiblewithadaptivetesting| theyredue thedynami nature ofthe questionseletion.
Adaptivetestingis anunder-representedoneptin bothCAA toolsand thestandardsthatspeify CAAquestioningformats. Asmallminorityoftools[8,7℄allowforonditionalquestioning,thoughthis oftentakesaratherunsubtleform,wheretheoutomeofsimpleonditionaltestsbasedonrangesofsores attained sofar aets the seletion of thenext setion, whih typially onsists of alarge sequene of questions. Asisdisussedlater,theIMSstandardsdonotyetoveradaptivetestingandtheonditional seletionmehanismsrequiredforit.
Generally speaking, theomputer has notbeensigniantly exploited asan enablerof new assess-ments,ratherithasbeenusedtoimplementtraditionalassessment.Featureswhihexploittheomputer's interativenature,suhasadaptivetestingandothertypesofguidedlearning,peerreviewsystemsand soon,aresurprisinglyrare.
4.3 Summative testing and plagiarism
Summative and diagnosti testing are two of the areas in whih omputer-assisted assessment brings signiant benets over traditional methods and are therefore desired appliation areas for leturers usingCAA tools. However,summativetesting hasseveralimportant requirements, stemmingfrom the neessitytoobtainanaurateset ofresults,andthereforetoredue thelikelihoodofheatingasmuh as possible. Plagiarism detetors may be employed after results have been olleted, but there is a strongargumentfor hoosingsoftwarewhihredues thelikelihood ofplagiarismin therstplae. The software thereforeneedsto addresstheissuesof impersonation,obtainingresultsbyomputerhaking, and`over-the-shoulder'opying:
byallowingstudentstoauthentiatethemselvesadequately;
bysupportingomputerandnetworkseurity;
overomingseurityonernsandthewaysinwhihabasiWebbrowseranbemisusedtoheat. Most produtsarein somewaysuitablefor self-testingand formativetesting, with thelatterrole beingthat in whih they are mostuseful. A general lakof adequatereporting tools meansthat interpreting the resultsofdiagnostitestingismoreonfusingthanitshould be.
4.4 Marking shemes
Theuseofmarkingorkeepingsoreisalmost universalamongCAAtools. Apointsvalue,orweight,is assoiatedwith eah response,and thesore awardedis thetotal of these. Most produtsallow hints, andthosethat dousuallyallowapointsvalueto besubtratedforthehintbeinggiven. Generally,the abilitytodenehowmarksareawardedoversetquestionsisfairlyexible,thoughtheyareusuallyseton aquestion-by-questionand response-by-response basis. Some produts gofurther, allowing ondene-basedassessment[25℄. Othersallowforthegroupingofmarksintogrades,givingaquikerinterpretation of a student's progress. Software whih assessesprogramming ability is, of ourse, based on dierent riteria, andCoursemaster allows markingbased on typographifeatures, languagefeatures aswell as orretnessin tests. A handful ofsystems, of whih Coursemaster is one, allowthegroupingof marks intogrades. Coursemaster alsoattributes aolourto eahgradeto makeWeb-reportedstatistismore easilyunderstandable.
4.5 Feedbak to students
Thepossibilityofinludingtimelyfeedbaktoquestionsallowstheomputerassistedassessmenttoolto beused forlearningratherthanmerelyassessment[10℄. Indeed, itisoftensaidthat theharateristis of CAA tools make them best suited to the pratie of regular and frequent formativeassessment for learning[5,26℄. Repeatableandaessibletestswithrihandexplanatoryfeedbakanomplementand reinforedeliveredmaterialeetivelywithlittleteahereortbeyond thereationofthetest[24℄.
Manytoolsavailableprovidesuhfeedbak,thoughoftenitisverymuhbasedonthedeliveryofstored textforeahpossibleresponse.Thisinonjuntionwithasoresofarorothersimplestatistispresented attheend ofatestisoftenallthat CAAtoolspresent. Giventhisrigid framework,thefeedbakgiven alwaysneedstobewellphrasedandneedstotakethetool'sfeedbakstrutureintoaount. Clearly,the feedbakprovidedannot be asexibleasthat from properdialoguewith aperson,thoughtheability todene feedbakon anythingother thantheresponse-levelis laking. There is generallynoprovision for the delivery of feedbak whih spans several questions,for exampleto give ageneral indiation of topisin whih thestudentis (onsistently)doingpoorly,thoughsometoolsallowstudentsto seetheir reentperformaneinaourse. Feedbaktostudentsisaweakarea,alearmissedopportunityforCAA, thoughitmaynotbeomefeasibleuntilshemesforquestionmeta-dataaredevised.
4.6 Feedbak to sta
Anotherimportantkindoffeedbakintheeduationalproessisthatwhihtheleturerreeivesinthe formofsummaries oftheresultsof theassessment,forexamplestatistissuh asaveragemarkandthe spreadofmarks overthe lass. This is anareain whih muh of thesoftware islaking, and againan areain whihomputersouldsigniantlyimprovetheeduationalproessbyvisualisingthe progress oftheirlass.
pretablefeedbaktoleturers[24℄. Thisouldallowleturersto weedoutproblem questionsoridentify areasof aoursewhihare notbeingunderstood, rather thanjust identifying studentswhoare under-performing. Thistypeoffeedbakisespeiallyvaluablesinewhatisbeingmeasuredhangesovertime [22℄,andatimelyreationouldimprovetheunderstandingofurrentandlatertopissigniantly. The inorporationofbreakdownsbysubjetortheuseofsoringgridsmightbeonewayinwhihthisould be ahieved [10℄, though this depends on the leturers ability to set appropriate questions as well as thesoftware's apabilities. Good feedbakis a ombinationof the produts apabilities and theuser's assessment-setting ability | good assessment and question design and meaningful feedbak are often related.
4.7 Question types
Akeyaimofmostomputer-assistedassessmentsoftwareistosupportlearningativities[10℄. Generally the question types oered are rather simple, though a number of tools have more exible provisions for extending questiontypes. Multiple hoie, multiple-response and ll-in-the-blanks questioning are typially provided, but often only afew question types beyond these exist. Thedialogue betweenthe student and the toolan often be rather shallow | the tools do notnaturallylend themselves to the typesofquestioningwhih would invite moresophistiatedthinking andlearningin students[22℄. The useof omputershas, however,enouraged newer questiontypes,suh asondene assessment,to be formulated,aswellasthevariationofquestionsbyrandomisingnumbersorhoosingdierentdistrators. However,bothof these areunder-representedin the tools, even thoughrandomisation is anadvantage foralass-basedomputerassessment.
Some produts are not oriented toward multiple hoie questioning, and oer dierentquestioning asaresult. Computerprogramming[14,16℄ isonemain fous. Diagrammingisoneinterestingareaof extension. Oneprodut[6℄hassimplehemialdiagrammingbuiltin,andCoursemaster'sdiagramming extensions[29℄areaveryexibleandapablesolutiontodiagrammatitesting.
Computerprogrammingassessmenttoolsareingeneralmoreapable,perhapshavingbeenmotivated bymorespei requirementsandlearninggoals. Coursemaster's[16℄ approah ofallowingtheuser to dynamiallylinkmodulestomarkresponsesprovidestheJava-apableleturerwithaveryexible solu-tion,enablingthemarkingsystemtobeextendedsigniantly,andlimitingthepossibilitiesofassessment tothat whihan bedonegenerallywithaomputerprogram, whilestillstayingwithintheframework oftheCoursemastertool.
Ativelearning,stylesofassessmentwhihaskfortextanddiagrams,andothermethodsofengaging the student in the response are reognised to be eetive forms of assessment-basedlearning. These tooareunder-representedin thetoolsprovided,possiblyduetothemoreomplexdevelopmentofthese questiontypes.
4.8 Relevane to Computer Siene
Manyofthe produtsreviewedhavenospeialbiastowardtheassessmentofComputerSiene, being general-purposequestion-and-answertools. Somenotableexeptions,however,aretheCoursemaster[16℄ andBOSS[14℄ systems,amongwhose apabilitiesarethetestingofprogrammingassignments.
4.9 Deeper learning and Bloom's taxonomy
Bloom'staxonomy[3℄haraterisesquestionsbytheirlevelofabstration,representingtheompetenies andativitiesinvolvedinansweringthem. Thoughquestiontypesoftenenouragequestionsbelongingto apartiularategorisation,agivenquestiontypewilltypiallybemoreappropriateforagivenrangeof dierentlevelsofabstration.Thoughmanyofthetoolsareabletotestthelowerlevelsofthetaxonomy (knowledge,omprehension,appliation),andthoughsomequestiontypesbetterenouragehigherlevels ofativitythan others,the letureroften needsto be skilledin questionsetting and designin order to evaluatethehigherlevels(analysis).
omplexsoringrubri[22℄,apparentlylimitsCAA softwaretoproessingthistypeofanswerbysimple tehniques suhas pattern-mathing ofinput. Thoughtehnologyis unlikelyto beabletodeliversuh assessmentin thenear future, it remainspossible that the appliation of CAA will beable to replae someassessmentstrategiesin partiularsubjetareas. Theprovisionforgroupwork islaking,though someinterestingsystemshavebeendevelopedbasedonpeerassessment,forexampleOASYS[2℄.
4.10 Student autonomy
Therapidinreaseinstudentnumbersoverreentyearshasinpartledtoahangeintheroleofstudent and leturer in the higher eduation environment. There has been a shift toward the student taking responsibilityfortheir ownlearningandtheleturer providing thelearningenvironmentneessaryand failitatinglearning[5℄. Indeed,self-evaluationisnowregardedasanimportantlifeskillwhihshouldbe enouragedaspartoftheeduationalproess[24℄. Self-evaluationisfailitated byanumberoffeatures insoftware. Provisionforastudenttoviewtheirprogressintheontextofthemodule|andultimately theourse|atsasbothanindiatorofweakareasandasamotivator. Someofthesoftwarereviewed, partiularlythosein the`virtuallearningenvironment'ategory,givethiskindofoverview,thoughthis isasfarasthepromotionofself-evaluationgoes.
Relatedtothisistheideaofstudent-entredlearningandthehangeinemphasisleadingtostudents havingmoreontrolovertheirlearningthanwouldbedonetraditionally[24,5℄. Student-entredlearning mightinludetheabilitytoundertakepratietestsbeforeundergoinganassessedone(orrepeatingtests untilagivengradeisattained). EventhisrelativelysimplemeasureisrarelysupportedinCAAsoftware. Student-entredlearningalsoisfailitatedbytheopennessoftheassessment,forexampleminimalserey of therubri and riteria for assessment[22℄, aswellas easeof aess. In these respets, muh of the reviewed software makes good provision. By making the assessment more open and less hidden and subjetive, the proessesof learning aremore visible; the student knows better the aims ofthe ourse andhowbest to undertaketheir ownlearning, and andeterminewhether oneptshave been learned and whih are yet to be learned[24, 4℄. Computer-based assessmentis often saidto makethe proess oflearningmorevisibleto thestudent[22℄,and thereforeenouragesgreaterautonomy,thoughexpliit eortstomakelearthelearningproessin thesoftwareareoftennegleted.
5 Developer and Institution
Thissetion disussesthetoolsfrom theviewpointsof the software developerand theuserinstitution, andtherelationshipbetweenthem.
5.1 CAA tool produers
WithCAAtoolproduerseitherbeingompaniesoraademiinstitutions,thereisanassoiationbetween the quality of a produt and its ost. Aademi produts are generally free, or heap, but are weak in support, where the level of input and ontinuing updates is onerned. Companies all harge for theirproduts,but thesetend tobemorepolished, arebettermaintained,andprovidemoreonsistent support. A numberof produtsomefrom spin-oompaniesfrom aademidepartments,oftengiving somepedagogialrelevaneto thetool.
5.2 Quality of interfae
Thequalityof support provided isdependent onthesoureof theprodut. Commerial produtsoer awiderangeofhelp frominstallationthroughtoadvie onquestionreation,butat aprie. Aademi institutionsarenotofteninaposition toprovidemorethaninformalsupport,buttypiallyarefriendly andwillingtohelpviatelephoneande-mail.
In many ases support onsists of supporting the produt only; assoiated learning materials are neitherprovidednorseemavailableanditisupto theourseorganiserto reatethem. A fewproduts havean emphasisonmaterial aswellassoftwareandbothselladditionallearningmaterialmodules as well as providing an infrastruture for the distribution of materials from textbook authors and other ontentproviders.
5.4 Time and money osts
Computer-assistedassessmentisoftenintroduedoutofadesiretoountertheinreasingtimedemands on sta while still delivering an aeptable eduational experiene. It should be onsidered, however, thatthisappliationarriesostsof itsown, bothintermsoftimeandmoney.
The most time-intensive stage of using CAA is when it is being introdued | long-term gains in eÆieny are oftenat aost of short-term eort[10℄. A majorproblem liesin thefat that materials needtobepreparedfordeliverybytheomputer. Thelakofadoptionofstandardsandtheonsequent lakof modularity leadsto major diÆulties in reusing materials, whih means that materials need to be onstruted from nothing, rather than being generated. There are also possible`hidden delays' in installation,upgradeandupkeepproedures,deadlineextensionandmarkadjustmentontheomputer, generalsystemfailuresandollationofneessarystatistisandtheirtransfertoaentraldatabaseinthe abseneofadequatereportinganddatabaseonnetivityfailities.
As with most software solutions, there are also nanial osts involved in applying CAA software. In addition to initial osts, some of the software requires third-part software. Many of the suppliers hargeupgradeandsupportosts. Therearealsoostsassoiatedwiththeuseofthird-partyassessment materials.
6 Ideas for further work
ThissetiondisussespossiblefuturediretionsfortheCAAprojet,rstlyforthereviewexeriseitself, andthentopisforlarger-saleCAAresearhwhihhavebeenpromptedbytheLTSNwork.
6.1 Review exerise
Further investigation of the pedagogy of omputer-assisted assessment, for example its role in learning.
Survey work to gain a better idea of the requirements of Computer Siene leturers in higher eduation,toatas asetof newriteriaagainstwhihtoevaluatesoftware.
A study of how CAA is used in higher eduation, partiularly omputer siene, to get abetter ideaof patterns of use, best pratieand areaswhih softwareneeds to address. Partiularly,in whih partsofthedeploymentofCAA softwareisthemosttimespent,and whatmeasuresmight betakentomaketheproesslesstime-onsuming?
Asurveyandsummary ofanti-plagiarismsoftwareandtehniques.
Further investigation into thepotentialvalueof onditionalquestioningand adaptivetesting and thedegreetowhihitissupported inCAAtools.
Astudyinto theusagesofCAAforsummativeand diagnostitestingandwhether CAAis intrin-siallyprimarily ofuseasaformativetool.
aeptableminimumlevelofseurityprovision.
6.2 Topis in CAA researh
A rethinkofthe formof feedbakgiven tousers. Atpresentfeedbakislimitedin itsform. How ouldfeedbak beextendedto beof moreuseto thedisparaterolesof studentandleturer, from apedagogialperspetive?
InvestigationofthestrutureandreuseofCAAmaterials,andwhethertheIMSstandardmarkup isappropriatefor`realworld'use. Thisisstronglyrelatedtoquestionbanksandtheirorganisation anduse.
A study into how omputers and CAA might be better applied to group work, and the degree of support theyalready give. Forexample, doVLEs provide thepedagogially-soundgroupwork supporttheypromise?
Extendingworkdoneinspeiassessmentsituations,forexampleomputerprogrammingorother subjetdomains,orwideningtherangeofquestiontypes,suhaswiththediagrammingwork.
Making provision for deeper learning. The dialogue between the student and omputer is often shallowinnature,andthetaskswhihthestudentsareexpetedtodorequireonlylow-levelthinking onBloom'staxonomy. Thereisnoprovisionfordeeperlearningin software| forexample,there isalakofonstrutedresponsequestioning. Towhat extentouldthisbeahieved?
Howould thepriniples ofstudent-entredlearningandstudentautonomybebetterenouraged in CAAsoftware? Weidentifyonepossibilityin thisreport, namelybyenhaningthevisibilityof thelearningproess,thoughitispossiblethatmanyothermeasureslikethisexist.
7 Conlusions
Developedassessmentmaterialsaremadelessreusableandlessdistributablebyagenerallakofprovision ofstandardisedorinterhangeableformats. Furthermore,standardsareslowto bedevelopedandoften lagbehindtherequirementsandapabilitiesof theproduts. Developedmaterialsarethereforeneither portable nor do they failitate sharing between leturers and produts (the question bank onept), reduingthemotivationfor,andvalueof,suhmaterials.
Seurity oftenisan issue,partiularly forthosewishing to undertakeassessmentin asummativeor diagnostiontext.
Forsuharuial partoftheeduation-by-assessmentproess,feedbakisnotexibleenoughinthe tools. Feedbaktostudentsis oftentooonentratedontheresponses themselvesratherthanat atest level.
Thereportingapabilitiesofmostofthetoolsislaking. Thegreatpotentialofusingtheomputeras aanalyseroflassstatistis,perhapsbyndingproblemstudents,areasofmisunderstanding,orsimply poorquestions,isgenerallywasted.
Themajorityof CAAtoolsannotbeapplied togeneralsituations. Theyaretypiallyintendedfor apartiularsettingin whihCAAommonlytakesplae. Thismightbeasimplequizinorporatedinto Web-deliveredmaterials,or amoreapablesummativeassessmentsystem,orapartofaVLE.Intheir limitedappliation areas, most of the tools are apable, and in many ases havebeen proven to some extentin eduationalinstitutions.
Every eorthas beenmade to ensurethe auray of the information supplied in this report and the assoiatedWebsite, and tothe best of ourknowledgeand belief the report andthe assoiatedWebsite fairlydesribesthetoolsand pakageswhih arereferredto. Anyerrororomissionanbereported to theauthorsand wewill orretthemassoonaspossible
Neither the University of Warwik nor the LTSN Subjet Centre for Information and Computer Sienes assumes any liability for errors or omissions, or for damages resulting from the use of the information ontained herein and on the assoiated Website, and are not responsible for the ontent of external internet sites or any other other soures. The information ontained herein and one the assoiatedWebsite isprovidedon thebasis that thereaderwill notrely onit asthesole basisfor any ationordeision. Readersareadvisedtoontatthestatedprimarysoureortheprodut'soriginating institutionbefore atingonanyoftheinformationprovided.
Thisreportinludes somewordsandotherdesignationswhihare,orareassertedtobe,proprietary names or trade marks. Where the authors are aware that a word or other designation is used as a proprietarynameortrademark,thisisindiatedinthelistbelow,buttheabseneofawordordesignation fromthislistdoesnotimplyanyjudgementof itslegalstatus:
Apahe, Authorware, Ceilidh, Coursemaster, DES, EQL, GRE, IMS, Interative Assessor, Java, JavaSript,Maromedia,Pereption, Perl,QuestionMark, TestPilot,Unix, Webssessor,Webassessor, WebCT,Windows,WebMCQ, XML.
9 Appendix: Summaries of the reviews
This appendix desribes the tools reviewed, and briey summarises the reviews, as presented on the Websitein January2002.
BOSS
BOSSis anopen-soure solutiondeveloped bythe Universityof Warwikto handle onlinesubmissions andoursemanagement. It allowsstudentsto submittheir work seurely,and forsta tomanage and markassignmens. Ithasbeendevelopedwithsound pedagogyin mind. Itusesarole-basedinterfaeto keeptrakof themarkingproessofmarking,moderationandpublishing. Thesoftwarestoresmuh of itsdata inanSQLdatabase,promotingsalabilityandintegrationwithothersystems.
Thesoftwareiseasytousedespiteitsomparativeomplexity,andhasgoodseurity. Alient/server approahistaken,anditslightweightlientswhihdownloadthelatestversionsofthestudentandsta lient ease large-sale deployment along with its platform independent Javaimplementation. Perhaps itsmainuseisforformativetesting;manyofitsfeatures aresuitedtofailitating regularprogramming assignments. It is also useful for diagnosti and even ontrolled summative testing, though does not lenditselfto ad-hoself-testingdueto themarkingproeduresitimplements. Automatitestingworks wellwithafewlimitations, thoughativedevelopmentwillmeanthis areawill beome moreexible as developmentprogresses. Doumentationisastrength.
CASTLE
TheCASTLE toolkitis aset of tools designed for tutors to reate multiple hoie basedonline tests. Emphasisis plaedonease ofuse throughout |no priorknowledgeof any languages(suh asHTML orJavasript)is required. CASTLEproduesHTML form-based quizzes apable of handlingmultiple hoie,multipleresponseandtext-mathing. Itisomparativelybasiinnature,itsmainstrengthbeing itseaseofuseanddeployment.
ClydeVU/Miranda
whihisreferredtoasMiranda.
Thoughinsomerespetsitisbeginningtoshowitsage,formostappliationssavesummativetesting inanexaminationenvironment,CVUisausefuland omparativelywell-thought-outtool.
Coursemaster
Coursemaster is alient-serverbased systemfor deliveringourses based aroundprogramming or dia-grammingexerises. It providesasolutionfor deliveringoursework speiations,allowingstudentsto developprogramsandsubmitthem,andtheautomatedmarkingandmonitoringofstudents'responses. Unlikemuhoftheothersoftwarereviewedonthissite,Coursemasteristailoredtotheneedsofteahing programming,andhasbeendevelopedandusedoverseveralyears.
Coursemaster is an updated version of the tried-and-tested (10 years) server/lient-based Ceilidh system. It hasa numberof uniqueapabilities, for examplethe assessmentof diagram-basedwork. It isrobustandeasy-to-use,partiularly forthestudent. Theleturerisgivenawidevarietyofstatistis, thoughthroughaslightlydatedinterfae. Markingisveryexible,thoughthebuiltintoolsdonotahieve anythingpartiularlyinnovative,but arestill useful. Installationmaybeaproblemarea,dependingon theplatforms and ongurationsused at your institution. All aspets of implementation seemto have beenthoughtthrough,andseurityisunusuallygood.
EQL Interative Assessor
EQLInterativeAssessorisanappliation-basedassessmentsystemforWindows. Itsemphasisisontests takenagainstthelokinthelassroomorathomewithautomatianalysisofresults. Theteaherusesa speialappliationfordesigningtestsandthestudentusesasimilaronefortakingit. Themanufaturers emphasiseseurity(separatingleturers'questionbanksandtheresultingtests)andexibility(freedom inquestiontypeandlayout).
EQL Interative Assessor is unusual in that its method of delivery is through a Windows-based appliation. However, the software is old and a newversionis due soon. The modular, exible tools makesthereationoftestseasy,thoughthedesignofquestionsthemselvesisalittlemoreinvolvedand requirealittle moreexperiene than with other tools. Ease of use from the student's pointof view is generallyverygood, withanumberof veryminordrawbaks. Thewhole suiteouldnotbetested due tothelimitationsonthedemonstrationsuite. Thesoftwareisperhapsbestappliedtoselfandformative testing,thoughunder therightonditionsouldalsobeapplied tosummativetesting.
Notethat EQLInterativeAssessor has sinebeenwithdrawn fromEQL'sprodutrange.
Netquest
TheNetquestprojetwassetuptoinvestigatehow`questionbanks'forCAAmightbesetuptofailitate testing. TheprojetisbasedaroundtheTML|thetutorialmodellinglanguage{asupersetofHTML intended to desribe question data while keeping formatting information separate from its semantis. SoftwarewasalsodevelopedtoproessTMLintoaformsuitablefordeliverytoaWebbrowser.
NetquestisademonstratorofTML,aproposedlanguageforassessment;itisnotintendedtotallyas aprodutfor deliveringtests. However,for this appliationit is ertainlyuseful, thoughit takesmore setting up than it possibly ould. Deliveryof tests is fast and lear, and basi analysis of individual students ispossible. Thoughthesoftware isnotimmediately appliabletosummativetesting, itworks wellasaregularformativeordiagnostitool.
Question Mark Pereption
fairlyomprehensiveCAA solution.
TACO
TACO is a Web-based appliation whih allows leturers to reate `ourseworks'for students. TACO allowstheleturerstoseletquestionsets,madeupofavarietyofquestiontypes,andTACOmarksand givesfeedbak.
TACOisawell-struturedHTML assessmentenginewhose developmentwasmotivatedbyastrong requirements phase. It has asimple user interfae and a good degree of exibility in the the form of assessments,though its analytial apabilities are laking. It is perhapsmost suited to formative and perhapsdiagnostiassessment,thelakoftightseuritybeingthemainissueforsummativeassessment.
Test Pilot
Test PilotisamoderatelyapableWeb-basedassessmentengine. Leturers authorquestions and build assessmentsfromthem,whihanbeautomatiallygraded,withgoodfeedbakandanalysispossibilities. Itsmain strengths liein its relativelyintuitiveinterfae,whih oerslots of help andis relativelyeasy to work with. Equal attention hasbeen paid to all stages of the assessmentproess, from reation to assessmenttostatistial analysis. Beingaommerial produt,it isrelativelyexpensive,espeiallythe supportfees. TestPilotwouldbesuitableforaformativeordiagnostilevelassessmentappliation.
TRIADS
ThenameoftheTRIADSprojetstandsfor`TripartiteAssessmentDeliverySystem',andisa ollabora-tionbetweentheUniversityofLiverpool,theUniversityofDerbyandtheOpenUniversity. Theprojet's fousisonlearningoutomesandtheirinueneonurriulumdesign. Byunderstandingthisitaimsto produeaCAAsystemwhih allowsatheoretiallyjustiedapproahtoformativeandsummative test-ingofknowledge,understandingandskills. TheTRIADSsystemisaseriesofprogramswhihrununder Maromedia'sAuthorware. TheAuthorwareeditoristhereforerequiredto makeuseofthesoftware.
The software presents an unrivalled number of question types and exibility in question-setting, thoughtheauthoringforquestionsrequiresprogrammingskills. DespitetheTRIADSprojet'snotiable eorts, the Authorware environment is omplex to use and introdues a steep learning urve. The authors'plans for abetter stauser interfaewill go alongwayto xing this problem. Taking atest ifstraightforwardandgoodfeedbakanbespeied,thoughtherearesomeuserinterfaeissueswhih need to be onsidered hereas well. Overall, apromising and exible aademi produt in need of an easieruserinterfae.
Webassessor
Webassessor(alsowrittenWebssessor)isanassessment,authoringanddeliverysystemaimedprimarily at industry, but whih also laims to be as relevant for use in aademia. Emphasis is plaed on ease of use both in assessment reation and administration, while still allowing exibility and `multimedia' apabilities. Themain appliation and area ofollaborationwith industryis seenas distanelearning ratherthaninorporationin mainstreameduation.
Webassessorisaapablesolutionwhih atersformostappliations. Its reportingapabilitiesmean thatithasfeaturessomewhatmoresuitablefordiagnostitesting. Ithasareasonablygooduserinterfae andiseasytouse,withgoodfeedbakapabilities. Thebasiquestiontypesareprovided. Seurityhas beenonsidered andithasbeenprovenin somelimitedommerialontexts. However,boththeserver softwarelieneandthehostingservierepresentsasigniantonsiderablenanialoutlay.
Webassign
reognisedtextbooks.
WebAssign is signiantly more apable in many respets than its ompetitors. It is partiularly strong on the question-bank philosophy, feedbak, randomisation, reporting and ommuniation with students. Attentionhasbeenpaidtoallowingaexible,modularapproahtoassignmentbuilding. After someexperiene with the mostlyintuitiveinterfae, thesoftware anbeused to reate agood variety ofassignmentsin astruturedmanner, though thelimited rangeof questiontypes maybea problem. Generally,aexibleandapablepakage,well-suitedtotherequirementsofauniversityleturerneeding tobuildassignments.
WebCT
WebCTissoldasaoursemanagementsystem,laimedbytheWebsitetobethemostpopular. Emphasis is oninreasing ommuniationbetween leturers and students, thoughonline assessment is oneof its apabilities. Another emphasis is the deliveryof online ontent, and the easy aquisition of delivered materialfrom publishingompaniesratherthandevelopingit`in-house'bytheleturer.
Theassessment omponent isonentrated onin thereview, though broaderonsiderationstakein themoregeneralappliation of thesoftware. The assessmentapabilities arelimited tothe pointthat WebCTmaynotbeappropriateasanassessmenttoolonly. Furthermore,WebCTisaimingtoreplaeits assessmentomponentwithQuestionMarkinthefuture,usinginter-programommuniationtoahieve theintegration.
WebCTisaomprehensiveoursemanagementsystem,withrelativelygoodCAAapabilities. Simply asaCAA tool WebCTwould be expensive, umbersome to manage, and overomplex forassessment reation,reetingitsaimtofullamuh widerrole.
WebMCQ
WebMCQ provides a Web based servie allowing quizzes foused on multiple hoie questions to be reated,editedanddeliveredthoughaWebbrowser. All administrationanddeliveryofquizzes anbe arriedoutfromtheWebMCQsiteitself. Emphasisisontheprovisionofservie,asopposedtosoftware, andoneaseofuse.
WebMCQprovidesarelativelyeasytousesystem,with strongseurityandagood supportservie. Itisslightlyletdownwithsomeonfusinginterfaefeatures andarestritiverangeofquestiontypes.
Referenes
[1℄ ApaheSoftwareFoundation. Apahewebsite. http://www.apahe.org/.
[2℄ Abhir Bhalerao and Ashley Ward. Towardseletronially assisted peer assessment: aase study. Alt-J| Assoiation for LearningTehnology Journal,9(1):26{37,2001.
[3℄ Benjamin S.BloomandDavidR.Krathwohl. TaxonomyofEduational Objetives: The Classia-tionofEduationalGoals,byaommitteeofollegeanduniversityexaminers.HandbookI:Cognitive Domain. Longman,1956.
[4℄ DavidBoud. Assessment andLearning: ContraditoryorComplimentary?, pages35{48. Sta and Eduational DevelopmentSeries.KoganPage,1995.
[5℄ Don Charman. Issues and Impats of using Computer-based Assessments (CAAs) for Formative Assessment,pages85{94. StaandEduational DevelopmentSeries.KoganPage,1999.
[6℄ ClydeVirtualUniversity. AboutCVU. http://vu.strath.a.uk/admin/vudos/.
[7℄ James R. Dalzieland Sott Gazzard. Assisting student learningusing web-basedassessment: An overviewofthewebmqsystem. http://www.webmq.om/publi/pdfs/ovrview.pdf.
[10℄ Jen Harveyand NoraMogey. Pragmati Issues When Integrating Tehnology intothe Assessment of Students,pages7{20. StaandEduationalDevelopmentSeries.KoganPage,1999.
[11℄ IMSGlobalLearningConsortium.IMSQuestion&TestInteroperability: ASISeletionandOrdering Speiation,Otober2001.
[12℄ IMSGlobalLearningConsortium.IMSQuestion&TestInteroperabilitySpeiation,Otober2001.
[13℄ Institute for Learning and Researh Tehnology. Netquest. http://www.ilrt.bris.a.uk/ netquest/.
[14℄ M.S.JoyandM.Luk. TheBOSSsystemforon-linesubmissionandassessment.Monitor: Journal of the CTICentrefor Computing,pages27{29,1998.
[15℄ MikeJoy, SimonRawles,andMihaelEvans. Computerassisted assessmenttoolsreviewexerise. http://www.ds.warwik.a.uk/ltsn-is/resoures/aa/reviews/,2001.
[16℄ Learning Tehnology and Researh Group. Coursemaster. http://www.s.nott.a.uk/ CourseMaster/,2001.
[17℄ EQLInternationalLtd. EQL. http://www.eql.o.uk/.
[18℄ LTSN-ICS. LTSN-ICSWebsite. http://www.is.ltsn.a.uk/.
[19℄ Don Makenzie. Triads (tripartite interative assessment deliverysystem) manual. http://www. derby.a.uk/assess/manual/tmanual.html.
[20℄ Maromedia. Authorware. http://www.maromedia.om/software/authorware/,2001.
[21℄ Mis. Perlwebsite. http://www.perl.om/,2002.
[22℄ Malolm Perkin. ValidatingFormative andSummativeAssessment,pages55{62. Sta and Edua-tional DevelopmentSeries.KoganPage,1999.
[23℄ QuestionMark. QuestionMark. http://www.questionmark.om/.
[24℄ KaySambell,AlistairSambell,andGrahamSexton.StudentPereptionsoftheLearningBenetsof Computer-assisted Assessment: A Case Study in Eletroni Engineering, pages179{192. Sta and Eduational DevelopmentSeries.KoganPage,1999.
[25℄ M.A.Sasse,C.Harris,I.Ismail,andP.Monthienvihienhai. Supportfor AuthoringandManaging Web-basedCoursework: The TACOProjet. SpringerVerlag,1998.
[26℄ Leith Sly andLeonieJ. Rennie. Computer Managed Learning asan Aid toFormative Assessment in HigherEduation,pages113{120. StaandEduationalDevelopmentSeries.KoganPage,1999.
[27℄ SunMirosystems. Javawebsite. http://java.sun.om/.
[28℄ Christian Su and BurkhardFreitag. Learningmaterial markup language: LMML. http://www. ifis.uni-passau.de/publiations/reports/ifis200103.pdf.
[29℄ AthanasiosTsintsifas. Diadalos. http://www.s.nott.a.uk/~azt/daidalos/.
