Profiling learner proficiency on the basis of memorization 1 Thedataset

Thedatasetistheproductofanin-depthstudyofsixintermediate/advanced

learnersofEnglishstudyingintheUK,engagingincyclesofmemorizationfor

thepurposesofrealconversations.Inanygivencycle,thesubjectworkedone-to-onewithanativespeakerofEnglishtoidentifyafutureconversation,andto

predictutterancesthatwouldbeusefulandappropriatetothatconversation.The

nativespeakerprovidedanativelikeformulationofeachutterance(the‘model’,

M)andrecordeditontoCD.Thesubjectthenmemorizedthemodelsathome,

returningafterafewdaystorehearsethemwiththenativespeakerina‘practice

performance’ (PP). Finally, the subject attempted to use the memorized utter-ancesintherealsituationthathadbeenanticipatedatthestartofthecycle.This

constituted the ‘real performance’ (RP). Some utterance sets were additionally

subjecttoanunpreparedrecalltestthreeorfourweekslater,the‘delayedper-formance’(DP).Allstagesoftheprocessweredigitallyrecorded.Eachcyclewas

aself-containedpackagethattookplaceover7to10days.Subjectstookpartin

betweentwoandfivesuchpackageseach.Allsubjectswerefemale,aged22to

35(meanage28.8yrs).Three(Ch,HiandSa)wereL1speakersofJapanese,and

three(Jo,LcandLo)ofChinese.FurtherdetailsofthestudyarereportedinFitz-patrick&Wray(2006).

Theaimoftheprojectwastoobservetheconsequencesforthesubjectsof

memorizingtheirpreparedutterances:theeffectontheirgeneralaccuracyand

fluency,theimpactontheirself-perception,theirprojectionofcompetenceand

confidencetowardsnativespeakerinterlocutors,andtheresponsesthattheirut- terancespromptedfromthoseinterlocutors.Wealsolookedforcorrelationsbe-tweentheuseandaccuracyofmemorizedmaterialinuseandindividualprofiles

ofproficiency,aptitude,learningbackgroundandmotivation.

Thestudywasdesignedtoavoidcertainconfoundingvariablesthatnormally

arisewhenlookingattheefficacyofpreparedconversation:

130 AlisonWrayandTessFitzpatrick

a. Material targeted for memorization is often deliberately generic (e.g. Gat-bonton&Segalowitz1988),anddoesnotfitthelearner’sprecisesituation

particularlyclosely.Tocounterthis,theutterancesweredesignedtomeetthe

subject’sownimmediateneedsinaspecificsituationidentifiedbyher.

b. Learnersoftenlackconfidenceaboutwhetherornottheyaremakingnative-likechoices.Tocounterthis,theutteranceswereknownbythelearnertobe

nativelike,havingbeengeneratedbyanativespeakerinherpresence.

c. Apoorperformanceinrealconversationcanreflectthelearner’slackofexpe-rienceinproducingthedesiredstringfluentlyandaccurately.Tocounterthis,

therewasampleopportunitytolearntheutterancesandpractisethemwith

thenativespeaker.

d. Learnersareoftencompromisedbyalackofunderstandingaboutthemate-rialtheyareusing.Tocounterthis,weusedsubjectswithagoodindependent

knowledgeofEnglish,sotheywerenotmemorizinganythingtheycouldnot

understand.

Twenty-oneconversationcyclesformedthebasisofouranalysis,consistingof

227modelutterances,oranaverageof10.8modelutterancesperconversation.

Themeannumberofwordspermodelutterancewas10.05.Thedataweretran-scribedfromdigitalrecordingsofthepractice(PP)andrealperformance(RP)

sessionsand,asapplicable,fromthedelayedperformance(DP).Sincewearenot

concernedherewiththetrajectoryofretentionovertime(seeWray2004fora

studythatdidexaminethisfactor),thedatafromdifferentstageshavebeenamal-gamatedforthepresentpurposes.

4.2 Dataanalysis

Atotalof2416memorizedwordscontributedtotheanalysis:theseconstitutethe

wordsinthetargetsthatwereattempted.Targetsthatwereneverattemptedwere

excluded,sinceitwasunclearwhethertheyhadbeenmemorizedatall,and,if

theyhad,whethertheyhadeverbeendeemedrelevantforuse.Table1showsthe

distributionoftargetmaterialacrossthesixsubjects.

Table 1. Profileofdataset

Ch Hi Jo Lc Lo Sa

Totalwordsin

attemptedmodels

158 731 360 360 151 656

Thesubjects’outputswereanalyzedandcategorizedaccordingtodeviations

fromthemodels.Adeviationisnotnecessarilyanerror,otherthaninthevery

specificsensethat it renders the output non-identicalto the model.There are

myriadreasonswhydeviationsfromamodelmightbelegitimateanddesirable.

Theyincludetheneedtoembedthewordstringappropriatelyinthediscourse

(forinstance,omittinganinitialadjunctsuchasthenorsobecauseitisnotneces-sary),andtheneedtoalterthefactualcontent(forinstance,thetimeofameeting,

ifthememorizedtimewasnotconvenient).Thecapacitytoeffectsuchdeviations

isinitselfamarkerofproficiency:abeginnermightbeabletomemorizeaphrase

successfullybutnottailoritinordertoextenditsuse(compareMylesetal.1999,

whofoundthislackofextensioninlearnersofFrenchasaforeignlanguage).

Typicallyinourdataset,amodelutterancewasrecalledwithtwoormore

deviationsoverthevariousattempts,andeachdeviationwasseparatelyclassi-fied.Forinstance,Figure3representsthematerialassociatedwithattemptsat

themodellabelled‘Jo2:3’,thatis,thethirdmodelutteranceinJo’ssecondcycle,

whichwasforaconversationinashopaboutoptionsforprintingphotographs.

Themodel(M)isfairlyaccuratelyreproducedinthepracticeperformance(PP),

withonedeviation,asubstitutionofcanforcould.Thisdeviationwascategorized

asanativelikemorphologicalvariant,thatis,achangethatanativespeakermight

make.Incontrast,intherealperformance(RP)thesubstitutionofif I want to I can XforI wonder if I could X wasjudgedtobenon-nativelike.Sotoowasthe

entireattemptinthedelayedperformance(DP).

Inadditiontobeingclassifiedasnativelikeornon-nativelike,deviationswere

classified as occurring at one of three levels: morphological, lexical or phrasal

(multiword).Thus,intheRPattemptinFigure3,if I want to I canwastreatedasa

phrasalsubstitution,andhenceasingledeviation,ratherthanasaseriesofsingle

wordreplacements,wordorderchangesandsoon.Althoughsuchjudgements

arenotanexactscience,decisionswereappliedconsistentlyacrossthedatabase,

andcheckedattheendoftheprocess,toensurethatsimilarlinguisticbehaviour

hadalwaysbeencategorizedinthesameway.

Theanalysisidentifiedatotalof922deviations.However,twominortypes

ofdeviationweresetasideinthesubsequentanalysis.Onewastheexpansionof

two-wordcontractionssuchasI’dandwouldn’t.Theseconstitutedatotalof36

M OhIwonderifIcouldaskyousomething PP OhIwonderifIcanaskyousomething RP OhifIwanttoIcanaskyousomething DP CanIhavesomequestion

Figure 3. ModelutteranceandattemptsforJo2:3

132 AlisonWrayandTessFitzpatrick

deviations,or4%ofthetotal.Althoughtheyoccurredinallsixsubjects’output,

itwasconsideredinappropriatetoincludetheminthefurtheranalysissincethey

didnotconstituteaninstanceofrecallfailureperseand,infact,wereindicativeof

knowingwhattheunderlyingformofthecontractionwas,ratherthannotknow-ingthecontraction.Therewerenoinstancesofincorrectexpansions–thatis,I’d wasnevermistakenlyexpandedtoI hadinsteadofI would,etc.–norwerethere

anyintroducedcontractionsofuncontractedtargetwordpairs.

Thesecondcategoryexcludedwasslipsofthetongue.Therewere,infact,

onlytwoofthese,bothfromthesamesubjectandbothentailinglexicalreplace- mentwithaphonologicallysimilaritem:problemsforprojects(Sa2:8)andlectur-ersforlessons(Sa3:11),thelatterpossiblyasemanticchange(tolectures)followed

byaphonologicalslip.Theseslipswereomittedbecausetheystoodapartfromall

otherdeviations,astheonlyonesthatmightalsobemadebyanativespeakeryet

resultinanon-nativelikeoutcome.Thedistributionoftheremaining884errorsis

shownintheAppendix.Deviationswerenativelikein57.7%ofcases.

5. Deviation profiling