Pilot study

Prior to the commencement of the testing procedure, two pilot participants completed all four tasks, on four separate days, over the course of a week as part of a pilot study. Both pilot participants were L1 English speakers. There were unfortunately not enough Grade 9 L1 isiXhosa learners whose Mandarin marks were above 50% to “sacrifice” a potential test participant for participation in the pilot study. The pilot participants were randomly selected (ensuring only that they had above 50% for

53  _{The  ECP  requires  that  a  trace  be  properly  governed  by  its  antecedent  (Crystal,  2013:  168).}  

87   Mandarin at the end of 2015). The purpose of piloting the study was to assess whether the tasks were of an adequate level for the participants, i.e. not too difficult or time consuming. In terms of task “difficulty”, the ease or difficulty of the tasks was not determined by the number of correct/incorrect wh-word placements/judgements, but rather on grounds of whether or not the pilot participants understood the vocabulary presented to them and could complete the tasks (regardless of whether their answers were grammatical or ungrammatical54_{) with relative ease. Each pilot participant was}

allocated a participant code, “P01” and “P02” respectively. Table 4.3 below illustrates the percentage of accurate word order placements/judgements achieved by each participant on each task, with a distinction between the three wh-words tested. The average total scores for the test as a whole are also provided. Table 4.4 presents the average total scores for the three respective words when the results of the four tasks are totaled.

Task Participant P01 Participant P02

Individual words Task as a whole Individual words Task as a whole SF shenme (“what”) 0% 20% 80% 60% nar (“where”) 0% 60% na (“which”) 60% 40% OP shenme 0% 0% 67% 55.6% nar 0% 100% na 0% 0% GJ shenme 60% 56.6% 90% 83.3% nar 40% 90% na 70% 70% ST shenme 0% 0% 100% 55.6% nar 0% 67% na 0% 0%

Total test score 19% 64%

Table 4.3: Pilot participants’ overall results on the four tests, presented as average percentage scores

54 _{As the accuracy of the participants’ word order judgements is the issue at hand, incorrect word order} judgements do not indicate that the test is too difficult, but rather simply that the participant is most likely transferring the incorrect word order for Mandarin from their L1 (English).

% correct: shenme (“what”) % correct: nar (“where”) % correct: na (“which”) 49.6% 44.6% 30%

Table 4.4: Pilot participants’ average scores for the three wh-words on the test as a whole

The pilot participants’ achievement of an average of 42% for the test as a whole indicate that there were no floor effects when the two sets of marks were considered together. Participant P01’s performance on the OP and ST tasks did, however, suffer from floor effects. Given P02’s relatively good performance on the these two tasks, the tasks were not deemed problematic but rather an ideal testing ground for possible differences between the performance of participants who do and those who do not have knowledge of a wh-in-situ language. The reason for this is that, by chance, pilot participant “P02” had knowledge of Swahili (a wh-in-situ Bantu language). This “in- situ” knowledge appeared to have a positive effect upon the pilot participant’s performance in comparison to the performance of the pilot participant who had no knowledge of a language that allows the wh-word to remain in-situ. Given that Swahili and isiXhosa share the wh-in-situ rule, participant P02’s performance was deemed a possible indication of what to expect in the L1 isiXhosa participants’ performance.

The formation of questions/sentences in the SF, OP and ST tasks indicated that the pilot participants generally formed constructions based either on the typical English or Mandarin word order for such questions/sentences and not random constructions with little resemblance to the word orders of these two languages. This suggests that the pilot participants clearly understood what was expected of them in order to complete the given tasks and were familiar with and understood the vocabulary employed. From the results above it was also predicted that the OP and ST tasks would be the most difficult for the participants and that the placement of the wh-word na would be acquired with the most difficulty.

Note that both the pilot participants and the 20 participants who volunteered to partake in the study were issued with a Mandarin vocabulary list with corresponding

89   English meanings (cf. Appendix 7). This was for revision purposes and as the present study is concerned purely with syntax, it was important to ensure that participants did not score badly on the tasks due to a lack of vocabulary. All participants kept the vocabulary list with them during testing and were encouraged to refer to the list when necessary. No Mandarin-isiXhosa vocabulary list was provided because, as mentioned above, 40% of the L1 isiXhosa-speaking participants professed to very low isiXhosa literacy levels in the language background questionnaire. Thus, a Mandarin-isiXhosa vocabulary list would not have been of much help and, additionally, posed the possible risk of triggering their L1 (discussed below).

Including the time taken to introduce and explain each task to the participants, the SF task was recorded as having an average completion time of 45 minutes; the OP task took an average of 15.5 minutes; the GJ task was completed by both pilot participants in just under 30 minutes; and the ST task was completed in approximately 35 minutes by both participants. According to a guideline set out by Sievertsen, Gino and Piovesan (2016), no task should take any participant longer than one hour to complete if fatigue effects are to be avoided. This requirement was met with in the pilot study. The pilot participants’ performance, in conjunction with the time it took them to complete the given tasks, was to the researcher’s satisfaction. Consequently, no changes were made to any of the four tasks and testing could commence.

It should be noted, once again, that the three wh-words under investigation are shenme (“what”), na (“which”) and nar (“where”). The participants, as “beginner-level” Mandarin students, have not yet learnt the wh-question expression shenme shihou (“when”) and wh-question words pertaining to more complex interrogative questions, i.e. weishenme (“why”) and zenme (“how”). These words were consequently not included in the testing.