Factor analysis

An exploratory factor analysis was conducted separately for the two conditions (single play and double play) to group questionnaire responses in order to validate the questionnaire categories and to simplify subsequent tests for statistical differences. It was decided to join the data for the individual tasks in each condition in order to achieve a larger sample size and higher common factor variance without cross-loadings, following recommendations by Osborne and Costello (2005). The datasets for the two separate conditions were therefore responses based on both MC and NF tasks, across all participants (N=304, with 2 missing responses). For this reason, one item had to be dropped prior to the analysis as this item was only included for the MC tasks but not the NF tasks (Item 2 in the questionnaire for MC tasks). The remaining items were the same for the two tasks.

Principal axis factoring with Varimax rotation was chosen as extraction method. De Winter and Dodou suggest using principal axis factoring for data with “a relatively simple factor pattern” (2012, p. 708), which was the case as it was hypothesised that the factors would cluster according to the four sections of the questionnaire (test-taking strategies, listening strategies, test-taking anxiety, and listening anxiety,). The analyses were run with both Varimax and Direct Oblimin rotation, which yielded essentially the

110 same results. Only the results based on Varimax rotation are presented here, as findings obtained by Varimax rotation are easier to interpret than findings based on Direct Oblimin rotation (Osborne & Costello, 2005, p. 3).

As suggested by Osborne and Costello (2005, p. 3), the factor analysis was run several times for both conditions to 1) identify the number of factors to be included in the final analysis by inspecting the scree plot each time and 2) detect outlier items which cross-loaded onto separate factors. Kaiser-Meyer-Olkin (KMO) test for sampling adequacy and Bartlett’s test for sphericity were performed for each separate analysis and were found to be adequate in each case (KMO ranged between 0.83 and 0.87 and Bartlett’s test was significant at the 0.00 level each time). For both conditions, the same three main factors were detected after inspection of the scree plots and five items were identified and removed for the final analysis, as these items each cross-loaded onto different factors:

Item 1: I read the questions/answer options before listening. (test-taking strategies)

Item 6: I filled in the answer sheet anyway, though I wasn’t sure. (test-taking strategies)

Item 14: I concentrated hard on what the speaker was saying. (listening strategies)

Item 18: I translated what I heard into my mother tongue. (listening strategies) Item 24: I enjoyed listening to the tasks. (listening anxiety)

The final analysis for both datasets was run with a fixed number of three factors. KMO was 0.87 for the single play condition and 0.86 for the double play condition and Bartlett’s test was significant at the 0.00 level for both conditions (Table 22). The total variance explained was 48.64 percent for the single play and 45.06 percent for the double play condition. For both conditions, the same items loaded mainly onto the same factors (Table 23), except for item 3 and item 21, which loaded mainly onto a different factor in the double play condition. However, it was decided to keep these items in the analysis as in the single play condition they loaded mainly onto the same factor. As hypothesised, the identified factors correspond to the pre-specified categories of the questionnaire. One factor relates to test-taking strategies (items 3 to 5), one to listening strategies (items 15 to 19), and one to anxiety in general (items 7 to 13 and 20 to 25);

111 items related to test-taking anxiety loaded onto the same factor as items related to listening anxiety.

Following this, as suggested by Dörnyei and Taguchi (2009, pp. 93–95), Cronbach’s Alpha was calculated based on the complete dataset but separately for the three identified factors to inspect reliability. For test-taking strategies Cronbach’s Alpha was .70, for listening strategies it was .75, and for anxiety it was .93, indicating that the individual factors were reliably measuring their respective constructs (see also Vogt, 2007). In addition, none of the items contributed negatively to overall reliability, that is no item would have increased Cronbach’s Alpha if deleted.

Table 22: Study 1: KMO measure of sampling adequacy and Bartlett’s test of sphericity for the factor analysis of the responses to Questionnaire 1

Single play Double play

KMO measure of sampling adequacy 0.87 0.86

Bartlett's test of sphericity approx. Chi-Square 1255.00 1062.38

df 171 171

112

Table 23: Study 1: rotated factor matrix for the responses to Questionnaire 1

Single play Double play Factor 1 2 3 1 2 3

3. I made a guess based on vocabulary used in the questions (and options).

-0.08 0.22 0.32 -0.15 -0.03 0.18 te s t- ta ki n g st ra te g ie s

4. I listened for the words that appeared in the questions (and options).

-0.10 -0.02 0.63 -0.21 -0.59 0.12

5. I only listened for relevant information to answer the questions.

-0.12 -0.08 0.56 -0.15 -0.57 0.03

7. Before taking the test, I felt confident and relaxed. 0.63 0.03 -0.14 0.64 0.15 0.13

a

n

xi

e

ty

8. During the test, I found myself thinking of the consequences of failing.

0.62 -0.01 -0.03 0.53 0.21 -0.14

9. During the test, I got so nervous that I forgot facts I really know.

0.70 -0.06 -0.14 0.68 0.09 -0.04

10. After taking the test, I felt I could have done better than I actually did.

0.51 -0.12 -0.20 0.35 0.26 -0.07

11. When I first got my copy of the test, it took me a while to calm down to the point where I could begin to think straight.

0.61 -0.15 -0.20 0.63 0.06 -0.03

12. While I took the test, my nervousness caused me to make careless errors.

0.71 -0.11 -0.20 0.68 0.08 -0.09

13. While taking the test, I found myself wondering whether the other students were doing better than I was.

0.50 0.07 0.13 0.46 0.27 0.03

15. I guessed the meaning of unknown words, using tone of voice as a clue. -0.10 0.30 0.05 -0.06 -0.05 0.40 list e n in g s tr a te g ie s

16. While listening, I made up a story line, or adopted a clever perspective.

0.01 0.52 -0.20 0.10 0.04 0.41

17. I made a mental or written summary of language and information presented in the listening tasks.

0.12 0.74 0.04 0.14 0.03 0.61

19. While listening, I monitored my understanding of the listening passage discourse structure.

0.01 0.40 0.04 -0.03 -0.05 0.48

20. I got upset when I was not sure whether I understood what I was hearing in English.

0.72 -0.10 -0.01 0.63 0.24 -0.02 a n xi e ty

21. I often understood the words but still couldn’t quite understand what the speaker was saying.

0.57 0.20 -0.17 0.40 0.43 0.11

22. I got so confused I couldn’t remember what I’d heard. 0.67 0.03 -0.17 0.60 0.25 0.19

23. I felt intimidated while listening to the tasks. 0.76 -0.08 -0.04 0.67 0.20 0.03

25. I felt confident while listening to the tasks. 0.70 0.16 -0.05 0.54 0.33 0.23

Extraction method: principal axis factoring

Rotation method: Varimax with Kaiser normalization

Rotation converged in four iterations for the single play and five iterations for the double play condition