Coding

Once all transcripts were checked by me and segmented into different data files as outlined in Table 9, the coding scheme was finalised. The scheme was based on the theoretical framework described in Section 2.4 and included the four main response processes of interest: cognitive processes, listening strategies, test-taking strategies, and anxiety. The main aim of the research in this thesis was to identify how these dimensions are influenced by double play as compared to single play. In the following, a summary of the four response processes and their various sub-dimensions will be presented, based on the discussion in Section 2.4. This summary served as the first version of the coding scheme.

Cognitive processes

Descriptions are based on Field (2013), Vandergrift and Goh (2012), and Rost (2011)  Input decoding (acoustic-phonetic processing): Recognising incoming sounds as

speech. Informed by phonological knowledge.

 Lexical search: Recognizing individual words. Informed by lexical knowledge.  Parsing: Putting individual words into a syntactic pattern to form the bare meaning

of an utterance at clause or sentence level. Informed by syntactic knowledge.  Meaning construction (micro-level conceptualization): Relating the literal meaning

of utterances to the context in which they occurred to construct higher-level meaning. Informed by pragmatic knowledge and external knowledge about the world, the speaker, and the topic.

81  Discourse construction (macro-level conceptualization): Relating the meaning of

the message to the discourse as a whole. Informed by external knowledge about the text type, the world, and the speaker.

Listening strategies

Descriptions are taken verbatim from Vandergrift and Goh (2012, pp. 277–284)

 Planning: Developing awareness of what needs to be done to accomplish a listening task, developing an appropriate action plan and/or appropriate contingency plans to overcome difficulties that may interfere with successful completion of a task.  Focusing attention: Avoiding distractions and heeding the auditory input in different

ways, or keeping to a plan for listening development.

 Monitoring: Checking, verifying, or correcting one’s comprehension or performance in the course of a task.

 Evaluation: Checking the outcomes of listening comprehension or a listening plan against an internal or an external measure of completeness, reasonableness, and accuracy.

 Inferencing: Using information within the text or conversational context to guess the meanings of unfamiliar language items associated with a listening task, to predict content and outcomes, or to fill in missing information.

 Elaboration: Using prior knowledge from outside the text or conversational context and relating it to knowledge gained from the text or conversation in order to embellish one’s interpretation of the text.

 Prediction: Anticipating the contents and the message of what one is going to hear.  Contextualization: Placing what is heard in a specific context in order to prepare for

listening or assist comprehension.

 Reorganizing: Transferring what one has processed into forms that help understanding, storage, and retrieval.

 Translation: Relying on one’s knowledge of the first language or additional languages to make sense of what is heard.

 Managing emotions: Keeping track of one’s feelings and not allowing negative ones to influence attitudes and behaviors.

82 Test-taking strategies

Descriptions are based on A. D. Cohen (2011)

 Test-management strategies: Controlled and goal-directed mental actions to find an answer to a question. Informed by both the test paper (the questions, answer options etc.) and the listening text. E.g. choosing an answer option out of four based on the meaning of the relevant passage.

 Test-wiseness strategies: Controlled and goal-directed mental actions to find an answer to a question, informed solely by the test paper (the questions, answer options etc.) or construct-irrelevant external knowledge. Not informed by the listening text itself. E.g. choosing an answer based only on its position among the other answers (a, b, c or d); guessing.

Anxiety

Descriptions are based on Cassady and Johnson (2002) and Horwitz (2010)

 Listening anxiety: Worries, stress, or concerns related to listening in a foreign language.

 Test-taking anxiety: Worries, stress, or concerns related to evaluative situations. I coded all documents using the qualitative data analysis software Atlas.ti (version 8.4 for Mac). Initially, I imported all data files into Atlas.ti and programmed the codes according to the coding scheme. The data files were then split into segments (referred to as quotations in Atlas.ti) corresponding to the codes in the coding scheme. For example, the following excerpt from Participant 1 was separated into two quotations:

[Quotation 1] Yes, because then I heard “this is the newest release“ [Quotation 2] and then I chose this answer.

Quotation 1 indicates that the participant put individual words into a syntactic pattern to form the bare meaning of the utterance at clause level, so this quotation was coded as parsing. Quotation 2 shows that the participant chose an answer to a question based on their understanding of the recording, thus, this quotation was coded as test- management. Another example of segmenting is the following:

83 [Quotation 1] Here I thought, ok now she will probably soon mention what Lego means for him. [Quotation 2] And then I already focused on that.

Quotation 1 shows that the participant tried to predict what the speaker was going to say next. This segment was therefore coded as prediction. Quotation 2 indicates that the participant focused on the listening text in a particular way, so this quotation was coded as focusing attention.

Whenever I could not unambiguously assign a single code to a quotation, two codes were applied to the same quotation. This was mostly the case for quotations relating to test-management, as test-management was sometimes only evident in combination with either a cognitive process or a listening strategy. The following excerpt is a typical example:

Then I read through everything again, checked everything, and looked whether I hear it again/whether I would choose the same answer again.

In this quotation the participant checked the outcomes of their listening comprehension against the answers they chose, so this was coded as evaluation. However, because the participant referred to the test paper (the answers they chose), the quotation was also coded as test-management. The quotation could not have been segmented further to differentiate between the two codes, as both codes applied to the quotation as a whole. During the process of coding three additional codes emerged based on recurring meta-commentary which appeared to be relevant for answering the research questions. Two of these codes were assigned to quotations which were specifically related to the single play/double play condition:

 Different behaviour: Reporting different listening behaviour between the first play and second play or between single play and double play. Example:

So most times during the second play/during the first play one hears the individual things more/so the things one is waiting for the whole time and during the second play one understands more about the context and more about the other details […].

 Prefer double play: Indicating a preference for double play compared to single play. Example:

84 It is nice when I can hear it twice, because then I can explicitly listen to that again, because I was relatively sure about the other questions, I don’t have to pay attention to that any more. But then I can see clearly which ones I haven’t answered so that I can answer these. That is nicer in double play.

One of the additional codes was assigned to comments in relation to the research methodology, to discern whether the research methodology was inhibiting natural processing during task completion:

 Reactivity: Indicating that the research method is distracting or is inhibiting natural processing. Example:

What irritated me a little bit was the thing with the head (referring to the fact that they had to keep their head still for accurate eye-tracking readings), it was not very bad, but I wasn’t used to it, only moving my eyes.

No evidence was found in the data for two of the original codes: the cognitive process input decoding and the listening strategy reorganizing. For input decoding, other studies have also found it difficult to isolate evidence from verbal report data (see, for example, Brunfaut & McCray, 2015; or Holzknecht et al., 2017) or had to rely on evidence from written notes to draw conclusions about mishearings (Rukthong, 2015). However, input decoding underlies all other cognitive processes (Field, 2013) and it is therefore reasonable to assume that test takers who engaged in cognitive processing would have relied on input decoding at a fundamental level, but it was so prevalent and automatised so as not to be retrievable by participants in their reporting behaviour. The strategy of reorganising, on the other hand, is operationalised by processes such as writing a summary, repeating words or phrases out loud, grouping information, or taking notes while listening (Vandergrift & Goh, 2012, p. 282). It is likely that reorganising was not observed because participants did not have the opportunity nor the time to engage in these processes while completing the tasks.

Finally, the two separate codes on anxiety (test-taking anxiety and listening anxiety) were merged after the coding. This was done as participants only indicated general levels of anxiety and did not specify whether their anxiety was related to the test-taking process or to listening in a foreign language.

85 In summary, the final coding scheme consisted of 20 individual codes grouped into five main categories: cognitive processes, listening strategies, test-taking strategies, anxiety, and meta-commentary. The codes are summarised in Figure 9. For ease of reading, further definitions of the coding categories and illustrative excerpts from the data will be provided in Chapter 5.

Figure 9: Study 2: final coding categories