Verbal protocols

To investigate the cognitive processes and strategies the participants used to complete the tasks (RQs 1 and 1a), this study employed verbal protocols, i.e. stimulated recalls. Verbal protocols are “oral records of thoughts, provided by subjects when thinking aloud during or immediately after completing a task” (Kasper, 1998, p. 358).

In information-processing theory, verbal reports are considered a useful data source for investigating cognitive processes and strategies (Ericsson & Simon, 1993). Ericsson and Simon (1993) recommended two forms of verbal reports which they claim can closely reflect individuals’ cognitive processes and strategies and experiences. Divided according to the period when information is accessed, these are concurrent verbal reports and retrospective reports. Concurrent verbal reports, or talk-aloud and think-aloud protocols, are the direct verbalization of information heeded during actual cognitive processing or when task performance is going on (Ericsson & Simon, 1993). Verbalizing thought processes at this period is not a description or explanation of what one is doing, but verbalization of what one is paying attention to while generating answers to the task. Retrospective protocols, on the other hand, are the reports of cognitive processes which have (just) finished (Ericsson & Simon, 1993). Immediately after task completion, some retrieval cues are thought to remain in short-term memory (STM), allowing individuals to recall their thought processes with supposedly high

91

accuracy and completeness. This is especially the case, as Ericsson and Simon suggest, when tasks take less than 10 seconds to complete. However, the longer the period between task processing and retrospective reporting, the more difficult and incomplete the recall. Therefore, stimuli that can help individuals to recall their thought processes are recommended to be included (Ericsson & Simon, 1993), such as a video recording of task performances, notes taken by the participants, or task output (Gass & Mackey, 2000).

Another form of retrospective protocol is retrospective verbal reports or interviews, requiring participants to explain or describe the processes they performed during task performance after they have completed the task. Unlike stimulated-recall, no prompts are given to stimulate the participants’ verbalization. Instead, specific questions are asked to guide the reporting, for example, a) ‘What were you focusing on when you responded to this situation?’ and b) ‘What made you reply in this manner?’(Ren, 2013).

A number of empirical studies (Goh, 2002; O'Malley et al., 1989; Ren, 2013; Vandergrift, 2003) have demonstrated that verbal protocols can provide useful

information on cognitive processing in the context of language processing and also on the strategies users employ to complete language tasks. Verbal report data, in addition, can provide evidence of sources of knowledge applied to complete tasks (Goh, 2002). Table 3.4 lists studies that have used one or more of the techniques to investigate learners’ and test-takers’ cognitive processes in language task performance. Focusing on the studies investigating listening processes, it was found that verbal reports were one of the main data sources; see for instance, in Yi'an (1998) and Vandergrift(1997; 2003). However,

92

each of these three techniques has its own strengths and limitations, as summarized in Table 3.4.

In essence, the key advantage, as found across the three techniques (think-aloud protocols, stimulated recall, and retrospective verbal report) and listed in Table 3.4, is that they seem to reveal processes and strategies used to complete the tasks which are not otherwise directly observable by the researcher(s). Studies that have used the techniques (see Vandergrift, 1997; 2003; Field, 2012) have found that verbal protocol data evidenced strategies and processes activated successfully and unsuccessfully by the participants. In addition, they revealed the knowledge sources participants use to complete tasks.

However, the distinctive benefit of think-aloud protocols (versus the other two

techniques), as pointed out by O’Malley et al. (1989), is that these can tap into cognitive processes and strategies that may otherwise be lost in retrospective techniques such as stimulated recalls or retrospective report/interview techniques due to the time lapse.

93

Study Data aimed to collect Tasks used Language skill(s) focused Strengths found in previous studies Limitations speculated in previous studies 1) Think-aloud protocols O’Malley (1989)

Cognitive processes and strategies L2 learners used

Lecture listening Listening  Well demonstrated strategies used by

participants (Plakans, 2009).

 Revealed how writers with different proficiency levels differed in their decision making and use of strategies (O'Malley et al., 1989; Plakans, 2009; Vandergrift, 1997, 2003).

 Cognitiv processing which was lost in retrospection was described (O'Malley et al., 1989).

Tapped into naturally occurring behaviours (Storey, 1997).

 Limited to the participant’s ability to articulate

information (O'Malley et al., 1989).

 The act of verbalizing distracted from task performance (writing task) (O'Malley et al., 1989; Plakans, 2009; Storey, 1997).

 Participants may provide only a limited range of strategies of which they were consciously aware while performing the task (O'Malley et al., 1989; Storey, 1997).

When not reported in the participant’s L1, thinking– aloud protocol data was somewhat incomplete (Weigle et al., 2013). Storey (1997) Cognitive processes test-

takers engaged in A multiple-choice discourse-cloze test Reading Vandergrift (1997, 2003) Strategies used by L2 French learners Multiple-choice comprehension questions Listening Goh (2002) L2 listeners' comprehension strategies

Listening texts in various topics

Listening Plakans (2009) Reading strategies used by

non-native English writers

An integrated reading- writing task Reading Weigle et al. (2013) Cognitive processes involved in short-answer questions

94

Study Data aimed to collect Tasks used Language skill(s) focused Strengths found in previous studies Limitations speculated in previous studies 2) Stimulated recalls

Yi'an (1998) Test-takers’ test taking processes

Multiple-choice questions

Listening  Provided insightful data on cognitive processes (Field, 2012; Yi'an, 1998).

 Pointed out to different strategies used by successful test-takers (Swain et al., 2009).

 Pointed out to knowledge used in comprehension process and interaction of different listening tactics used (Goh, 2002).

 Could possibly not capture some processes activated by the participants (Field, 2012) (Barkaoui et al., 2013).

 Might not tap into some processes automatically performed by the

participants (Swain et al., 2009).

Not able to provide information on whether the strategies were effective at the production stage (Barkaoui et al., 2013). Vandergrift (1997; 2003) Strategies used by L2 French learners Multiple-choice comprehension questions Listening Goh (2002) L2 listeners' comprehension strategies

Listening texts in various topics

Listening Swain et al.

(2009)

Processes and knowledge test-takers used to complete integrated and independent speaking tasks Independentspeaking Integratedreading- listening-speaking Integrated listening- speaking Speaking

Field (2012) Cognitive processes used in listening performance  Listening comprehension Lecture listening Listening Barkaoui et al. (2013)

Strategic behaviours test- takers reported using when responding to integrated and independent speaking tasks Independent speaking Integrated speaking tasks Reading Listening Speaking

95

Study Data aimed to collect Tasks used Language skill(s) focused Strengths found in previous studies Limitations speculated in previous studies 3) Retrospective interview/report Vandergrift (1997; 2003) Strategies used by L2 French learners Multiple-choice comprehension questions

Listening  Allowed the researcher to follow up the point unclear (Field, 2012).

 Permitted analysis of learners’ attention to information and factors influencing their production (Ren, 2013).

 Pointed out successful and unsuccessful use of

strategies (Graham et al., 2008).

 Some processes may not be reported by participants (Ren, 2013).

Goh (2002) L2 listeners'

comprehension strategies

Listening texts in various topics

Listening Graham et al.

(2008)

Development of strategy use over 6 months listening

Multiple-choice listening task

Listening Field (2012) Cognitive processes used

in listening performance

Multiple-choice listening tasks

Lecture listening

Listening

Ren (2013) Cognitive processes of advanced L2 learners during their study abroad programme

Listening to audio- recordings of different topics

Listening

96

Despite their advantages, concerns have been raised over the use of verbal reports as research data (Barkaoui, Brooks, Swain, & Lapkin, 2013; Crutcher, 1994; Ericsson & Simon, 1993). As verbal reports rely mainly on individuals’ access to information in their own memory and their ability to verbalize such information, concerns over veridicality or the extent to which the verbal information accurately reflects thought processes have been expressed (Ericsson & Simon, 1993). For example, verbal data can be accurate only when individuals are truthful or reporting exactly what they were thinking while

performing the tasks and not reporting things that they think the researcher(s) want to hear. Also, because some processes occur automatically and are unlikely to be available to the participant’s consciousness, some processes may have been activated but not articulated by participants, resulting in incomplete data. This risk has been discussed in the literature on both listening and reading comprehension (e.g., O’Malley et al., 1989; Swain et al., 2009), and some evidence was found in the case of studies employing think- aloud protocols (see O'Malley & Chamot, 1990), stimulated recalls (see Barkaoui et al., 2013; Swain et al., 2009), and retrospective interviews (see Ren, 2013). Furthermore, the ability of individuals to verbalize information and in particular their proficiency in the language used for verbalization were also found to affect the accuracy and completeness of information (O'Malley et al., 1989; Weigle et al., 2013). It is thus recommended to use the participants’ first language in data collection (if at all possible).

A second issue is the risk of reactivity, especially with respect to think-aloud protocols. It has been found that the act of thinking-aloud during task performance may affect or alter the performance. O’Malley et al. (1989) specifically found that the requirement to perform tasks and at the same time report one’s processing appeared to

97

alter participants’ task processing. The act of verbalizing during writing task performance in particular, as noted in O'Malley et al. (1989), Plakans (2009), and Storey (1997), disturbed the writing task performance. As a result, the scores obtained might or might not represent participants’ actual ability.

A further issue concerning verbal protocols relates to the reliability and validity of data coding. The verbal data obtained from these techniques are not a direct report of the thoughts or cognitive processes that the participants performed in completing tasks but the information they attended to during the tasks (Ericsson & Simon, 1993; Kasper, 1998). Hence, in order to arrive at the processes involved, the researchers have to make inferences on the basis of the data provided by the participants. When protocol data are not carefully analyzed (and, for example, no coding reliability is established), the validity and reliability of the results are highly questionable (Kasper, 1998).

In language testing research, verbal reporting data have been used to investigate the construct underlying tasks. Despite the potential weaknesses, the methods have been found useful for analyzing processes and strategies used in task performance (Barkaoui et al., 2013; Plakans, 2009; Storey, 1997; Weigle et al., 2013). As mentioned by Barkaoui et al. (2013), however, protocols are not able to provide information on whether the

strategies are effective during the task performance. They are unlikely to explain whether the processes and strategies used contribute either to success or failure in task output. Therefore, to reveal the effectiveness of each process and strategy, Barkaoui et al. (2013) have advised also analysing test-takers’ task output.

98

In summary, the literature review has indicated both the benefits and

disadvantages of verbal protocols as a data source for investigating cognitive processes and strategies activated to complete tasks. On the positive side, the method has been shown to lead to useful insights into cognitive processes and strategies, which are

otherwise difficult to observe. In language testing studies in particular, previous research suggests that verbal reports provide useful information on test-takers’ cognitive processes and strategies, which is crucial to describe the construct underlying the tasks in question effectively. However, the accuracy and completeness of verbal data can be affected by a number of factors, including the automaticity of cognitive processes, individuals’ abilities to articulate the information heeded, the language used for verbalization, the time period between processing and verbalization, and issues with the data coding process.

Nevertheless, careful research design and procedures for verbal protocol data collection and analyses can help minimize the potential risks of the methods (Green, 1998). Therefore, since the aim of this study was to describe the listening construct by

investigating test-takers’ cognitive processing of listening (RQ 1), it was decided to use a verbal report method to collect data. At the same time, great care was taken to try and avoid some of the pitfalls of this method.

In practice, a stimulated-recall technique was used for three main reasons. First, the task responses in this study were scored for the participants’ language ability. Stimulated recalls, which were conducted after task completion, were considered appropriate to minimize the effect of the data collection technique (if any) on task

performance. Second, as it was necessary to collect data after task completion, stimulated recalls were considered important since they provided participants with some stimuli (a

99

video recorded during the task performance and the notes taken during the listening) to stimulate their thought processes. Third, as the study concerns research on listening, using a think-aloud would likely be very disruptive since participants have to talk while trying to listen.