Data collection materials and instruments

The data collection materials and instruments consisted of a pre-task questionnaire, a pre-test for the selection of target idioms, reflective journals, the immediate and delayed posttests, stimulated recall protocols, post-task survey and interviews, along with computer equipment and software. The details of these materials and instruments are elaborated below.

3.6.2.1. Pre-task questionnaire

The questionnaire focused on the participants’ background information, their attitudes toward peer-peer collaboration, English vocabulary learning, the use of text- based online chat for communication, and their knowledge of English idioms. It involved eight short-answer questions that elicited their demographic information such as

nationalities, L1s, TOEFL or IELTS scores, ages, major areas of study, length of learning English, and residence in the US. It also contained two open-ended response items: one asked the participants to elaborate on their experiences of and use of strategies for learning English vocabulary, and the other instructed them to write down the two idioms they knew so that how they conceptualized English idioms can be better understood. Additionally, the questionnaire used twenty five Likert-scale items ranging from “strongly disagree” to “strongly agree” to determine their perceptions of and feelings about collaboration with their peers, the usefulness of discussions with peers for

vocabulary learning, and the role of text-based online chat in their daily communication (see Appendix A).

3.6.2.2. Pretest and target idioms

The pretest consisted of forty idioms compiled from various sources, including Simpson and Mendis’ (2003) list of useful idioms for English for Academic Purposes (EAP) curricula and their findings on the most frequent idioms in the Michigan Corpus of Academic Spoken English (MICASE), along with the idioms recommended by O’Keefee et al. (2007) as suitable for ESL teaching and learning. The reasons for confining the target idioms to highly frequent ones in academic discourse are their relevance to the participants’ learning needs and the comparatively high level of difficulty they may pose for intermediate English learners. According to Simpson and Mendis (2003), idioms are widely employed in monologic and dialogic academic speech, and their lack of

transparency of meaning is very likely to cause L2 learners’ listening and reading comprehension problems. Therefore, the collaboration on the definitions of English idioms closely resembles the real-life language issues that the participants will be required to solve in their academic studies. Additionally, the substantial metacognitive efforts needed for decoding the meanings of English idioms on the part of L2 learners (Cooper, 1999) justifies the choice of them as the learning targets for the current study simply because in generating collaborative dialogue around English idioms, the

______________________________________________________________________ I. I don’t remember having seen this word before.

II. I have seen this word before, but I don’t know what it means.

III. I have seen this word before, and I think it means ______ (synonym or translation). IV. I know this word it means ________(synonym or translation).

V. I can use this word in a sentence: _________. (Write a sentence) (If you do this section, please also do Section IV)

_______________________________________________________________________ Figure 4. The vocabulary knowledge scale (VKS) (Based on Paribakht & Wesche, 1997, p. 180)

participants are able to demonstrate how they, in face of challenging L2 issues, make use of languaging for the resolutions and internalization, without the assistance from the experts such as the teacher.

The aforementioned two native speakers reviewed the forty idioms and confirmed their accuracy and common use in academic speech. The test asked the participants to indicate their familiarity with the meaning of these forty idioms through their responses to the Vocabulary Knowledge Scale (hereafter referred to as VKS). Originally developed by Paribakht and Wesche (1993), the VKS measures L2 learners’ knowledge of the definitions of a particular word on a five-point scale ranging from “total unfamiliarity through recognition of the written word and some idea of its meaning, to the ability to use the word in a sentence” (p. 15; see Figure 4). For clarity, I replaced the use of “word” in the VKS with “idiom” to better reflect the learning objectives of the current study (see Appendix B for the list of the forty idioms and a sample idiom question using the VKS). In completing the pretest, the participants were instructed to put a check mark by the level that represented their current knowledge about the meaning of the idioms. Among the forty idioms, sixteen that all participants indicated that they had never seen before or had seen before but did not know what they meant (marking I or II on the VKS) were selected as the target idioms for instruction. To reduce the cognitive burden on the participants, the sixteen target idioms were structurally and semantically different from each other (see Appendix C for the target idioms used in the current study and their definitions). Once again, the two native speakers’ opinions were referred to for the confirmation of the structural and semantic differences.

3.6.2.3. Reflective journals

After completing the idiom-in-context and text-reconstruction tasks, each participant was instructed to write a reflective journal of his or her experiences of collaborating with peers on the definitions of the target idioms in SCMC. The writing prompts for the reflective journals were adapted from Lee’s (2008) guidelines for student reflection on online feedback negotiations and error correction, and specifically asked the participants to elaborate on their overall experiences of online exchanges with their partners, provide details about the dialogic co-construction of the meaning and the degree to which they perceived the collaboration as beneficial, describe the moments of the SCMC-based dyadic interaction they found particularly useful or challenging, and offer additional comments to explain their feelings about learning the target idioms through pair work (see Figure 5). As Dörnyei (2007) noted, reflective journals are “by definition

an insider account” (p. 157) in that they “try and elicit the participants’ own descriptions and interpretations of events and behaviors…as they keep records of their own feelings, thoughts, or activities” (ibid). Additionally, the fact that the participants wrote their reflective journals immediately after their collaborative work under my supervision offered “a self-report format that reduces inaccuracies stemming from not remembering something correctly, because when writing their entries participants recall recent rather than distant events” (ibid, p. 158). The journal entries were saved into the archives of the “Journal” activity plugin of Moodle and subsequently retrieved for data analysis.

3.6.2.4. Immediate and delayed posttests

In this study, the posttests assessed the participants’ understanding and retention of the meaning of the target idioms. In other words, the current study regarded

“vocabulary acquisition as only one aspect of vocabulary learning, learning word meaning” (Kim, 2008, p. 119). Particularly there are three types of posttests: immediate posttests, short-term, one-week delayed posttests, and long-term, two-week delayed posttests. The participants took the immediate posttests right after they completed the idiom-in-context tasks and the short-term delayed posttests immediately after the

completion of the text-reconstruction tasks. It should be noted here that the participants’ performance on the short-term delayed posttests was viewed as the manifestation of the delayed effects of SCMC-based collaborative dialogue rather than immediate gains given their prior exposure to the meanings of the target idioms in the idiom-in-context tasks.

Both the immediate and short-term delayed tests were intended to evaluate the participants’ recognition and production of the target idioms. For the assessment of recognition or receptive knowledge, the immediate posttests used matching questions in

which the participants were asked to pair the target idioms with the corresponding definitions. The definitions were taken and adapted from a wide array of online dictionaries such as Longman Dictionary of Contemporary English Online (www.ldoceonline.com), The Free Dictionary (www.thefreedictionary.com), and

Merriam-Webster Online (www.merriam-webster.com). The assessment of production or productive knowledge in the immediate posttests, on the other hand, involved supplying the definitions of the target idioms. The format of the short-term delayed posttests was the same as the immediate posttests, and to minimize the effect of memorization, in the delayed posttests the definitions were phrased differently and the order of the eight target idioms was also altered. Both the immediate and short-term delayed posttests took approximately ten minutes to complete. To ensure that the participants’ performance on the posttests was a true reflection of their learning of the target idioms as a result of their collaborative dialogue, they were not informed that they would take a posttest in advance. For the long-term delayed posttests, the participants were instructed to use the VKS to indicate their knowledge of the target idioms. An independent trained coder who was also a doctoral student majoring in Applied Linguistics and Technology and I scored and analyzed their responses (See Appendix F for both the immediate and delayed posttests).

3.6.2.5. Stimulated recall protocols

As Fox-Turnbull (2009) pointed out, stimulated recalls have been widely adopted in research studies concerning classroom interaction for gathering introspective data. Especially for studies on collaborative dialogue, stimulated recall data reveal the participants’ “perspective of their behavior during their interaction which may not be apparent from the recorded pair talk alone” (Watanabe & Swain, 2007, p. 127). In this

study, stimulated recall protocols were developed for the elicitation of information regarding 1) the participants’ thoughts during the occurrence of IFD episodes, 2) their grasp of the meaning of the target idioms, and 3) clarification of fragmented sentences and spelling errors (See Appendix G). Additionally, in accordance with the interactional features of SCMC discourse, a few adjustments were made to the protocols. For instance, for the patterns of interaction, following Watanabe and Swain (2007, 2008), the focus was on Storch’s (2002) description of the characteristics of the collaborative, dominant, passive, expert, and novice role of the participants. However, in real-time communication, it was very unlikely that one member of the dyads would formulate a long stretch of discourse without the interruptions of his or her partner. As a result, if a participant produced significantly more turns than his or her partner, he or she would be asked during the stimulated recalls what he or she was thinking at that point of time.2 Likewise, a participant would be asked to explain his or her own thoughts if he or she generated significantly fewer turns than his or her partner or there were time lags between the initiation of and reply to a message. The participants’ responses to the stimulated recall questions were recorded by the Macintosh software application GarageBand.

3.6.2.6. Post-task survey and interviews

A survey was used to inquire about the participants’ feelings of their collaboration with their partners on the target idioms via text-based online chat. It consisted of sixteen items that asked the participants to rate each one on a Likert-scale ranging from “Strongly Disagree” to “Strongly Agree” (see Appendix H). A follow-up interview with each participant was also conducted after the survey. It consisted of ten semi-structured

2_{Following Smith’s (2003) definition of turns in SCMC, a turn in this study is counted as “a} transfer of the ‘floor’ from one participant to the other” (p. 42) rather than each line of the chat transcripts.

questions that sought to clarify the participants’ responses to the survey questions and gather their further comments and thoughts on working with their partners, the use of SCMC for collaborative interaction, and the usefulness of SCMC-based collaboration for learning the target English idioms (see Appendix I). The dialogues that took place during the interviews were audio recorded via GarageBand.

3.6.2.7. Computer equipment and software

Much of the data collection was conducted through the plugins on Moodle. Particularly text-based online chat was operationalized through the Chat activity plugin. During the data collection period, eight chat rooms were set up that allowed the

participants to engage in real-time interaction with their partners (see Figure 6). Their chat transcripts were automatically saved after the completion of the tasks and were retrieved and reviewed for the analysis of collaborative dialogue. Other plugins such as Wiki, Journal, Quiz, Questionnaire, and Survey were employed to administer the tasks, record the participants’ reflections, submit their responses to the posttests, and document their perceptions and attitudes. The Macintosh software application QuickTime Player was used for screen recordings.