Quantitative research tools

Unlike some research that has investigated cognition through teachers’ self-reports – that do not necessarily reflect actual practices (Robson, 1993) - this project seeks to look at relationships with actual practice. Therefore, observing actual class behaviours and actions was an essential research tool for the present study. Two main types of observation practices are commonly used to investigate teachers’ and/or students’ behaviours: participant observation, and structured observation (Cohen et al., 2011). The former is widely used by participatory practitioners involved in investigating assumptions about classroom practice, while the latter is typically designed and carried out by a more detached researcher to evaluate participants’ behaviour in different contexts,

79

and is “a systematic method of data collection, where there is considerable pre-coding and the observation takes the form of recording when, how often, or for how long the pre-coded behaviours occur”, (Allen, 2012:109).

The present study used structured observation to gain a systematic analysis across the classes observed. Each observation session lasted for 45 minutes, and involved a standardization procedure: I observed teachers each teaching three/four of their classes, in their own classroom. Observation session data were analysed quantitatively and qualitatively.

4.3.1.1 The observation grid

The use of an observation scheme has the advantage of minimising observer bias as the data that are collected are pre-determined (Nunan, 1992a; Lightbown and Spada, 1999). In addition, when a previously validated scheme is used, researchers are able to replicate the work of others in different settings (Gerrish and Lacey, 2010). Nonetheless, a number of researchers (Chaudron, 1988; Seedhouse, 2004; Borg, 2008) believe that the use of highly structures schemes for coding and recording behaviours might have the disadvantage of preventing the capture of unanticipated classroom events and behaviours that can be crucial to the research question.

Despite the above criticisms, an observation grid was employed in the present enquiry to address a potential limitation related to the particular research context in Albania. One of the main problems I have encountered in my previous research projects in the country was the continual concern of Albanian teachers about the use of any recording device during the observations and interviews (privacy issues are fully discussed in section 4.5.1). Therefore, to solve this potential limitation, an observation scheme was used to document and quantify what went on in the classrooms observed. The Communicative Orientation of Language Teaching (COLT) – a validated scheme widely used in the observation of teaching and learning in L2 classes - was chosen as an instrument to collect observation data. COLT, developed by Allen et al. in 1984 and believed to be one of the most sophisticated observation schemes (Nunan 1992a; Byram and Hu, 2013), features two parts. Part A (see Appendix 8) contains five parameters that aim to describe the type of the classroom activities and episodes (such as drills, role-plays, dictations), the participant organisation (i.e. class, group, or individual), the content (i.e. management, language, or other topics), the content control (i.e. teacher, students, or teacher/text/students), the student modality (such as speaking, writing, and so on), and the materials used (i.e. the type and the source).

80

Part B “consists of an analysis of the communicative features occurring within each activity” (Allen et al., 1984:240). As shown in Appendix 8, it provides a set of seven categories, each of which is covered by several subcategories in the coding scheme. The selection of the categories, according to Allen et al. (1984), is informed by the L2 theories of the time when the scheme was devised.

The scheme has been widely used in classroom-based research to examine the relationships between teaching and learning (Spada, 1986), to investigate the role played by individual learner variables (Harley, 1993), to help teachers themselves reflect on their own teaching (Ellis, 1997), and so on. For the purpose of this study, an adapted version of the COLT scheme was created (see Appendix 9). The adapted version of the COLT scheme is further explained in the following paragraphs.

It was seen in chapter 2 that there are four central concepts to CLT: 1) content-based; 2) whole language based; 3) learner-centred; and 4) cooperative. In accordance with these features

of CLT, three central categories were created in the adapted version of the COLT to see 1) whether teachers engage learners in activities that focus on the use of language or on the language learning process – the category created for this purpose was “activity” and the subcategories were “productive”, “receptive”, “system”, “function”, and “other”; 2) whether teachers make use of language activities that integrate different skills and aim to develop students’ fluency or focus mainly on their accuracy – the category created for this purpose was “focus” and the subcategories were “fluency”, “accuracy”, and “other”; and 3) whether teachers believe that languages are best learned with the help of the others through use and exposure or through continual lecturing – the category created for this purpose was “interaction” and the subcategories were “whole class”, “pair or group work” and “individual”. Twelve subcategories were included to pinpoint the type, focus, and interaction patterns during each activity. In addition, to reflect the central place of materials in the present study, teachers’ instructional behaviours were classified into three major groups: “instructions similar to teacher’s book”, “instructions partially similar to teacher’s book”, and “instructions different from teacher’s book”. Lastly, a new category (“notes”) was generated so that the observer could write details in case the instructions were partially similar to the guidance given in the teacher’s book (i.e. what exactly was similar or different).

Cone and Foster (2006) note that any observation scheme that is created or adapted by the researcher should be piloted and validated before it is used for research purposes. Accordingly, the observation scheme used in this project was piloted twice. Firstly, I had the opportunity to assess the reliability of the modified version of COLT while completing a small-scale study in preparation for my

81

main research study in March 2011. I used the checklist of target teaching behaviours while observing a colleague of mine teaching a class of English for Academic Purposes (EAP) at York University, Toronto.

One potential problem was identified with the adapted scheme. It was realised that there were observations that could fall into more than one “type” of subcategory. For example, a spoken grammar exercise was coded simultaneously under the subcategories “conversation”, “grammar”, and “exercise” as it was employed as an activity (i.e. exercise) that aimed to develop students’ speaking skills, as well as to reinforce their grammar knowledge.

Therefore, the use of the adapted COLT scheme was revised in light of the first pilot test results. The new grid (see Appendix 10) included most of the same categories. However, some changes were made to the subcategories of the category “activity”. Indeed, six new mutually exclusive subdivisions (i.e. “listening”, “reading”, “writing”, “speaking”, “pronunciation”, and “system”) were created to better reflect the type of activities that take place in a class. Likewise, to make it easier for the researcher to identify the focus of an activity, the previous three behaviour checklists of the category “focus” (i.e. “fluency”, “accuracy”, and “other”) were replaced by “meaning”, “form”, and “use”. For the same reason, a new subcategory (i.e. “class – teacher-led activity”) was added to the instruction section. In addition, the importance of time in this observation scheme was recognised by adding a new category, “timing”, with the following three subcategories: “start”, “finish”, and “duration”. Lastly, the adapted COLT scheme version 2 featured another new category “language use” as the amount of L1 use in an L2 class can be an indicator of how communicative the class is. Indeed, Majer (2003) claims that excessive use of L1 by L2 teachers can be ineffective because it has the potential of turning a seemingly communicatively L2 classroom into a content one in which all authentic communication is done in L1.

The inter-reliability of the adapted COLT scheme (version 2) was assessed during a pilot study conducted in January 2012 with an Albanian EFL teacher who teaches English at a private EFL school in a southern city in Albania. The instructor, a former colleague of mine, was observed teaching a sixty-minute EFL class; the data were independently coded by two raters (a senior lecturer in Linguistics at the University of Vlora, and me).

Two main issues related to the structure of the observation scheme were identified. It was observed that the instructor shifted the focus of two activities from fluency (material target) to accuracy (teacher target). Likewise, on one occasion, the teacher asked the class to do the activity in their groups (teacher’s instruction) but performed a teacher-led classroom discussion (actual class

82

interaction). The adapted COLT scheme version II could not account for these significant events and, thus, two new categories were created: “teacher target” (i.e. teacher actual use of material) next to “material target” (i.e. textbook authors’ intended use of material), and “class dimension” (i.e. the interaction pattern the teacher selected to do the activity) as opposed to “teacher’s instructions” (the interaction pattern recommended by textbook authors to do the activity). Moreover, a new subcategory titled “other” was added to the “activity” category after realising that there was no room in the actual grid to describe non-teaching related activities, such as classroom management and discipline issues that may occupy “a considerable amount of the actual class time” (Savage and Savage, 2009). Lastly, a modification to the layout of the grid was made to render the subcategories “instructions” and “if textbook material12)_{” more visually salient so that the similarities and}

differences between the teacher’s instructions and teacher’s book instructions were more highlighted. During the field data collection, the final structure of the observation scheme (see Appendix 11) was completed partially during the lesson observed (i.e. while the teacher was teaching, I jotted down the start and the finish time for each activity, the way the teacher instructed the student to do the activity, the activity, teacher’s focus of the activity, the class dynamic, and the language the teacher used), and partially after the observation session (i.e. If the teacher was using textbook materials, I checked after the observation session whether the teacher’s instruction were similar to the textbook/teacher’s book instructions, as well as whether the teacher was shifting the focus of the activity or not. If the teacher was not using textbook materials, I found out the source of the material during the post- observation interviews). See Appendices 34 and 38 for two completed observation schemes.

Twelve observation sessions were recorded (see Appendix 33 for the transcript of one of the lessons observed), so I had the possibility to review the observations at home and check the accuracy of the in-classroom observation grid completion. Three observation sessions were not recorded, and the notes jotted down during the observation sessions were used to document and quantify what went on in the classrooms observed.

Cohen et al. (2011) claim that several factors (such as the influences of the researcher’s own interests on observation judgements, influences of the observer’s physical presence on participants’ behaviours, and the possibility that the researcher becomes unaware of important antecedent events while exploring the present) might prevent researchers from collecting representative data during observation sessions.

---

12) _{Activities based on the textbook content, as opposed to teacher-made activities or activities from other}

83

To ensure trustworthiness and credibility in the present study, a number of actions were taken into consideration during the preparatory phase. They are fully discussed in section 4.5.