Instruments

All instruments in this study were piloted (questionnaire, interviews, retrospections and document analyses). Piloting the research tools was beneficial because various flaws were found in some of the data collection tools, which later led me to modify these tools before conducting the main study.

4.9.2.1 The questionnaire.

The questionnaire was piloted in the second week (April 1st) after recruiting the participants and obtaining the signed consent forms. Hardcopies were given to the programme directors in both sectors (male and female), who were requested to hand it out to the participants. The questionnaire included 50 items addressing various issues related to EFL teachers’ beliefs about English grammar assessment. The items of the questionnaire were mixed-up and written in English. For the purpose of analysis, items were grouped into their pre-set

categories. Items 4, 7, 10 and 50 were excluded from the analysis because the majority of the participants refrained from answering them, which indicated they were problematic and, hence, were deleted from the final draft of the questionnaire. Due to the small sample size, the collected data were analysed based on descriptive statistics. What follows is a description of the results.

Figure 25. Teachers’ general beliefs and attitudes about English grammar assessment.

As shown in the above figure, the two high agreement results (items 1 and 3) indicate positive attitudes towards English grammar assessment. These results could be attributed to the fact that the majority of participants are experienced teachers. In addition, one might expect less experienced teachers not to be sure of their ability to conduct grammar assessment appropriately, which can explain their middling on the second item. This could also be related to assessment literacy and identify whether the participants had undergone any relevant training on any sort of assessment.

As one might expect, being experienced teachers, the participants saw the value of grammar assessment for learning and did not agree with item 4. Also, since the participants themselves would construct their own assessment tasks, especially the male teachers, they did not feel pressurised by their task greatly. That pressure is more likely to arise among female

respondents, since an external body – the Ministry of Education– constructs the final exams. In this context, most of the teachers can design their exams to fit whatever they have taught.

Thus, they are not highly pressurized and hence, this item witnessed scores achieved below midpoint.

Figure 26. Teachers’ beliefs about the purposes of English grammar assessment.

The responses to these items reveal that the majority of participants believe that grammar assessment fulfils different purposes: summative, formative and accountability. High scores were achieved for items 1, 3 and 6, which suggest that the participants agreed more that grammar assessment serves a more summative function than formative. The lowest scores were for items 7 and 8, related to grammar assessment used for accountability purposes. The teachers believed that grammar assessment should be used to diagnose and report students’ strengths and weakness in relation to grammar learning. This could be associated with the nature of the context where assessment is used to assign scores and report learning progress.

Figure 27. Teachers’ beliefs about English grammar assessment methods.

Concerning the content and delivery of English grammar assessment, participants seem to respond positively, with 14 items involving scores above midpoint. Interestingly, items 3 and 15 saw the highest scores, which indicate teachers’ agreement that the content of grammar assessment should focus more on the level of communication. Agreement on both items also suggest a level of consistency among the participants’ responses. Item 5 is ranked second highest, which means that the teachers believed that grammar is better assessed in an integrative manner. Again, this response corresponds with items 3 and 15 that focus on communication. This, however, does not deplete the value of the segmented content of grammar assessment, in which items 4, 6, 7, 8, 11, 12, 13 and 14 heralded scores of high midpoints. This indicates teachers’ positive attitude towards these items.

As for the means of delivery, teachers seem to agree positively with items 1, 2 and 9 but respond negatively to item 10. This shows that teachers believe that computer-based and paper-based assessment both have merits when it comes to grammar assessment. Participants' negative response to item 10 indicates once again a level of consistency (i.e. reliability, cf. 4.7.2) in the participants’ answers. Since they agreed that grammar should be assessed continuously throughout the course (item 9) it would be logical to disagree with conducting grammar assessment only at the end of a grammar course.

Figure 28. Teachers’ beliefs about English grammar assessment formats.

The two highest scoring tasks mark an interesting contrast. Item 1 targets spontaneous spoken communicative ability and is more message-focused, while item 9 depends on

awareness/metacognition of language and typically written language focus. Responses to these items would yield results once, compared to their actual assessment tasks, to identify consistency or incongruity between the teachers’ beliefs and practices. To summarise, the

participating teachers seemed to agree positively with the various grammar assessment formats suggested in this section.

Figure 29. Teachers’ beliefs about their roles and the sources used in constructing English grammar assessment tasks.

As seen in Figure 29, the purpose of items in columns 6, 7 and 8 was to explore EFL

teachers’ beliefs about their preferred source with regard to constructing grammar assessment tasks. The results indicate that most of the participants agree that ready-made grammar assessment items, either extracted from textbooks or adopted from previous assessment tasks, were appropriate sources to be used when assessing grammar.

Items 1, 2, 3, 4 and 5 refer to teachers’ beliefs about their role in constructing grammar assessment. The participating teachers believed that they should construct their own assessment tasks, preferably individually (female participants scored the highest) and

collaboratively otherwise. Moreover, the participants seemed to believe that using self-and- peer assessment could be acceptable, agreeing positively on this stance. However, item 2 shows a huge contrast between the male and female respondents. Female participants were less inclined to support the idea that exams are best written by experts, while male

participants were more accepting of this notion.

4.9.2.2 Interviews.

Piloting the interview questions was crucial for two reasons: first, I needed to test myself as an interviewer, since this is the first time I have engaged in interviews. I wanted to know how I could carry myself around my interviewees and how acceptable I would be to them. Second, I had to intuit what sort of answers the interview questions would generate.

The interviews took place on various days during the pilot study. Nine teachers (F = 5; M = 4) agreed to participate in the interviews, which were about 20–30 minutes long. Interviews were audio-recorded and then transcribed into MS Word documents in preparation for analysis. Each document was given a name that would help me identify my participants and simultaneously maintain their anonymity. Once the transcription of audio recordings was complete, the documents were imported to MAXQDA to begin the coding process (see Appendix L).

Figure 30. A screen print of how the coding system and the transcript documents are presented in MAXQDA.

The coding process involved identifying meaningful segments that were considered as relevant to teachers’ beliefs and practices. I assigned each code a colour and an appropriate label. The first teacher’s data that I analysed resulted in a large number of codes. However, as this process went on, the number of codes tended to decrease as the themes and categories emerged.

Preliminary data analysis showed that there are commonalities among the participating teachers’ beliefs about EGA (Figure 29). For example, all the teachers stated that grammar is assessed to identify any grammatical mistakes and to report on the progress of learning. This belief indicates a summative role of grammar assessment. This result coincides with what has been found in the questionnaire responses where the majority of teachers believed that

grammar assessment fulfils various functions and summative purposes witnessed the highest scores.

With regard to teachers’ roles in constructing assessment tasks, the results were very

interesting. Female participants strongly asserted that a teacher should construct her/his own assessment tasks or have a saying in such construction matters. This again is supported by how female participants responded to item 2 in Figure 29. The result shows that the female teachers disagreed with assigning the role of assessment construction to an expert or otherwise. The male participants, on the other hand, were more receptive to the idea of allowing a second or third party to design the assessment tasks, which they would just administer to the students. This amenable perception is also detected in their responses to item 2 in Figure 29. This can be attributed to the fact that male teachers choose their exams or quizzes (second party) from test banks. At the end of each level, students take Oxford

placement tests as a final exam (third party).

As for grammar assessment methods, the teachers agreed that grammar should be assessed in an integrative manner, focusing on production level to enhance communication. Their

practices, on the other hand, show some discrepancies. Female teachers assess grammar separately (discreet-point); there is a section in the midterms and the final examination where students are asked only about grammatical rules. This kind of practice is influenced by the

guidelines dictated by the academy and approved by the Ministry of Education. Males, however, assessed grammar in an integrative milieu with respect to writing and speaking during midterms, but they would resort to assessing grammar separately in their quizzes because their students would ask them to do so (because those questions would be easier to answer and guarantee high marks).

Other relevant themes, factors and comparisons between results, are addressed in more detail in the main study.

4.9.2.3 Document analysis.

Documents in the form of written exams (quizzes, midterms and finals) were collected on week 1. Surprisingly, document collection was easy. As I asked for samples of their exams, the programme directors (both male and female) offered me a large number of samples on the spot. Unfortunately, the majority of these samples were hardcopies, and so during the

analysis I coded the scripts manually. Later, I generated a content analysis form in an MS Word document (see Appendix K) that could be easily imported to MAXQDA for the purposes of comparing and contrasting the themes and categories across the participating teachers.

4.9.2.4 Retrospection.

As it was not possible to conduct ‘think aloud’ protocols, I conducted retrospection instead (see 4.4.3 for details). Only one participating teacher agreed to the retrospective thinking. This teacher sat with me and gave me a sample of a quiz she had designed for her students. I asked her to walk me through the process of constructing the quiz; below is the transcript of the whole session (3 min 05 sec).

1. Mashael: This is your quiz, yeah?

2. Hana: Yes, I just give quizzes to train my students for the midterm and the finals. No points are allocated on this.

3. Mashael: So how did you decide what to put in there?

4. Hana: I just finished the chapter of tenses, so that is what I am testing them on.

5. Mashael: How did you come up with these items? 6. Hana: I just took everything from the book. 7. Mashael: I see. You mean you took the sentences…

8. Hana Yes. No. These are exercises I don’t do in class and ask them to do at home. Similar to what they will have in the midterms and finals.

Evidently, the session was too short and not enough data was generated. I strived for a richer and deeper context for my main study. The retrospective thinking showed that the teacher constructed her quiz as a training task (not required by the academy): ‘I just give quizzes to train my students for the midterm and the final. No points are allocated on this’ (line 2). In the process of constructing this quiz, she seems to be following the academy guidelines of midterm and final exam construction: ‘Similar to what they will have in midterm and final’ (line 8).