Document analysis

‘Document analysis is a systematic procedure for reviewing or evaluating documents—both printed and electronic (computer-based and Internet-transmitted) material’(Bowen, 2009). ‘As a research tool, documents can produce intensive rich descriptions of a single

phenomenon and represent a specific version of realities for specific purposes’ (Stake, 1995; Flick, 2014).

Mixed-methods studies sometimes include document analysis either as a standalone method or as a supplementary research tool. For example, in an attempt to explore a teacher training programme designed to address the assessment needs of LCTL18 educators working in short- term foreign language programmes in the US, Montee et al. (n.d.) conducted a mixed-method research using questionnaires and document analysis. Document reviews became the primary source of data to identify the participants’ performance while developing assessment tasks in their language programmes. Also, Sogunro (1997) used questionnaires combined with interviews, document analyses and direct observations to examine the impact of training on leadership development. The documents were additional tools which provided supplementary information about the training programme’s history, goals, objectives, enrolments and

substantive content.

In this study, documentary sources refer to any written classroom-based assessment in the form of quizzes, midterms and/or final exams. These documents serve a variety of purposes as part of the research undertaken. First, the documents provide core research data regarding the participants’ actual practices of grammar assessment and the factors influencing them (RQ 2). Second, information and insights from the documents allow the detection of any

mismatches between what the participants say they do and what they actually do (RQ 3). Third, document analysis played a vital role in data triangulation.

However, document analysis is not always advantageous. Here are some of its limitations: • Insufficient detail: Documents are produced here for assessment purposes and are

created without a research agenda. Consequently, they usually do not provide sufficient details to answer a research question (Bowen, 2009). Therefore, in this study document analysis was used in combination with other qualitative tools, such as interviews and retrospective thinking, as the means of triangulation.

• Low accessibility: Due to the nature of the documents to be obtained (written exams), participants may be reluctant to show these exams because they are considered as official documents. However, in this study, once permission was given from their administrative parties, exam papers were made available based on the participants’ willingness and cooperation.

• Bias selectivity: Because this study aims to analyse classroom-based grammar assessment in the form of written exams, only grammar teachers were requested to submit samples of these written exams. Also, the teachers got to choose which exams to deliver to the researcher, which could be modified versions of examination modes that had been administered to their students.

These are some potential flaws, but the important role document analysis plays in data collection outweighs these limitations.

4.4.4.1 Data collection for the document analysis.

Hardcopies of the EFL teachers’ exams were collected according to the data collection timeline (see Table 10). Documents were submitted after the participating teachers administered them to their students. Only hardcopies were obtained because these exams were signed off by the respective departmental chairpersons. I was allowed to have these copies because I had already obtained permission letters from educational authorities, otherwise the teachers would not have shared them with me. Some of the exam copies were given to me during the retrospective thinking sessions (N = 20), while others (N = 8) were left in a sealed envelope at the department chairpersons’ offices for me to collect.

These documents represented in the form of midterms, quizzes and finals are of high-stake status in the sense that their outcome is used to make important decisions about students’ advancement (grade promotion or graduation for students). These assessment tasks are assigned specific marks (see section 4.3.1.2) which should all add up to 100 points. Students must score at least 60% to pass any course. Exam scores have direct consequences of

students’ passing or failing. Failing has major disadvantages, such as being forced to retake classes until they can be passed, not being allowed to progress to the next level or even being expelled due to low GPA. Through scores of midterms, quizzes, homework or any other course work during the semester, students are able to know if they have enough marks to pass the course after sitting for the final exams. if students do not get at least 60% of their overall course work before the final exams, they are allowed to drop the course without affecting their GPA negatively.

4.4.4.2 Participants and sample information.

28 hardcopies of EFL teachers’ grammar written exams represented the sample of the document sources. The majority of the documents were midterm exam papers (N = 24), followed by final exam samples (N = 3) and, finally, one quiz. Out of the total participants who wrote these assessments, 22 were female. The majority of them were Saudi (N = 25) and PhD holders (N = 17). The table below summarizes the participants’ information and the document types.

Table 11.

Participants and Documented Information

Facility Document Gender Qualification Country of Origin

Male Female MA PhD Saudi Non-

Saudi A Midterm 5 12 6 11 15 2 B Quiz Ø 1 Ø 1 1 Ø C Midterm 1 6 5 2 7 Ø D Final Ø 3 Ø 3 2 1 Total (%) 6 (21%) 22 (79%) 11 (39%) 17 (61%) 25 (89%) 3 (11%)

4.4.4.3 Document data analysis.

Because all the documents obtained were hardcopies, the analysis was done manually and then the tables and figures of the results were created as MS Word documents. Document analysis involved three stages: skimming (superficial examination), reading (thorough examination) and coding. Thus, I first went through all the exam papers and familiarized myself with their content. Afterwards, I started reading each exam paper and, using different

highlighters, also began coding. The coding was a straightforward process. Since it was manual, a code was written alongside each section of the exam paper. The codes were then clustered into categories that reflected the themes (see Fig. 23).

Next, content and thematic analyses were carried out. Content analysis allowed the

identification of pertinent information, which were later organized into categories related to the relevant research question (RQ 2). I constructed my own document analysis form based on one created by the National Archives and Record Administration (see Appendix K) to summarise the document contents. Information gleaned from the content analysis of teachers’ written exams target the type, medium, author, audience, context, purpose, number of items, and focus (Table 12).

Table 12.

Content analysis of exam papers

Thematic analysis enabled pattern recognition and resulted in the generation of the salient research themes. During thematic analysis, I reread and reviewed the coding and the category construction to explore the themes pertinent to teachers’ practices of EGA. Figure 23 shows the themes resulting from the document analysis.

Figure 23. Resultant themes and categories from document analysis.