Chapter 3 Research design and methodology
3.5 Methods and Instruments II: Automated text analysis software
3.5.1 Automated analysis software and judges’ checklist: design
(FYA). Although content analysis is perceived to be highly rooted in quantitative-based approaches (Bryman, 2001), in this part of the study, the data was reliant on judges’ opinions, which to some extent reflected qualitative judgments. Nevertheless, the analysis was based on quantitative approaches as a prepared checklist was used for the judges to indicate their decisions (see Appendix 8 for checklist).
The use of expert judges in L2 research is widely acknowledged as a useful method (e.g. Bachman et al., 1996; Bachman 2004; Davies, 2011) as well as for testing and content validity (e.g. Pilliner, 1968; Davies, 1990; Green et al., 2013).
3.5.1 Automated analysis software and judges’ checklist: design and content
In this sub-section, the variables that were to be tested were identified following (Green et al., 2010). Also, a checklist based on Weir et al (2009), and Weir (2005, pp.56-84) was designed for the use of the expert judges in making their evaluations.
It is useful, at this point, to list all of the context validity features that were to be examined and to indicate in exactly what manner they would be tested in this research. Table (3.2) below sets out these features and indicates by which method they were to be tested (judges, software or both):
Table 3.2 Contextual features and methods of testing
CONTEXT VALIDITY FEATURES
METHODS Expert judges Automated analysis software Linguis tic de m a nds: Ta s k i nput a nd out put
Overall text purpose √
Writer-reader relationship √ Discourse mode √ Functional resources √ Grammatical resources √ √ Lexical resources √ Nature of information √ √ Content knowledge √ Ta s k s e tt ing Response method √ Weighting √ Knowledge of criteria √ Order of items √ Channel of presentation √ Text length √ Time constraints √
The table below contains a column for the variables that were to be tested by expert judges’ opinions and a column for variables that were to be tested by the different software discussed above (see Section 3.5). For the variables examined by the judges, statements were constructed to enable the judges to express their judgements. These statements are presented in Table (3.3):
113
TABLE 3.3 Variables were examined by expert judges and different automated software
Variables for judges Variables for software
Linguistic demands Task input and output 1. Overall text purpose
√ The category that best describes the overall text purpose
N/A
2. Writer-reader relationship
√ Identify the intended audience/reader of the text that is targeted by the writer. Hyland’s 2005 cited in Weir et al. (2009, p.138 and p.110)
N/A
3 Discourse mode
√ Genre: Identify the most appropriate category for the text.
Whether it is textbook, magazine/newspaper article,
research/academic journal article, report Weir et al. (2009, p. 137)
√ Rhetorical task: Identify the most appropriate category for the text. Exposition,
argumentation/persuasion/evaluation, historical biographical/autobiographical narrative
√ Pattern of exposition: Identify the pattern(s) used in the text.
114 Define, describe, elaborate, illustrate, compare/contrast, classify, cause/effect, problem/solution, justify
√ Rhetorical organisation:
The organisational structure of the text is… explicit or not explicit
4 Functional resources
√ Identify the most appropriate category for the text. Ideational, manipulative, heuristic, imaginative
(Bachman and Palmer, 2010; Weir, 2005)
N/A
5 Grammatical resources √ Grammar:
The sentences in the text are: range from mainly simple sentences to mostly complex sentences
Cohesion
Throughout the text, are relations between the ideas explicitly marked through reference, conjunctions and connectors or are such relations not explicit? Whether explicit or not explicit (Weir et al., 2009, p.137)
√ adopted from Green et al. (2010)
Grammatical complexity
G1: average number of words/sentences G2: average number of sentences/paragraph
G3: the proportion of words included in noun phrases
G4: number of modifiers per noun phrase (concerns the occurrence of complex noun phrases (these being a recognized feature of academic text)
G5: the mean number of words before the main verb in sentences (structurally opaque texts tending to have proportionately more higher order syntactic constituents and great number of words before the main verb)
115
G6: logical operators incidence score: include and or, negotiations and a number of conditionals > density means difficult >
hypothesizing and linking ideas> a predictor of text adaptation p.198 Note: In Coh-Metrix paragraph length (number of sentences per paragraph index 4 G2) is measured using an algorithm based on natural language processor from open source library (Grok) where a paragraph is delimited by the number of hard return symbols counted by a text word processor. Index 6 G1: The average number of words per sentence is based on part of speech counted by the Charniak parser. (from Coh-Metrix guidebook) 6 Lexical resources √ adopted from Green et al. (2010)
Word length Lexical density Frequency levels
V1: average number of characters
per word), this being a crude
indicator of reading difficulty V3 lexical density (number of
content words as a proportion of the number of grammatical words) and word frequency levels
V4, V5 and V6 being the
percentage of words occurring among the most frequent and the second and third most frequent
116
1000 words in the British National Corpus (BNC) V7 represents the percentage
falling outside the 15,000 word frequency level (likely to be technical words or proper nouns) V8 the percentage of words in a text
also appearing on the AWL (sub- technical vocabulary)
measured V2 Standardized type-
token ratio (TTR – the ratio of
types or different words to tokens: the total number of words occurring in the text) V9: Average number of higher-level constituents per word in the text >>based on semantic
hierarchy>>words with more higher- level constituents are more specific – academic texts have more specific terms
Vocabulary
Wordsmith used to calculate TTRs based on 250-words sections of text. (TTR- Standardized type – token ration) the ratio of types or different words to tokens: the total number of words occurring in the text)=
percentage / The higher the TTR, the more demanding the passage is likely to be as TTR affected by text length, it is generally recommended
117
that standard length to be sued (250)
7 Nature of information
√ Text abstractness (Weir, 2005, p.138) Is the text concrete or abstract? Range
Whether concrete or abstract (1-5) To measure the abstractness of verbs.
Note: (A4) their hypernym values are
taken into account. The hypernymity of a verb is measured in terms of a
conceptual hierarchy in which the number of levels of superordinate levels above and the number of subordinate levels below the verb
being measured are taken into account. This is an indicator of the
level of concreteness of abstractness of the verb is measured (McNamara et al., 2012;
McNamara et al., 2005). 8 Content knowledge
√ Is the topic of the text of general
interest or does it require subject specific knowledge on the part of the reader? Range from general to specific (1-5) Cultural background:
Is the topic of the text culture-neutral or is it loaded with specific cultural content? Range from cultural neutral to cultural specific (1-5).
Language background:
The text is significantly easier to understand for readers from a specific first language background. Range from strongly agree to strongly disagree (1-5). Religion knowledge:
Is the topic of the text religion-neutral or
118 is it loaded with specific religious
content? Range from religion neutral to religion specific (1-5).
(Weir et al., 2009, p.138)
Task setting
9 Response method
√ The test response method format is likely to affect the test performance? Range from strongly agree to strongly disagree (1-5).
The test tasks provide a variety of response methods Range from strongly agree to strongly disagree (1-5) (Weir, 2005, p.63)
10 Weighting
√ In general, the weighting for different test components are… Range from justified to not justified (1-5). (Weir, 2005, p.64)
11 Knowledge of criteria
√ The criteria to be used in the marking of the test for the candidates and the markers are…Range from explicit to not explicit (1-5).
(Weir, 2005, p.63)
12 Order of items
The items and tasks in the test are presented in a justifiable order. Range from justifiable to not justifiable (1-5). √ (Weir, 2005, p.65)
119 13 Channel of presentation
√ The channel for the target situation requirements of the students being tested is…Range from appropriate to not appropriate (1-5).
(Weir, 2005, p.73)
14 Text length
√ The text length for the target situation requirements of the students being tested is…Range from appropriate to not appropriate (1-5).
(Weir, 2005, p.74)
15 Time constraints
√ The test time of 40 minutes for the test (e.g. preparation and completion)
is...Range from appropriate to not appropriate (1-5).
(Weir, 2005, p.68)
Variables adopted from (Weir, 2005; Weir et al., 2009; Green et al., 2010)
To facilitate judges in making their evaluation, a checklist was drawn up which they could use in making their evaluations (see Appendix 4 for first draft checklist).
The guidelines for constructing this checklist and framing the questions were similar to the ones used for the students’ questionnaire in Section (3.4.1) above which followed (Oppenheim, 1992; Gillham, 2000; Brown, 2001; Nardi, 2003; Procter, 2008; Simmonse, 2008 and Wisker, 2008) and in particular (Bachman , 2004; Weir, 2005; Weir et al., 2009). To assist the judges in their task, examples and definitions of the particular features being assessed were included following the advice in Bachman (2004).
120
3.5.2 Sampling
In this section, sampling strategies for text analysis and expert judges are presented and discussed. A selection of 29 texts was chosen from first year texts books used in Ibra College of Technology and in the Higher College of Technology in Muscat in Oman. The academic texts were selected to be representative of first year programs across all faculties (see Appendix 4). Each of the 29 text extracts was evaluated by three different judges resulting in an overall sample size of 87. The extracts were approximately 500 words in length to correspond with text length of the test passages. The selected texts were taken from different departments as presented in Table (3.4):
Table 3.4 FYA Sample texts
Department/specialization Number of courses for first year
Engineering 8
Business Studies 8 Information Technology 12
TOTAL
29 courses with the addition of a shared course (English Technical writing 1 & 2)
Also, a selection of recent reading test tasks was prepared based on the Level Exit Exam (LEE). Test passages were selected from previous test papers from 2012 to the present. In 2012 the format of the test changed substantially rendering it pointless to include tests earlier than this date. This resulted in a sample size of 5 as shown in Table 3.5:
121
Table 3.5 LEE Sample tests used in this research
Test passages Academic year
Semester (1, 2, 3)
Text Length (approximate word count) 1 2012-2013 1 544 2 2012-2013 2 503 3 2012-2013 3 584 4 2013-2014 1 558 5 2013-2014 2 518
The tests were internally created by the teaching staff but following general guidelines governing the design of test tasks followed by all the Colleges of Technology. The reading texts contained approximately 500 words each and comprehension was tested by approximately 25 questions. It is these texts that were loaded into the analytical software.
Identifying and selecting expert judges was based on their qualifications and experience in the general field of L2 teaching and learning, linguistics and assessment, from researchers, curriculum developers, language teachers, and language testers.As a minimum, a qualification of Masters or equivalent was set. In addition, a pedagogical qualification (e.g. PGCE or TESOL) was desirable. A minimum of five years of teaching or lecturing in adult education was also required. A total of 30 judges participated in this part of the research. In the selected Colleges of Technology, the Heads of Departments for each of the three faculties (Engineering, IT, Business Studies) were requested to supply sample passages from the standard textbooks representing each module of the first year programme. The function of the judges was to rate the various contextual features that were not directly amenable to scalar measurement using a Likert scale following a prepared checklist as discussed in the subsection above. Texts from test tasks of the foundation program (LEE) were matched with texts taken from first year academic (FYA) textbooks. The mean response rates from the test tasks and the academic texts were then compared and a statistical test for equality of means was carried out.
122
3.5.3 Limitations
In this subsection, some limitations of the methods are discussed.
The sample size of the test tasks being used was relatively small. However, it has been explained that, since 2012, the format of the test has been completely altered and that it would have been pointless to include test samples predating the 2012 changes. Accordingly, there were only 5 available tests since the changes of 2012.
The sample size of 87 academic texts may also appear as a limitation. It could be argued that the sample could have been larger; however, this would have placed a great burden on the judges who were giving their services and their time free of charge. 87 texts would amount to reading and evaluating texts of 43,500 total word-count. In view of this, the sample size seemed to be appropriate and adequate to the task.
The variables used to measure some of the features may seem
questionable, for example, the number of characters per word as a measure of word complexity. However, the various instruments have been empirically validated and found to be reliable.
Another limitation is that the judgements of experts could be inconsistent due to the degree of subjectivity involved (Bachman, 2004). However, it was likely that highly experienced professional judges would collectively reach decisions that could be deemed to be trustworthy and dependable, as each text/passage was independently examined by three different judges. Having a relatively large sample of judges, 30 in this study, increased the likelihood that an extreme view taken by one judge would be balanced by the ratings of other judges. This process is referred to by Derrida (2004) as “differance” which, in French, is a play on the word which can mean both opposition as well as deferring: “difference as the process of differing-deferral requires us to renounce a logocentric (or epistemic) conception of truth and admit the possibility of that which might always surpass the limits of our knowledge at any given time…there is no extra-linguistical reality by which our various
123
statements, hypotheses, predictions, etc. might ultimately be assessed in point of their truth or falsehood” (Norris, cited in Derrida, 2004, p. xxxv). Thus, differance explains how consensus can be reached among people of differing viewpoints. Inter-subjectivity overcomes the effect of a single extreme subjective view. So the larger number of judges in this study increases the reliability of the findings. Furthermore, the judges’ checklist contained clear instructions and brief definitions of the features being assessed and this should have eliminated any possible misconception thereby minimising the effects of subjectivity (Bachman, 2004). Additionally, training was provided for the judges, which consisted of a briefing of the general aims and objectives of the study and the methods being used to address the research questions. Such a briefing provided an introduction for the judges to their particular tasks and gave them an overview of the roles that they would play in the overall study. It is, therefore, concluded that the judges were appropriately orientated towards their tasks of evaluation with this overview in mind.