Automated textual analysis tools

In addition to expert judgement, automated textual analyses were performed to analyse a range of textual features of the input texts in this study. Automated textual analysis has been regarded as a more systematic and efficient way to assess textual features than the more traditional expert judgement method, especially when a large number of texts are involved. Many researchers have used automated textual analytic tools to evaluate the features of different types of texts such as L1 students' scripts (e.g. Crossley & McNamara, 2010), L2 students' scripts (e.g. Crossley & McNamara, 2012), reading materials (e.g. Crossley, Louwerse, McCarthy, & McNamara, 2007; Green, 2012), undergraduate reading texts (e.g. Green et al., 2010), reading texts in language tests (e.g. Green et al., 2012; Wu, 2012), and L2 test takers' scripts (e.g. Weir, 2012)

94

In this study, two automated textual analysis tools were chosen - CohMetrix version 2.1 (Graesser, McNamara, Louwerse, & Cai, 2004) and VocabProfile version 3 (Cobb, 2003). CohMetrix was used in all the above-mentioned textual analysis studies. It is one of the most popular textual analysis tools in the literature. It is freely assessable on the Internet and it produces a very comprehensive list of about 60 textual indices. More importantly, CohMetrix was designed to explore attributes of cognitive language use. Graesser, McNamara & Kulikowich (2011) argued that CohMetrix's automated indices measure 'deep-level factors of textual coherence and processing' (223). VocabProfile (Cobb, 2003) is another popular textual analysis tool which provides a profile of texts in terms of different vocabulary frequency bands based on BNC (The British National Corpus, 2007) (e.g. the most frequent 1000 words) and different types of vocabulary (e.g. academic words based on Coxhead, 2000). The tool has been used to assess the difficulty level of reading texts in many studies.

Both tools have been used in the testing literature. For instance, Green et al (2010) compared IELTS reading texts and undergraduate texts at British universities, Green (2012b) investigated reading texts targeted at different levels of the Common European Framework of Reference for Languages (Council of Europe, 2001), Green et al (2012) investigated the features of reading texts in CAE, and Wu (2012) compared Cambridge Main Suite and GEPT Taiwan examinations at the B1 and B2 levels. Weir (2012) investigated features of the test takers' scripts of the TEAP test in Japan.

While CohMetrix and VocabProfile allow researchers to automate a large number of textual indices in an objective and reliable way, the results have to be interpreted with caution. Researchers have argued that not all indices produced are equally useful or interpretable. Green et al (2012) criticised the fact some of the indices seem to overlap and Green (2012b) attempted to identify those indices which are helpful to distinguish texts between adjacent CEFR levels.

It is seemingly important for individual researchers to establish which of the indices are helpful in their context of study. Green et al. (2012) showed that 17

95

CohMetrix and 2 VocabProfile indices meaningfully exhibited significant differences in the reading texts across three levels of Cambridge examinations: FCE (B2), CAE (C1) and CPE (C2). Based on 25 CohMetrix indices and 6 VocabProfile indices, Wu (2012) compared the features of reading texts between the GEPT and Cambridge examinations at B1 and B2 levels. Weir (2012) found that 12 CohMetrix indices were useful in establishing criterial differences in the L2 test takers' scripts rated at the A2 and B1 levels. Green et al. (2010) compared the features of undergraduate texts and IELTS reading texts by 19 CohMetrix and 5 VocabProfile indices.

Drawing upon previous studies, especially those looking at reading texts (e.g. Green et al., 2010; Green et al., 2012 and Wu, 2012), the usefulness of the all CohMetrix and VocabProfile indices were examined by the researcher in a pilot analysis. 30% of the real-life input texts were analysed in the pilot analysis. Based on the results of the pilot analysis, 13 CohMetrix and 4 VocabProfile indices were selected to analyse the features of the input texts and draw comparisons between the real-life and reading-into-writing test tasks (See Table 3.5 for a glossary of the selected indices). The selection of the indices in this study was similar to the previous studies. However, it was considered more appropriate to categorise the selected indices in terms of lexical complexity, syntactic complexity and degree of cohesion for the context of this study (i.e. reading-into-writing tasks for academic purposes), rather than categories such as vocabulary, grammar, readability, cohesion and text abstractness used in Green et al's (2010) study. The list of the deleted indices and reasons for deletion are presented in Table 3.6.

96

Table 3.5 Selected automated textual indices

Contextual index Definition

(Extracted from the official documents of the two tools) Automated analysis tool Lexical High frequency words (K1)

The ratio of words which appear in the first most frequent 1000 BNC (2001) wordlist to the total number of words per text

VocabProfile

High frequency words (K2)

The ratio of words which appear in the second most frequent 1000 BNC (2001) wordlist to the total number of words per text

VocabProfile

Academic words The ratio of words which appear in the Academic Wordlist (Coxhead, 1998) to the total number of words per text

VocabProfile

Low frequency words (Offlist)

The ratio of words that do not appear in either the most frequent 15000 BNC wordlist to the total number of words per text

VocabProfile

Log frequent content words

The log frequency of all content words in the text

Cohm 46 Average syllables

per word

The mean number of syllables per content word, a ratio measure

Cohm 38 Type-token ratio

(content words)

The number of unique words divided by the number of tokens of these words

Cohm 44 Syntactic

Average words per sentence

The mean number of words per sentence Cohm 37 Sentence syntax

similarity

The proportion of intersection syntactic tress nodes between all sentences

Cohm 56 Mean number of

modifiers per noun-phrase

The mean number of modifiers per noun- phrase

Cohm 41

Mean number of words before the main verb

The mean number of words before the main verb of the main clause in sentences

Cohm 43

Logical operator incidence

The incidence of logical operations (i.e. connectives), such as and, or, not, if, then, etc

Cohm 26 Cohesion

Adjacent overlap argument

The proportion of adjacent sentences that share one or more arguments (i.e. noun, pronoun, noun-phrase) or has a similar morphological stem as a noun

Cohm 16

Adjacent overlap stem

The proportion of adjacent sentences that share one or more word stems

Cohm 17 Adjacent overlap

content word

The proportion of content words in adjacent sentences that share common content words

Cohm 58 Proportion of

adjacent anaphor references

The proportion of anaphor references between adjacent sentences

Cohm 18

Adjacent semantic similarity (LSA)

The measure of conceptual similarity between adjacent sentences

97

Table 3.6 List of deleted CohMetrix indices and reasons for deletion Reasons CohMetrix indices deleted

1 They overlapped with other indices – they showed very similar or no difference in results of other indices. For instance, 40 (Flesch-

Kincaid grade level) and 39 (Flesch reading ease) are already covered by 38 (Average syllables per word and 37 (Average words per sentence). Celex measures are covered by word frequency measures.

 Cohm 24 Number of conditional expressional, incidence score

 Cohm 25 Number of negations, incidence score

 Cohm 34 Number of sentences

 Cohm 39 Flesch Reading Ease

 Cohm 40 Flesch-Kincaid Grade Level4

 Cohm 55 Sentence syntax similarity adjacent

 Cohm 57 All within paragraphs

2 They were difficult to interpret in terms of text complexity. For instance, 23 (ratio of pronouns to noun phrase) is affected by text type.

 Cohm 12 Incidence of negative additive connections

 Cohm 13 Incidence of negative temporal connections

 Cohm 14 Incidence of negative causal connections

 Cohm 23 Ratio of pronouns to noun phrase

 Cohm 30 Personal pronoun incidence score

 Cohm 36 Average sentences per paragraph

3 They were not applicable to the data in this study due to the sampling procedures of the real-life texts.

 Cohm 33 Number of paragraphs

 Cohm 35 Number of words

4 They were not useful or effective in determining the complexity of a text

because

a) the index produced a score which is difficult to interpret;

b) insufficient explanation was provided in the

CohMetrix menu; and/or c) results obtained

contradict human judgment.

 Cohm 7 Incidence of causal verbs, links and particles

 Cohm 8 Ratio of causal particles to causal verbs

 Cohm 9 Incidence of positive additive connectives

 Cohm 10 Incidence of positive temporal connectives

 Cohm 11 Incidence of positive causal connectives

 Cohm 15 Incidence of all connectives

 Cohm 19 Argument Overlap, all distances

 Cohm 20 Stem overlap all distances unweighted

98

 Cohm 21 Anaphor reference, all distances

 Cohm 22 Noun phrase incidence score (per thousand words)

 Cohm 28 LSA all sentences combination mean

 Cohm 29 LSA paragraph to paragraph mean

 Cohm 31 Mean hyponymy values of nouns

 Cohm 32 Mean hyponym value of verbs

 Cohm 42 Higher level constituents per word

 Cohm 45 Celex, raw, mean for content words

 Cohm 47 Celex raw minimum in sentence for content words

 Cohm 48 Celex, logarithm, minimum in sentence for content words (0-6)

 Cohm 49 Concreteness, mean for content words

 Cohm 51 Incidence of negative logical connectives

 Cohm 52 Ratio of intentional particles to intentional content

 Cohm 53 Incidence of intentional actions, events and particles

 Cohm 54 Mean of tense and aspect repetition scores

 Cohm 59 Mean of location and motion ratio scores

99

In document Establishing the validity of reading-into-writing test tasks for the UK academic context (Page 111-117)