Comparison between the real-life and Test

the automated indices, this section examines the extent to which the input texts set in the two reading-into-writing test tasks were similar to the real-life texts. The 17 indices obtained from Test Task A and Test Task B input texts were compared with those obtained from the real-life input texts. The differences between the real-life and Test Task A input texts were analysed by the Mann- Whitney test, which is a non-parametric, between-subjects test (The results will be discussed below). The differences between the real-life and Test Task B input texts were compared descriptively only, due to a small sample size of the Test Task B input texts (The results will be discussed in Section 4.3.2.6. Overall, the difficulty level of sampled Test Task A input texts was comparable to the level of the real-life input texts (See Table 4.6). The differences in the 14 out of the 17 indices obtained between the two conditions were non-significant. In the remaining three indices, with the exception of low frequency words (Offlist), the differences obtained were slight.

160

Table 4.6 Comparison of the difficulty level between real-life and Test Text A input texts

Real-life tasks (60 extracts from 20 texts) Mean Test Task A (20 texts) Mean Mann- Whitney U Wilcoxon W z Asymp. Sig. (2- tailed) Lexical features High frequency words (K1) 77.20 76.54 556.000 766.000 -.489 0.63 High frequency words (K1+K2) 87.76 87.69 600.000 810.000 .000 1.00 Academic words 10.37 8.84 437.000 647.000 -1.811 0.07 Low frequency words (Offlist) 2.41 10.41 11.000 1841.000 -6.545 0.00 Log frequent content

words 2.10 2.05 508.000 718.000 -1.022 0.31 Average syllables per word 1.70 1.72 591.000 2421.000 -.100 0.92 Type-token ratio (content words) 0.69 0.72 448.000 2278.000 -1.689 0.09 Syntactic features

Average words per sentence 21.38 20.49 514.000 724.000 -.956 0.34 Sentence syntax similarity 0.08 0.09 401.500 2231.500 -2.206 0.03 Mean number of

modifiers per noun- phrase

1.03 0.91 336.000 546.000 -2.933 0.00

Mean number of words before the main verb 5.50 5.76 493.500 2323.500 -1.183 0.24 Logical operator incidence 45.12 43.76 560.500 770.500 -.439 0.66 Cohesion Adjacent overlap argument 0.55 0.60 520.500 2350.500 -.884 0.38 Adjacent overlap stem 0.58 0.65 500.500 2330.500 -1.106 0.27 Adjacent overlap content word 0.10 0.09 490.500 700.500 -1.217 0.22 Proportion of adjacent anaphor references 0.25 0.28 457.000 2287.000 -1.589 0.11 Adjacent semantic similarity (LSA) 0.23 0.25 488.000 2318.000 -1.245 0.21

161

With regards to lexical complexity, Test Task A input texts contained similar proportions of high frequency words (K1 and K1+K2) as the real-life input texts. However, they contained slightly fewer academic words (though the difference was not significant) (See Table 4.6). Interestingly Test Task A contained significantly (z=-6.545, p<0.01) more low frequency words than the real-life input texts and the mean difference was as large as 8% (real-life: 2.41%, Test Task A: 10.41%). The low frequency words on the Test Task A input texts were mainly proper names of places and organisations/companies. For the remaining lexical indices concerning the content words, there was not much difference in terms of the frequency of content words and the average syllables per word. Test Task A input texts had a slightly higher type-token ratio but the difference was not significant.

Regarding the syntactic complexity, there was no significant difference in three syntactic indices (average words per sentence, mean number of words before the main verb and logical operator incidence) between the Test Task A and real-life input texts (See Table 4.6). However, Test Task A input texts had a significantly higher sentence syntax similarity index than the real-life input texts, and contained significantly fewer modifiers per noun-phrase than the real-life input texts. This suggests that the Test Task A texts might be less complex to process than the real-life input texts in terms of syntactic complexity, although the actual mean differences were very small.

The degree of text cohesion in Test Task A and the real-life input texts was similar. There was no significant difference in all cohesion indices obtained between Test Task A and the real-life input texts (See Table 4.6).

4.3.2.6 Comparison between the real-life and Test Task B input texts At the time of the study, only one set of operationalised Test Task B was available. Due to a limited sample size, only descriptive statistics of the textual indices of the two Test Task B input texts are presented here (See Table 4.7). The indices obtained from the real-life input texts are provided in the table for a descriptive comparison. For the 17 textual indices, larger descriptive discrepancies were found in 6 indices (2 lexical, 1 syntactic and 3 coherence indices) between the Test Task B input texts and the real-life input texts.

162

Table 4.7 Descriptive comparison of the difficulty level between real-life source texts and Test Task B input texts Real-life tasks (60 extracts from 20 texts) Mean Test Task B (2 texts) Mean

Descriptive comparison of the difficulty level between real-life and Test Task B input texts

Lexical features High frequency words (K1)

77.20 81.9 The Test Task B input texts had slightly more first 1000 frequency words in proportion than the real-life input texts.

High frequency words (K1+K2)

87.76 91.99 The Test Task B input texts had slightly more first 2000 frequency words in proportion than the real-life input texts.

Academic words 10.37 14.46 The Test Task B input texts had more academic words in proportion than the real-life input texts.

Low frequency words (Offlist)

2.41 6.63 The Test Task B input texts had more low frequency words in proportion in proportion than the real-life input texts.

Log frequent content words

2.10 2.11 The Test Task B input texts had almost the same frequency level of the content words as the real-life input texts.

Average syllables per word

1.70 1.79 The Test Task B input texts had almost the same number of syllables per word as the real-life input texts.

Type-token ratio (content words)

0.69 0.77 The Test Task B input texts had a slightly higher type-token ratio than the real-life input texts.

Syntactic features Average words per sentence

21.38 20.32 The Test Task B input texts had a slightly shorter average sentence length than the real- life input texts.

Sentence syntax similarity

0.08 0.10 The sentence structures used in the Test Task B input texts were slightly more similar to each other than those in the real-life input texts.

163

Mean number of modifiers per noun- phrase

1.03 1.41 The Test Task B input texts had slightly more modifiers per noun-phrase than the real- life input texts.

Mean number of words before the main verb

5.50 4.75 The Test Task B input texts had fewer words before the main verb per verb-phrase than the real-life input texts.

Logical operator incidence

45.12 28.16 The Test Task B input texts input texts had a much lower proportion of connectives than the real-life input texts.

Cohesion

Adjacent overlap argument

0.55 0.73 The Test Task B input texts had a higher percentage of the adjacent sentences that shared one or more arguments (i.e. nouns, pronouns, noun-phrases) than the real-life input texts.

Adjacent overlap stem 0.58 0.73 The Test Task B input texts had a higher percentage of the adjacent sentences that shared one or more word stems as the real-life input texts.

Adjacent overlap content word

0.10 0.80 The Test Task B input texts had a higher percentage of the adjacent sentences that shared one or more content words as the real-life input texts.

Proportion of adjacent anaphor references

0.25 0.18 The Test Task B input texts had a lower proportion of adjacent anaphor references than the real-life input texts.

Adjacent semantic similarity (LSA)

0.23 0.28 The Test Task B input texts were slightly more conceptually similar across the text than the real-life input texts.

164

Regarding the lexical complexity, Test Task B input texts seemed to be slightly easier than the real-life input texts due to a higher proportion of the first 1000 and 2000 frequency words. However, larger discrepancies were obtained in other indices (academic words and low frequency words), which apparently suggested that Text Task B input texts were actually more difficult than the real-life input. Text Task B input texts contained more academic words (14.46% vs 10.37%) and low frequency words (6.63% vs 2.41%) than the real-life input texts. In addition, Text Task B input texts had a slightly higher type-token ratio of the content words than the real-life input texts. There was not much difference between the Test Task B input texts and the real-life input texts in terms of the frequency level of the content words and the number of syllables per word. Therefore, while containing a slightly higher proportion of high frequency words, Test Task B input texts could be more difficult to process than the real-life input texts, due to a higher proportion of academic words and low frequency words and a higher proportion of unique content words (type-token ratio).

Regarding the syntactic features, Test Task B had a much lower proportion of connectives (logical operator incidence score) and a slightly more modifiers per noun-phrase than the real-life input texts. This could indicate a higher demand to process the noun-phrases and to sort out the logical connections between ideas in the Test Task B input texts than in the real-life input texts. On the other hand, Test Task B input texts contained a lower average number of words per sentence, a higher sentence syntax similarity score, and a lower number of words before the main verbs in verb phrases, but the actual differences were very small. Therefore, the results seemed to suggest that Test Task B input texts were more syntactically challenging than the real-life input texts due to a noticeable lower proportion of connective in the texts.

Regarding the degree of text cohesion, the Test Task B input texts had a lower proportion of adjacent anaphor references than the real-life input texts. This indicates a more demanding process of resolving the anaphor references in the Test Task B input texts and the real-life input texts. However, the other four text cohesion indices seemed to suggest that Test Task B input texts had a better cohesion than the real-life input texts, and hence were less challenging.

165

Test Task B input texts had higher proportions of adjacent sentences sharing one or more arguments (i.e. nouns, pronouns, noun-phrases), word stems and content words than the real-life input texts. This means it would be easier to process the main themes in Test Task B input texts than in the real-life input texts. Test Task B input texts also had a higher adjacent semantic similarity score than the real-life input texts, which indicates that the adjacent sentences in the Test Task B input texts were more conceptually similar than those in the real-life input texts.

In short, when compared descriptively to real-life input texts, Test Task B input texts were more demanding in terms of lexical complexity (more academic words and more low frequency words) and syntactic complexity (less proportion of connectives), but less demanding in terms of text cohesion (higher proportions of shared arguments, words stems and content words). Due to the small number of testlets available for Test Task B, it was not possible to do any inferential statistics on the textual indices between Test Task B and real-life input texts. The descriptive results reported here are only suggestive.

4.4 Summary

Chapter Four aims to address RQ1: What are the most appropriate contextual parameters of the EAP writing tasks? To what extent do the reading-into- writing tests resemble these contextual features in the testing conditions? The chapter has reported the results of the salient contextual features of the two selected real-life writing tasks to shed light on the most appropriate contextual parameters for EAP writing tests. The chapter has also reported the contextual features of two types of reading-into-writing test tasks (essay with multiple verbal inputs and essay with multiple verbal and non-verbal inputs), and discussed the extent to which the contextual features of the reading-into- writing test tasks resembled the target contextual features of the real-life academic writing tasks. This section provides a summary of the findings regarding the contextual validity of the two reading-into-writing test tasks.

166