Difficulty level of the input texts

Generally speaking, the difficulty level between the real-life input texts and undergraduate input texts (Green et al, 2010) was similar in terms of most

Input text features Real-life essay task Real-life report task

Test Task A Test Task B 8. Input format verbal (80%)

verbal and non- verbal (20%)

verbal (30%) verbal and non-

verbal (70%) 2 passages 2 passages with non- verbal information 9. Verbal input genre book Chapter (60%) report (10%) journal article (10%) news article (10%) case study (10%) news article (50%) journal article (30%) review (10%) report (10%) essay (100%) report (50%) news article (50%) 10. Non-verbal input pictures (50%) diagrams (50%) graphs (71%) tables (29%) Nil diagrams (100%) 11. Discourse mode argument/ evaluation (60%) exposition (40%) exposition (70%) argument/ evaluation (30%) argument/ evaluation (100%) exposition (100%) 12. Concreteness of ideas 3.3 2.9 4.1 4 13. Explicitness of textual organisation 3.5 3.15 3.9 4 14. Cultural specificity 3.3 3.4 4.2 3.6

169

lexical, syntactic and cohesion automated indices investigated in the study. The only discrepancies were that

(1) the real-life input texts contained more high frequency words (the first 1000 and the first 2000) but less low frequency words than the undergraduate course texts.

(2) the real-life input texts contained slightly higher number of modifiers per noun phrase and words before the main verbs of the main clauses than the undergraduate texts, and had a slightly lower logical operator incidence score than the undergraduate texts.

In other words, the real-life input texts were apparently easier than the undergraduate course book texts in terms of the proportion of word frequency bands. However, it would be slightly more demanding to work out the meaning and syntactic structure embedded in the noun phrases and main clauses in the real-life-input texts, as well as more demanding to build the textual representation of the real-life-input texts than the undergraduate texts. Regarding the comparison of the difficulty level between the real-life input texts and the test task input texts, the results again showed more similarities than discrepancies. The major discrepancies are summarised below:

(1) Out of 17 indices, only 3 indices obtained significant differences between Test Task A and real-life input texts. Test Task A input texts had a significantly greater density of low frequency words, mostly proper nouns, than the real-life input texts. This would probably increase the difficulty of processing the texts if the test takers were not familiar with these proper nouns. However, Test Task A input texts had a significantly higher sentence syntax similarity index than the real-life input texts, and contained significantly fewer modifiers per noun-phrase than the real-life input texts. This suggests that it would be less demanding to build the textual representation of the Test Task A input texts than that of the real-life-input texts. The degree of text cohesion in Test Task A and the real-life input texts was similar. There was no significant difference in the cohesion indices obtained between Test Task A and the real- life input texts.

170

(2) Due to the small number of testlets available for Test Task B, only descriptive statistics analysis was performed. Test Task B input texts contained more high frequency words (the first 1000 and 2000) than the real- life input texts. However, Test Task B input texts had a higher proportion of academic words and low frequency words and a higher type-token ratio of all content words than the real-life input texts. This indicates that the lexical complexity of Test Task B input texts was seemingly more demanding than the real-life input texts. Regarding the syntactic complexity, Test Task B had a much lower logical operator incidence score and a higher mean number of modifiers per noun-phrase than the real-life input texts. This indicates a higher demand on reader to process the noun-phrases and to sort out the logical connections between ideas in Test Task B input texts. Lastly, Test Task B input texts had higher proportions of adjacent sentences sharing one or more arguments, word stems and content words, and a higher adjacent semantic similarity score than the real-life input texts. All these indicated that Test Task B input texts were more cohesive than the real-life input texts.

As summarised above, the results of this study showed that the linguistic complexity of real-life input texts and those used in the two test tasks was largely comparable with only a few discrepancies. Nevertheless, it is worth investigating the issue of whether the apparent major disjunctions, such as proportion of academic words, proportion of low frequency words, sentence syntax similarity index, and modifiers per noun-phrase, were a result of test writers modifying or developing input reading texts for test design purposes. For example, in addition to requirements such as style and genre, test writers are often required to include a certain number of 'idea units' into an input text with a certain number of words. It is interesting to investigate how test writers select and edit real-life texts for test design purposes and the effects of such practice in further studies. Recommendations for test takers on developing appropriate input texts for reading-into-writing test tasks for academic purposes are provided in Chapter Seven.

According to the literature review, the difficulty level of a test task is largely determined by its contextual features (Khalifa & Weir, 2009; Shaw & Weir,

171

2007; Weigle, 2002; Weir & Wu, 2006; Wu, 2012). Many studies were conducted to survey the common writing tasks in the real-life academic context and these studies concluded that most academic writing tasks involved integration of reading materials (e.g. Bridgeman & Carlson, 1983; Carson, 2001; Horowitz, 1986; Johns, 1993; Weir, 1983). However, while revealing the general features of academic writing tasks, very few studies in the literature to date systematically provided detailed information of the contextual features of these academic writing tasks. With the use of expert judgement and automated textual analysis, this study analysed the features of two selected real-life academic writing tasks (the essay task and the report task) and the two reading-into-writing task type in terms of 7 parameters of overall task setting, 7 parameters of the features of input texts and 17 automated textual indices of the linguistic difficulty level of the input texts. In addition, researchers believe that the reading-into-writing task type has good context validity because such integrated task type can arguably reflect the characteristics of real-life academic writing tasks (e.g. Johns, 1993; Read, 1990; Weir et al., 2013). To the knowledge of the researcher, this study was the first study to compare the characteristics of the authentic real-life academic writing tasks and operationalised reading-into-writing test tasks in terms of a range of contextual parameters. The results of this study showed that the two reading-into-writing test tasks largely resembled the contextual parameters of the real-life academic writing tasks.

This chapter has reported and discussed the contextual validity of the two reading-into-writing tests. Chapter Five will shift the attention to the cognitive validity, which concerns to what extent the selected real-life tasks and the two reading-into-writing tasks elicited from the participants in this study the same cognitive processes.

172

5 INVESTIGATING THE COGNITIVE VALIDITY OF READING- INTO-WRITING TESTS TO ASSESS ACADEMIC WRITING ABILITY

5.1 Introduction

Chapter Four presented and discussed the results of the context validity of the reading-into-writing tests to assess academic writing ability. This chapter presents and discusses the results of their cognitive validity which is concerned with the extent to which a test elicits from test takers cognitive processes that correspond to the processes that they have to use in the target language context (Glaser, 1991; Shaw & Weir, 2007). There are two major steps involved in investigating cognitive validity. First we need to define the target cognitive processes to be measured in a writing test by investigating the processes that students employ in a real-life context. Secondly we need to investigate the extent to which these target cognitive processes can be elicited by the test tasks.

This study investigated the cognitive processes employed by over 200 participants in both real-life academic and authentic test conditions through a carefully developed and validated Writing Process Questionnaire (See Section 3.4.1 for the procedures of developing the questionnaire). As explained in Chapters Two and Three, the processes of translation and micro-planning, which also play an important role in the writing production, were not investigated in this study based on the following reasons:

 When compared to other processes such as macro-planning and organising, writers tend to be less aware of the use of translation and micro-planning processes because these processes are usually not

173

taught explicitly in normal classroom settings. Methods such as think- aloud protocols would be more appropriate for the investigation of these processes. Previous studies tended to investigate these processes solely under experimental settings (see Kellogg, 1994 for a review)

 Previous studies have indicated that writing-only and reading-into- writing tasks each elicit a distinct set of processes from writers. For example, the processes of creating textual or intertextual representations are not addressed by writing-only tasks. The processes of translation and micro-planning, on the other hand, might not differ as much as other processes between the independent and integrated test types.

A total of 443 questionnaires were collected from the real-life and test conditions in the study - 70 questionnaires on the real-life essay task, 73 on the real-life report task, 160 on the reading-into-writing Test Task A, and 140 on the reading-into-writing Test Task B (See Section 3.4.2 for the details of data collection).

This chapter begins with the results pertaining to the cognitive processes performed by the participants in real-life conditions. Descriptive statistics of individual questionnaire items from each of the real-life tasks and the comparison of the cognitive processes employed on the two real-life tasks are reported in Section 5.2.1. Results from the exploratory factor analysis (EFA) of the number of distinct cognitive processes involved in five academic writing cognitive phases and the underlying structure of these processes are reported in Section 5.2.2. After defining the EFA-generated underlying structure of the real-life cognitive constructs, further comparisons of the two real-life tasks, to compare the extent to which each cognitive process was elicited, are presented in Section 5.2.3. In the context of language tests, it is important to collect evidence to show if the cognitive parameters can distinguish how more proficient writers and less proficient writers employ these processes. A comparison of the cognitive processes employed by high- achieving and low-achieving participants in the real-life context is presented in Section 5.2.4. Sections 5.2.5 summarises the results of the data relating to real- life academic writing processes.

174

After investigating the target cognitive constructs, Section 5.3 reports the results elicited by the two reading-into-writing test tasks. Section 5.3.1 compares the cognitive processes elicited by Test Task A and real-life tasks (Section 5.3.1.1), and by Test Task B and real-life tasks (Section 5.3.1.2). Section 5.3.2 discusses the comparison of the cognitive processes employed by high and low achieving groups on Test Task A (Section 5.3.2.1) and Test Task B (Section 5.3.2.2). Section 5.3.3 discusses the comparison of the processes, in groups of high-, medium- and low-achievement, elicited by Test Task A and real-life tasks (Section 5.3.3.1), and by Test Task B and real-life tasks (Section 5.3.3.2). A summary is given in Section 5.3.3.3. Section 5.3.4 discusses the underlying structure of the cognitive processes elicited by the test tasks (Section 5.3.4.1 – Section 5.3.4.5). A summary is given in Section 5.3.4.6. Section 5.4 provides a brief synopsis of the whole chapter.

5.2 Investigating the target cognitive constructs in the real-life context