Pre and Post Instruction Written Protocols

The written protocols used in this strategy instruction, that is the pre and

post instruction essays (Appendix A) were administered to both the

treatment and control groups at the start of the semester, that is the 1st week for the pretest and in the 12th week for the posttest. As the writing instruction employed the process approach in which students were taught

to reflect and revise in a multistage recursive composing activity, the pre-

and post instruction written assignments writing were not timed nor

confined to a particular venue as is the norm for most written tests.

However, students were given a duration of one week to submit their type-

written assignment where the stipulated length is 350 words, similar to the

length specified in their final examination paper for the writing section.

This resulted in a more authentic mode for assessing their writing ability

that also ensured validity in the testing. As the course involved is academic

writing, students were required to refer to other sources of reference as

well as use computer based technology such as word processing.

The topic given for the task was neutral in terms of culture, gender and

prior experience, and was of interest to the students. This was ascertained

test. The probes for the pre and post tests were then finalised upon

consultation with two colleagues who also taught the course. Although the

probes were closely related to each other in similarity, the topic of the

post-test was not the same as that of pre-test. The decision not to provide

the same topic for the pre and post tests was made based on the assumption

that the element of familiarity may skew results.

The International English Language Testing System or IELTS scale

(Appendix B) was used in grading the students’ pre and post instruction

assignments. This scale was considered appropriate as it has been widely

used for assessing the writing proficiency of those wanting to enrol in

institutions of higher learning. The band scale for writing ranges from 0

("Did not attempt the test") to 9 ("Expert User"), and a profile score for

this skill is calculated by taking the mean of the scores awarded for the

four criteria presented in the IELTS writing scale, that is task response,

coherence and cohesion, lexical resource, and grammatical range and

accuracy. Each of the four criteria is awarded a score out of 9 as the

criteria are equally weighted (Appendix B). These scores are added and

then divided by 4. For example, in the pretest, a student may have scored

the following:

Task response 5

Coherence and cohesion 3

Lexical resource 3

Grammatical range and accuracy 2

In this study, the scores were not rounded to the next whole band or half

band as is the norm for IELTS band scores. Instead, the original scores

were retained to reflect the students’ performance more accurately.

The benefits of the analytical mode of grading such as that of IELTS are

that it discourages norm-referencing and allows greater discrimination

across a broad range of assessment bands (nine bands in all) and exercises

a greater control over what influences the impressions of examiners, thus

preventing an inclination to evaluate impressionistically. It was felt that

adopting an analytical approach in this study would enhance the reliability

of the marking since it would increase the number of observations, and

discourage impressionistic marking. Furthermore, an analytical approach

also allowed the students as well as the instructor to identify areas of

strengths and weaknesses which could then be used in evaluating and

improving students’ writing.

The IELTS Academic Writing scale was chosen as the scoring guide for

the written protocol in this study as it has been well-researched and used

extensively to assess an individual’s ability to write in clear, formal

English, as this is the requirement that is generally expected in an

academic setting. Furthermore, using a standardised internationally

recognised grading scale such as the IELTS will ensure greater external

reliability and validity through replication of the SRSD studies.

By adopting the IELTS writing scale, this study advocates a different

improving students’ academic writing skills. The evaluation was based on

the following criteria, that is task response which deals with how

accurately the task is addressed, coherence and cohesion which involves

how organized one’s writing is, lexical resource which refers to the range

of one’s vocabulary and lastly, grammatical range and accuracy which

examines the correctness of one’s grammar. A score was assigned to each

of these criteria based on the nine bands in the IELTS scale. This provided

valuable insights into the development of students’ writing as the focus

was on assessing students’ response to a given writing task by examining

whether a clear position was presented, the main ideas were well-

developed, with relevant and fully extended supporting details and

appropriate conclusion. Also through the coherence and cohesion

component, the assessment focused on logical sequencing of ideas and

appropriate paragraphing as well as effective use of cohesive devices.

Other aspects of writing that were included in the IELTS writing

assessment were flexibility and appropriateness in the use of vocabulary,

accuracy of spelling and word formation, and good control of structures,

grammar and punctuation.

SRSD studies on writing in the past had mainly looked at features such as

the overall length, number of essential elements, number of transitions,

number of descriptive words and overall quality of the stories, narratives

or persuasive essays (Graham & Harris, 2003). This change in assessment

was deemed necessary as academic writing in all its complexity should not

There were two independent ESL lecturers who rated the pre and post-test

scripts. Both of them have vast experience in the teaching of English as a

second language, especially the EAP course. They volunteered to co-assess

the pre and the post-instruction essays after having gone through a series

of moderation sessions with the instructor using the IELTS scale. The two

evaluators then assessed all the essays and all their scores were checked if

there was any great disparity in assessment. An allowance was made for

one to three marks’ difference as the scores were to be averaged. However,

if the disparity was greater than this, the essay scores were discussed and

remarked or resolved by reaching a consensus. Of the 132 scripts marked

by each rater, about seven had to be remarked for this reason. On the

whole, the Pearson correlation between the scores given by Rater A and

Rater B was 0.763. The correlation is significant with p-value less than

0.05. The reliability or consistency of scores between both the raters is

thus considered strong with a high correlation value.