ISSN: 2347-7474
International Journal Advances in Social Science and Humanities
Available online at: www.ijassh.com
RESEARCH ARTICLE
Test-taking Strategies: The Case of TOEFL Reading
Cıngıllıoğlu Salih, Akay Ceylani*
Department of English Language and Literature, Faculty of Education, International Burch University, Bosnia and Herzegovina.
*
Corresponding Author: Email: [email protected]Abstract
This study aims to contribute to the exploration of test-taking strategies readers use in the reading component of the TOEFL. By means of verbal reports and observation, this study explores how the two types of readers respond to multiple choice tests and use test taking strategies. The results yielded that average readers used significantly fewer strategies than did good readers, which contradicts the 1991 study by Anderson that found both types of readers use the same number of strategies. However, the results support Anderson’s claim that good readers can manage their strategies better than average readers.
Keywords: Reading proficiency, Test strategies, TOEFL.
Introduction
The purpose of this paper is to examine the strategies adult second language learners while they are doing the reading part of the TOEFL test. In their attempt to clarify the meaning of strategies for second language reading and to show how they differ from skills, Urquhart and Weir [1] propose the distinction that strategies are reader-oriented and conscious acts, unlike skills, which are text-oriented and automatic. Although other researchers like Grabe and Stoller [2] find this distinction blurry on the grounds that strategies are equally automatic for skilled readers, I’ll keep the former proposal and acknowledge strategies as conscious efforts which can be ‘accessed for deliberate report’ [3, 4].
Strategies that students use when they are dealing with various reading contexts have started to be a center of attention in recent three decades. This new trend is in contrast with the usual ‘focus on the product of reading’ [3]. Reading strategies attract researcher’s attention because ‘they reveal about the way readers manage their interaction with written text and how these strategies are related to text comprehension’ [5]. A common impetus that fed this growing concern over the doubts it sheds on the validity of tests widely administered for reading comprehension. If students can get to correct answers without comprehending the passage as is envisioned by the test constructor, the validity of tests will necessarily suffer.
Applying the same reasoning to academic reading contexts, Cohen and Upton [6] state that ‘the premium needs to be on designing tasks calling for test takers to actually use academic reading skills in responding to items, rather than being able to rely on test-wiseness tricks.’
Thompson [7] cites a study by Block which concludes that students across the world use the same reading strategies and lists sixteen good reader strategies developed by Hosenfeld and improved by Van Perreren and Schouten:
Concentrate on meaning rather than on word identification,
Read the text as though they expect it to make sense,
Use their knowledge of the world,
Take chances making inferences but evaluate their guesses,
Use titles and illustrations to make inferences,
6) Use preceding and following con- text,
Continue if unsuccessful,
Utilize different types of context clues,
Skip unknown words and make contextual guesses,
Identify grammatical category of words,
Recognize cognates,
Analyze unknown words and use glossary as a last resort,
Skip unnecessary words
Recognizing text type,
Recognizing different types of text structure,
A recent borrowing from the fields of psychology and education, verbal reports, or Think-Aloud Protocols, are widely used for looking into ‘the cognitive strategies of adult learners and children reading L2 texts’ and validating judgments in qualitative methods [8]. Everson and Kuriya [9] cite Ericsson and Simon (1984) who places value on verbal reports because they have the potential to ‘reveal information that might otherwise be hidden or lost from the researcher.’ Cohen [4] discusses the types of verbal report data, which he classifies into two as the process of responding
and reactions to items and subtests. The former concerns with test items and procedures; the latter with the ease of passages and students’ attitudes toward them. In their overview of research methods used for strategy research, Oxford and Crookall [10] one of the two ways of finding out learner strategies that can be used ‘instead of observing or intuiting,’ the other being interviews.
This study will basically replicate a part of Anderson’s 1991 study [3]. Anderson looked into what reading strategies weaker and stronger students employed in reading comprehension tests and academic tasks, what their individual differences in strategy use were, and what distinguished ‘good comprehenders’ from bad ones. The primary research question to be addressed in this paper is as follows: Will weaker students differ from good students in their use of strategies while they are doing a TOEFL reading test?
Method
Participants
The participants for this study were selected from among twelve university students preparing for the TOEFL. They were all attending a TOEFL preparation course at an educational consultancy that specialized in test preparation. Two good readers and two average readers were chosen according to their performance after the administration of a TOEFL reading test. The participants had all studied English at prep classes for two years, once before high school and once at their university. They had been studying English for at least six years. The poor readers overall TOEFL score were 490 and 513 on a recently given TOEFL test. The good readers had overall scores of 553 and 557.
Materials
The materials used in the study were from original tests previously given by the ETS. The initial test given at the beginning of the study
comprised five reading passages, each 300-400 words in length, accompanied by a battery of 9-12 questions based on the information stated or implied and vocabulary questions about some specific words in the passage [11].
Design of the Study
In the beginning twelve students were given a 50-question standard TOEFL reading comprehension test. The two students who got the highest grades were chosen as good readers, and the two who got the lowest were chosen as average readers. The results were further confirmed both by checking the students’ grades against their performance on the TOEFL tests they have taken before and by interviewing their teachers about the students’ overall performance and reading performance.
The students were instructed on how to think aloud when they took a test and were given some opportunity to try it out. Later the students were given a single reading passage with accompanying questions on two separate occasions and their responses were recorded on the computer. The students were also observed for their behavior and approach to the task by the author. The recorded data were then transcribed and analyzed for any emerging pattern that might characterize the students’ strategy use. Then the utilized strategies were translated into the ‘Categories of Processing Strategies’ checklist adapted by Anderson [3].
Data Analysis
The original Anderson checklist [3] included forty seven strategies grouped under five headings: supervising strategies, support strategies, paraphrase strategies, strategies for establishing coherence in text, and test-taking strategies. Although this checklist was derived from an analysis of strategies used both in general comprehension and test taking, it served a useful guide for test taking strategies alone because comprehension is integral to test taking. At least, this is what test constructors strive to achieve. Some of the strategies used in the Anderson checklist did not emerge in this study, which does not imply that the students are not aware of them but simply that they did not use them during the data collection procedure.
Table 1: Supervising strategies
The reader: S1 S2 S3 S4 S1 S2 S3 S4
1. refers to the experimental task 7. makes a prediction about the meaning of a
word or about text content X X X
2. recognizes loss of concentration X 8. refers to lexical items that impede
comprehension X X
3. states failure to understand a portion of the text
X X X X 9. confirms/disconfirms an inference X
4. states success in understanding a portion of the text
X X X 10. refers to the previous passage
5. adjusts reading rate in order to increase
Comprehension
X 11. responds affectively to text content X
6. formulates a question
The supervising strategies employed by the students showed similarities. All the students expressed failure to understand a portion of the text. ‘I can’t understand the passage’ was a typical statement by both poor readers. The good readers knew where they did not understand. One said, ‘I
can’t understand the middle part; I’ll read again.’ S1, a good reader, adjusted his speed, and responded affectively to the text content. ‘This reminds me of a memory,’ was such a sentiment.
Table II: Support strategies
The reader: S1 S2 S3 S4 S1 S2 S3 S4
13. expresses a need for a dictionary 15. scans reading material for a specific
word or phrase X X X X
12. skips unknown words X X X X 16. visualizes X
14. skims reading material for a general
Understanding
X X X
The students were uniform in their use of skimming, scanning, and skipping unknown words. Only S4 did not use skimming, who started each passage by underlying the words asked in the questions and moved on question by question.
No one mentioned the need for a dictionary probably they were used to exam procedures. Only S1 talked about visualizing the information in any other passages, saying, ‘A pattern started to form in my mind.’
Table III: Paraphrase strategies
The reader: S1 S2 S3 S4 S1 S2 S3 S4
17. uses cognates between L1 and
L2 to comprehend 20. translates a word or a phrase into the L1 X
18. breaks lexical items into parts 21. extrapolates from information presented
in the text X X
19. paraphrase X 22. speculates beyond the information
presented in the text
X
Table III shows that the students did not employ most of the paraphrase strategies. The results were interesting in that students three of them did not use translation at all and failed to break lexical items into parts, two supposedly widely taught strategies. Also interesting is the use of translation by S2, a good reader. However, such a use might still result from the use of verbal protocol and the student’s urge to express himself or inability to give a definition or synonym. In
fact, translation emerged on few occasions and even then in simple word translations. In addition, the students did not use cognates not because there were not any items in the reading passages but probably because they did not relate directly to the questions. Only S1 used paraphrasing and speculating beyond the information presented in the text.
Table IV: Strategies for establishing coherence in text
The reader: S1 S2 S3 S4 S1 S2 S3 S4
23. rereads; X X X X 27. uses background knowledge X X
24. uses context clues to interpret a word or
phrase
X X X X 28. acknowledges lack of background knowledge
25. reacts to author's style or text's surface structure
29. relates the stimulus sentence to personal experiences
X
Table IV presents strong proof that good readers employ more strategies to establish coherence in text. Although all students used rereading and context clues to interpret a word, only good readers read ahead and used background information. One of the good readers said, for example, ‘I got an anthropology class,’ implying
his familiarity with the topic and relating the passage to personal experiences. Interestingly enough, none of the students commented on their lack of background knowledge probably because they are familiar with the unfamiliarity of the passages in the TOEFL.
Table V:Test-taking strategies
The reader: S1 S2 S3 S4 S1 S2 S3 S4
30. guesses without any particular considerations
X 39. reads the questions and options before reading the passage
X X X X
31. looks for the answers in chronological order
in the passage
X X X X 40. changes an answer after having marked one
X
32. selects an answer not because it was thought to be correct, but because the others did not seem reasonable, seemed similar, or were not understandable
X X X X 41. receives clues from answering one question that are helpful in answering another
X
33. selects an alternative through
deductive reasoning 42. stops reading the options when they reach the answer X 34. matches the stem and/or
alternatives to a
previous portion of the text
43. expresses uncertainty at correctness of
an answer chosen X X X X
35. selects a response because it is
stated in the text X X 44. skips a questions and returns to it later X X X X 36. selects a response based on
understanding the material read
X X X X 45. skips a questions that is not understood and leaves the response blank
37. makes reference about time allocation
X X 46. marks answers without
reading in order to fill the space 38. reads the questions and options
after reading the passage X X 47. recognizes during the think-aloud protocol that an answer marked is incorrect X X X
Of the eighteen strategies included in the test-taking strategies, three were not applicable to the participants in this study. However, at least one of them used any one of the other strategies. All the participants used four of the test-taking strategies and used them extensively. The most common of these was reading the questions before reading the passage. The participants generally started with reading the question and, skipping the options, referred to the passage and then came back to the options. They all expressed their feelings of uncertainty on different occasions, but average readers responses abounded with such feelings. ‘I can’t decide between the choices,’ said one of the average readers, but then answered the correct choice, admitting, ‘I’m just moving on by guessing at the answers.’
One of the good readers confessed that he was at a loss while answering one of the questions, later changing strategy, rereading the related parts of the passage, eliminating answer choices one by one and finally finding the correct answer. Eliminating answer choices was rampant, especially by the good readers. In fact, both explicitly referred to the process of elimination, whereas average readers tended to use it without verbalization, perhaps because they did not feel confidant when they were eliminating choices.
Some of the strategies were exclusively used by the good readers. Both participants made reference about time allocation. Although no limitations were set on timing, they closely monitored how much time they spent on questions. In fact, they took twice as much time to answer each passage. It took the average readers less than ten minutes to answer a set of questions, whereas for the good readers it was no less than sixteen minutes. Moreover, none of the average readers changed their answers once they chose one, nor did they receive clues from other questions to answer another.
Discussion and Conclusion
A finding of this study that contradicts to Anderson’s study [3] is that good readers employed a greater number of different strategies than the average readers. Good readers used an average of twenty-five strategies, while average readers used an average of sixteen. However, even when average readers used the same strategies as good readers did, they displayed failure to monitor the success of their strategies. Their strategies basically focused on reading the question, referring to the text, feeling frustration when they did not understand, skipping the question, and returning later to guess at an answer. When the good readers skipped a question, they were more confident on their revisit because they understand the overall picture in the passage better. For example, when he was answering the next question after skipping the previous one due to frustration, one of the good readers exclaimed, ‘I found it!’
The total number of correct answers for each student resembled that in the reading test given at the beginning of the study. The students answered 17, 19, 14 and 12 questions, respectively. However, in one of the tests, S3 scored almost as well as S1, answering 9 questions correctly out of 11. This is highly interesting because on at least six occasions he expressed uncertainty about his choice and admitted having guessed at the answer. In Cohen’s [4] terms he was getting the items right for the wrong reasons, or by pure luck.
Several explanations might account for the average readers’ use of fewer strategies. It might be due to their lower level proficiency, which prevents them from going beyond ‘bottom-up’ approaches. They seemed to be struggling with word processing skills and admitting defeat in a sense when they unable to make sense of the passage. One of the average readers said, ‘there are way too many questions! I’ll guess at the answers,’ ‘I have to plod through this passage,’ ‘I’m going to cry!... the Latin names are frustrating.’ The other average reader did not take the passages seriously at all, and walked through them seemingly professionally. For example, when looking for the synonym for ‘massive’ in the phrase ‘massive blankets’ he went for the answer choice ‘cold’ because the passage was about glaciers.’ Another possible reason is that they were not instructed on how to use strategies effectively and so got stuck with fewer alternatives. However, the average readers’ threshold levels and acquisition of automaticity seem to be playing a more important factor.
Clarke’s [11] short-circuit hypothesis that explain good readers lapsing onto poor reading habits presents a more plausible explanation for the average readers’ use of fewer strategies.
An obvious limitation of this study was the number of participants. A greater number would have yielded more reliable results. Another limitation was that especially average readers were not very willing to follow the think aloud protocol. Despite some practice and urges by the author, they frequently fell silent, trying to deal with the passages individually. Also, the findings may not be generalizable not only because of the abovementioned reasons but also because of Turkish students’ extreme familiarity with test-taking strategies. Especially the average readers might have been transferring skills from their L1, in which they have answered a multitude of multiple choice questions, thus employing fewer strategies.
This study focused on the general strategies TOEFL test-takers use when they are doing the reading component of the test. Further studies should research into the strategies students use for different kinds of questions. For example, detailed analyses of the way students respond to inference questions or to main idea questions in particular can be of enormous help to test constructors. In fact, studies addressing this question are underway. For instance, a recent study by Cohen and Upton [8] examined students’ test management and test wiseness with respect to question types and concluded that students used the former rather than the latter perhaps due to the effects of the experiment. Still, by using such information test constructors can move a step closer to achieving greater validity.
Stoller’s [2] advice: The goal of reading instruction
is not to teach individual reading strategies but rather to develop strategic readers, a development process that requires intensive instructional efforts over a considerable period of time.
References
1. Urquhart AH, Weir CJ (1998) Reading in a second language: Process, product and practice. Longman, London.
2. Grabe W, Stoller FL (2002) Teaching and researching reading. Pearson Education, London.
3. Anderson NJ (1991) Individual Differences in Strategy Use in Second Language Reading and Testing. The Modern Language Journal, 75(4):460-472.
4. Cohen AD (1992) Test-Taking strategies on ESL language tests. Minne TESOL Journal. 10:101-115
5. Carrell PL (1989) Metacognitive Awareness and Second Language Reading. The Modern Language Journal, 73(2):121-134.
6. Cohen AD, Upton TA (2006) Strategies in responding to the new TOEFL rading tasks. Monograph Series, ETS, New Jersey.
7. Thompson I (1988) Reading research and its curricular implications. The Slavic and East European Journal, 32(4):617-642.
8. Cohen AD, Upton TA (2007) ‘I want to go back to the text': Response strategies on the reading subtest of the new TOEFL®. Language Testing, 24:209- 250.
9. Everson ME, Kuriya Y (1998) An exploratory study into the reading strategies of learners of Japanese as a foreign language. The Journal of the Association of Teachers of Japanese, 32(1):1-21.
10.Oxford R, Crookall D (1989) Research on language learning strategies: Methods, findings, and instructional issues. The Modern Language Journal, 73(4):404-419.
11.ETS (2003) Practice tests volume 1. Educational Testing Service, New Jersey.
12.Clarke MA (1980) The short circuit hypothesis of ESL reading – or when language competence interferes with reading performance. The Modern Language Journal, 64(2):203-209.
