Test Construction

Studies conducted by vocabulary researchers show that reading compre- hension tests and vocabulary tests are highly correlated in first language assessment studies (Anderson & Freebody,1981) and second language (L2) assessments studies (Pike, 1979). Vocabulary proficiency is highly correlated with reading comprehension (Baker, Simmons, & Kame'enui, 1995; Kuhn & Stahl, 1998; Stahl & Fairbanks, 1986; Stanovich, 1986). CRT tests are able to determine whether a student possesses spe- cific skills or knowledge (Huitt, 1996). The present CRT- diagnostic type tests to be administered at the beginning of the course are designed to measure vocabulary knowledge in scientific or engineering contexts in first year undergraduate studies and are based on four reading texts. (see Appendix 4.1, Appendix 4.2, Appendix 4.3 and Appendix 4.4)). Multiple texts were selected for relevant content areas and for testing a range of appropriate vocabulary items. The tests measure contextualised lexical

meanings as recommended by Weir, Yang & Jin (2000) for both receptive and pro- ductive vocabulary knowledge and vocabulary depth knowledge, authenticating a comprehensive vocabulary skill profiling of the test participants. A total of four pre and post tests were designed so that a sufficient number of test items could be selected from context. A total number of 53 content words were selected from context for measuring receptive vocabulary skills, vocabulary depth and productive vocabulary skills. Spolsky (1994 as cited in Weir, Yang & Jin 2000) recommends using a variety of reading assessment measures in order to be able to account for multiple aspects of proficiency.

Other well-known tests such as Advanced English Reading Test (AERT, Weir, Huizhong and Yan, 2000) conducted in Chinese Universities and at CALS, University of Reading, used around 42 items to test the students’ ability to compre- hend academic texts and to extract important information from the texts. While the AERT is not specifically a vocabulary test, it tests readers in context. Other profes- sional English language competency tests such as IELTS and TOEFL tests vocabulary skills more inclusively on language skills with less than 50 items. It is believed that 53 items in the current tests provide a reasonable measure of the range of vocabulary skills tested.

Cronbach, 1942; Nation, 1990; Laufer & Nation, 1995; Bachman and Palmer 1996; and Waring, 1999, among others have emphasised the importance of assessing vocabulary on a multi-dimensional scale. Word knowledge was traditionally assessed in a decontextualised, dichotomous fashion (Wagner, Muse & Tannenbaum, 2006). However what it means to know a word cannot be measured without taking into account the domain and context of reading. The CRT- diagnostic pre-tests for measuring vocabulary proficiency attempt to measure a range of lexical knowledge such as generalisation in domain (knowing the definition). This is important because defining a word in context relates to knowledge of linguistic semantics in context. In reading each sentence, definitions and contexts carry crucial roles, and must be exam- ined in order to assess the reader’s vocabulary proficiency. Context and definition combine and contribute most to reading comprehension (Crist, 2007). The tests meas- ure application of lexical information gathered from the texts (knowledge about use)

through tasks such as explaining meaning of technical vocabulary in context, in short sentences.

Vocabulary researchers (Singleton, 1999; Lewis, 2000; Hunt & Beglar 2005) have acknowledged the significance of the lexicon as a measure of being able to learn to learn and use a language. The notion of vocabulary knowledge, in particu- lar, lexical competence has been the centre of scholarly investigations (Pavlenko, 1999; Singleton, 1999; Nation 2005), and the breadth and depth of lexical competence has been emphasised by other researchers (Wesche & Paribakth, 1996; Laufer & Goldstein, 2004; Nation, 2005 ). The tests analyse breadth of meaning (knowing dif- ferent senses of a word) and precision of meaning (knowing how to use the word in different situations) through summary reading tasks and tasks that require accurate selection of words for sentence completion.

Mohseni-Far (2008) argues that the difference between knowing a word and using a word is one way to study the task of vocabulary learning. Beck, McKe- own & Omanson’s (1987) final stage in the continuum of word knowledge refers to decontextualised knowledge of a word’s meaning, its relationship to other words, and its extension to metaphorical uses. While knowledge of a word’s meaning and its rela- tionship to other words are important in a general sense, it poses limitations on the ability of the word to context-sensitive meanings. The present CRT diagnostic tests measure the ability of the reader to use context to determine the meaning of words and the study participants’ level of vocabulary proficiency. Similar criterion refer- enced achievement post-tests in control and experimental groups of study participants are administered in order to measure vocabulary gains (if any) in experimental groups where specific instruction of vocabulary learning is conducted and for comparison to the performance in vocabulary tests by control groups where standard vocabulary in- struction will be carried out by experienced colleagues .

Waring (1999) claims, that meta-cognitive knowledge is necessary for learners to be able to report on their vocabulary knowledge proficiency. Additional self-reports gathered from study participants through retrospective verbal protocol on vocabulary learning strategies employed provide further information on study partici-

pants’ meta-cognitive knowledge competence. Davis (1968) documented five skills of reading: identifying word meaning; drawing inferences, identifying a writer’s tech- niques and recognising the mood of the passage, and finding answers to the questions asked explicitly or in paraphrase. Weir & Porter (1996) identified vocabulary profi- ciency as a second factor that contributes to reading competence.

In studies in English as a Foreign Language (EFL) it is established that background knowledge plays a role in demonstrating reading ability. Bernhardt (1991, as cited in Weir, Yang and Yan, 2000) maintains that in L2 reading, both language features and textual features are perceived as elements, whereas knowledge elements included in the text are not easily observed. Weir, Yang and Yan’s (2000) review of studies in background knowledge effect on EFL reading summarises a few points that researchers should be aware of. The authors (Weir, Yang, & Yin, 2000) refer to readers’ schematic knowledge, especially content and formal knowledge, which they say affect how and which skills or strategies are used. Schematic knowledge is related to background knowledge and provides a framework for readers to connect to incoming text information.L2 vocabulary research establishes that the path to L2 vo- cabulary acquisition is complicated and complex and for comprehension to occur learners need to know at least 95 % of the words in the text at any level (Hirsch and Nation, 1992). Research also indicate that the relationship between L2 learners’ syn- tactic knowledge of English and text comprehension intersect with vocabulary knowl- edge, background knowledge and L2 proficiency (Garcia, 2003). These assertions validate the use of reading texts to measure vocabulary proficiency in L2 learners.

Laufer and Nation (1995) developed a newer version of the Vocabulary Levels Test proposed by Nation ( 1983; 1990) with similar target words, but instead of matching words and definitions, the test takers need to write the missing target words in a set of sentences. This blank filling version was used to provide evidence for the lexical frequency profile (LFP) test. This test item requires more word knowl- edge and more use of contextual information indicating that it might be a more effi-

cient test of vocabulary knowledge. However, these tests are decontextualised and do not account for the sphere and context of reading.

Read (2000: 9) further proposed three dimensions in connection with vo- cabulary assessment to account for a wide variety of testing procedures. These are discrete to embedded, selective to comprehensive, and context-independent to con- text-dependent. The communicative approach to language teaching has gained promi- nence in the last three decades and poses questions to the validity of the traditional decontextualised vocabulary test and is stimulating a reconsideration of the type of lexical ability, and how it is possible to best assess it. (Read, 2007). In their analysis of current vocabulary assessments, Pearson, Hiebert and Kamil, (2007) found that many vocabulary assessments tested words in a decontextualised manner and recom- mended contextualised vocabulary assessments as a priority to predict both passage specific and general comprehension. The current vocabulary proficiency tests are based on reading texts.