Materials

The test length was determined so that the participants could complete the test within a certain period of time. As the test was administered to high school and university students during their normal class hours as part of their class activities, it needed to be completed within 50 minutes which corresponded to one class period at high school. As

9

The TOEIC scores available may not be fully representative of the participants, because some classes required students to take TOEIC, whereas others did not. However, the purpose here is to show that the participants’ proficiency levels varied widely.

74

it took 20 minutes to distribute the test, explain about the consent form, provide the instructions, and collect the answer sheet, the test time was set at 30 minutes. Pilot study 3 in the previous chapter indicated that the participants would be able to complete 20 items in 30 minutes (1.5 minutes per item); thus, each participant worked on 20 out of the 60 items in the GCT.

Six different test forms were created in order to evaluate all 60 items in the GCT using Rasch analysis. As shown in Table 8, the 60 items in the GCT were randomly classified into six groups (Item groups 1 - 6) each of which consisted of ten items. Six forms (Forms A - F) were created by systematically combining the items in two of the six item groups. Each form consisted of a total of 20 items, ten of which overlapped with another form and the other ten of which overlapped with another different form. For example, Form A shared the ten items in Item group 1 with Form F and the ten items in Item group 2 with Form B. This systematic link between any two forms was designed for linking the six forms in order “to put all the items together into one item hierarchy, and to produce one set of measures encompassing all the persons” (Linacre, 2010a, p. 449). The items that were not included in a form (e.g., the 40 items in Item groups 3 - 6 for Form A) were treated as missing data. Although this design allowed a large number of missing data, researchers (Bond & Fox, 2007; Linacre, 2010a) have argued that Rasch analysis is robust with missing data which can be used intentionally by design.

Table 8. Test design (GCT)

Item group Form A Form B Form C Form D Form E Form F

1 (10 items)   2 (10 items)   3 (10 items)   4 (10 items)   5 (10 items)   6 (10 items)  

75

The test was written in a paper-based format so that the test could be administered effectively in classroom settings. As discussed in the previous chapter, the three questions were ordered as follows: part of speech, contextual clue, and meaning. In order to avoid a learning effect from one question to another for each passage, each question was measured separately in different sections. The first section was about part of speech. In order to minimise a fatigue effect, only the sentence that contained the test word was presented for each item, because the other sentences presented in the other two sections are of little use for determining the part of speech of the test word. The instructions asked the participants to mark 1 for noun, 2 for verb, 3 for adjective, and 4 for adverb on the answer sheet. The test words were written in bold and underlined for easy recognition. Here are two examples from the part of speech section.

1. Like a wine candintock who only has to smell the wine to know how good it is, a cat can learn all it wants to know without actually eating the food.

(1) noun (2) verb (3) adjective (4) adverb

2. Some words sound like other words, even though they are

decontanically different.

(1) noun (2) verb (3) adjective (4) adverb

The second section was about the contextual clue. The participants were asked to choose from the three underlined options the phrase or sentence that was most helpful in guessing the meaning of the test word. Here are two examples.

1. Cats have a good nose for food. Many cats smell food and then (1)walk away without even trying it. Like a wine candintock (2)who only has to smell the wine to know how good it is, (3)a cat can learn all it wants to know without actually eating the food.

(1) walk away without even trying it

(2) who only has to smell the wine to know how good it is

76

2. When we try to look at the (1)process of reading carefully, we will meet a further problem. (2)Some words sound like other words, even though they are decontanically different. An example would be the words (3)“see” and “sea.” These two words sound exactly the same, but they include different letters.

(1) process of reading

(2) Some words sound like other words (3) “see” and “sea.”

The last section was about the meaning of the test word. The participants were asked to choose the meaning of the test word from the three options. Here are two examples.

1. Cats have a good nose for food. Many cats smell food and then walk away without even trying it. Like a wine candintock who only has to smell the wine to know how good it is, a cat can learn all it wants to know without actually eating the food.

(1) consumer (2) specialist (3) seller

2. When we try to look at the process of reading carefully, we will meet a further problem. Some words sound like other words, even though they are decontanically different. An example would be the words “see” and “sea.” These two words sound exactly the same, but they include different letters.

(1) relating to quality (2) relating to spelling

(3) relating to ability

For each section the order of the items was randomised so that an order effect might be minimised. In order to make sure that the participants did not go back to the previous questions, the participants were asked to put the question sheets for each section under the desk every time they finished one section.

For efficient data input, the answer sheet was made in optical mark recognition (OMR) format where the participants mark their answers by darkening pre-printed circles. This format was familiar to the participants because most of them had worked on this format for university entrance examinations such as the National Center Test for University Admissions.

77

The information sheet, the consent form, and the instructions were translated into Japanese, the participants’ L1. This ensured that even low-proficiency students were able to fully understand the necessary information involved in the test. (See Appendix E for the six forms of the GCT used in this study.)