The methodology of Project Two

As can be seen above, Project One employed a quantitative method in order to answer its primary research question. While there were 150 readings available for statistical analysis, the sample size was relatively small (N=30 for the quantitative analysis). Thus, a questionnaire survey, Project Two, was undertaken in order to investigate whether or not the partial results of Project One would be confirmed with a larger sample size (See Appendix 10 for a part of the questionnaire used in Project Two). The survey investigated the following research questions which were some of the focal points of Project One:

1. How does a larger sample-size of students perceive an unmodified text and three differently modified versions, i.e., a simplified text, an elaborated text and an easified text?

2. Which do learners prefer, hard copy reading or computerised reading?

Also, the same question that was answered by the participants in Project One was used in the questionnaire to investigate what factors the current respondents find to be more influential determiners of text difficulty.

5.2.1 Respondents of Project Two

The number of students who answered the questionnaire was 51.18 They were from second (N=28) and third year level (N= 23) Japanese language courses from the same university at which the participants were recruited for Project One.

Therefore, the respondents’ L2 linguistic proficiency and learning experiences were similar to those of the participants of Project One. The average L2 learning duration of the respondents was 4.43 years for the second year class (hereafter, Class 2) and

18

130

5.63 years for the third year class (hereafter, Class 3). As previously explained, the goals of the two classes were to reach the former JLPT Level Three (Class 2) and Level Two (Class 3). The detailed descriptions of these two levels are previously provided in section 5.1.1.

5.2.2 Ethical issues

The survey was carried out in three classroom sessions at the university in August, 2011. The university’s Human Ethics Policy (Victoria University of Wellington, 2003, p.6) does not ask researchers to obtain ethics approval when participants only complete written questionnaires without a possibility of their identities being disclosed, and when such research meets certain criteria including complete anonymity, no inclusion of sensitive questions, guarantee of disposal of the completed questionnaire forms, and so forth. The conducting of the

questionnaire survey was reported to the Head of the School and the Human Ethics Committee was informed by email.

5.2.3 Actual administration of the questionnaire

The researcher visited the two second year classes and carried out the survey, giving an oral explanation of the survey’s purpose, total anonymity and

voluntariness although the potential respondents could find all such information on the questionnaire form. The researcher was an instructor of the third year class at the time of the survey. Therefore, she clarified that the present students in class would have the right not to respond to the questionnaire, which would not affect their course grades.

The questionnaire survey took approximately 20 minutes. This time

allocation was judged as appropriate from the pilot testing by a learner of Japanese as described in the next section. Since the researcher was present for all three of the

131

survey sessions, the respondents could ask her to disambiguate the meanings of instructions/questions when necessary.

5.2.4 The questionnaire used in Project Two

The questionnaire used in Project Two is provided in Appendix 10. One intermediate learner of Japanese went through all the questions so that the researcher could detect possible problems. Since the researcher is not a native speaker of

English, this learner’s input into the wording of the questionnaire was of importance. Also, this process was important in terms of the issue of face validity of the

questionnaire. Brown (2001, p. 176) mentions that “typically, the face validity of a survey instrument is assessed by asking a group (similar to those who will

eventually be the respondents) to look at the instrument or go through the process of answering the questions, and then make a judgment about the degree to which the instrument seems valid.”

Prior to this test run with this learner, two academics (the aforementioned linguist and another scholar of Japanese who has acquired near-native-level

accuracy and fluency in Japanese and have also been in Japanese language teaching fields for more than a couple of decades) were consulted for the appropriateness of its content. It was believed that this process enhanced the construct validity of the questionnaire. Brown (2001, p. 177) recommends researchers “to use experts, who are by definition people who know a lot about whatever area of psychology,

education, linguistics, or language teaching your construct belongs to” in order to study the construct validity of survey instruments. Similarly, the content of Section B is thoroughly based on theoretical foundations as argued in section 5.1.7.2. Basing on theoretical foundations is also proposed by Brown (2001, p.177) for the construct validity.

132

5.2.4.1 Section A

The questionnaire consisted of four parts. Section A asked the respondents’ learning history, reading habits, desire for mastery of L2 reading and preference between hard copy reading and computerised reading. This section was limited to only five questions due to time constraints. This was unavoidable since it was anticipated that the following sections would require much time to respond.

Nevertheless, more than half of the questions in Section A had an open-ended option, which would encourage the respondents to report detailed responses if time allowed.

5.2.4.2 Section B

Section B asked the respondents to rate eight factors in terms of the influence on text difficulty. This is the four-scale rating question used in the biodata

questionnaire in Project One. And the rationale for the eight factors is explained in section 5.1.7.2.

5.2.4.3 Section C

Section C asked the respondents to read a short Japanese passage

(unmodified) and circle parts which were difficult for them to understand. It also asked them to provide comments about what caused the difficulties in the passage. An example of how to respond to the question was presented in section C (Figure 5.3 below) because this question may be unfamiliar to some respondents and as a result incur ambiguity. This example was made by the learner of Japanese who pilot- tested the questionnaire. For this example, a different passage was used in order to avoid implicitly guiding the respondents’ reaction possibly caused by the use of the same passage.

133

Figure 5.3. Example of response in Section C

Section C was motivated by Harrison (1980). He was inspired by one researcher who asked children to mark easy parts in a text with a blue pen and difficult parts with a red pen. Then Harrison advises that teachers and researchers ask learners to underline difficult parts of a text and accumulate such underlined parts as important signals which indicate problematic textual features. Harrison claims that such features are candidates for re-writing. Cramer (2005) took a similar method in her text modification research. She asked her participants to “mark any words that they had never seen before” and used the obtained result to examine her participants’ different vocabulary knowledge between original texts and lexically simplified texts.

The respondents are not guided to respond in any particular way with this method. They have total discretion regarding what they choose as causes of text difficulty and what they comment about the chosen items. Since the example (Figure 5.3) mentioned kanji factors, the researcher orally clarified that comments did not have to be kanji factors but could be about anything. However, the researcher refrained from exemplifying any further in order to avoid implicitly guiding respondents’ answers.

134

5.2.4.4 Section D

Section D asked the respondents to read three short passages which were three differently modified versions of the original text read by them in Section C. They were a simplified version, an elaborated version and a version with a marginal gloss. These names were not used in the questionnaire since the names per se might be misleading. Therefore, the three texts were simply called ‘version 1,’ ‘version 2,’ and ‘version 3.’ The respondents answered which was the easiest/most difficult version and why it was so. The texts were modified in a similar way to the modified texts in Project One. That is, each had a key word section at the top of the text, and a gloss was provided in Japanese, which had the same word/phrases used in

simplification and elaboration.

The original topic of the text used in sections C and D was a story about a new type of homeless people in Japan nicknamed ‘internet refugees.’ It was assumed that this story did not require any expert knowledge. Therefore, the respondents’ differing background knowledge was unlikely to affect their reading comprehension.

5.2.5 Coding of the respondents’ motivational factors

Question four contained in Section A (Figure 5.4) presented below involved subjectivity in the process of interpretation.

Figure 5.4. Question (4) of Section A on the questionnaire used in

Project Two

(4) Do you want to master or improve your Japanese reading abilities? Why? Yes, I do want to improve them.

 Why? _________________________________________________________ No, I don’t want improve them.

135

The respondents’ answer either demonstrated some motivational sources or reasons for the absence of motivation. In order to minimize the researcher’s subjectivity/bias, the aforementioned academic who acquired near-native-fluency in Japanese and an accomplished linguist also categorized the respondents’ answers. Then the

categorizations presented by three of them were matched.