Collecting quantitative data: Phase Four

The process for collecting quantitative data comprised five steps: selecting participants,

obtaining permissions, selecting types of data, developing and piloting the questionnaire,

and administering the questionnaire. I will describe the process in detail next.

Step 1 Selecting participants

The participants involved in collecting the quantitative data were, as in the interviews,

intended to be those who have past or present experience of selecting or helping to

select a textbook for a General English course to non-English major students in

Universities and Institutes of Technology in Taiwan. I attempted to collect Taiwanese

EFL teachers‟ e-mails, including full-time and part-time teachers, from the websites of Departments of Applied Foreign Languages, Language Teaching Centres or Centres for

General Education, in all 78 Taiwanese Universities and Institutes of Technology (see

Step 5 below for further information). Part-time teachers were invited to participate in

this research in order to increase the number of responses. Those who participated in the

interviews were also invited to fill in the questionnaire. The reason for this was that I

121 Step 2 Obtaining permissions

An invitation letter was included as part of the online questionnaire to obtain

participants‟ permission. The information included in the invitation letter included the purpose of the study, the time required for completing the questionnaire, how I would

be using the data or results, the benefits of the study and the provisions I had made to

protect the anonymity of participants.

Step 3 Selecting types of data

Brace (2004:4) points out that the role of the questionnaire in large-scale surveys is to

provide a standardized interview across all respondents, that is, the same set of questions is asked „in a predetermined order‟ (Gary, 2004:187) to different respondents. Therefore, the researcher can handle and interpret large levels of data with a

standardized question format. Although inadequate design or data collection could be a

challenge for achieving „sufficient (and well-documented) psychometric reliability and validity‟ (Dörnyei, 2003:3) to answer the objectives of the research, Gillham (2000:1-2) still stresses that questionnaires „have their place as one method, of most value when used in tandem with other methods‟. The results of multiple methods converge; the findings will be more reliable and valid to the research. Borg (2006: 169) adds further

that

questionnaires elicit beliefs, knowledge and attitudes through instruments, typically consisting of a series of questions or tasks, which require a written response. These written responses are then used as evidence of the respondents‟ cognitions.

122 Dörnyei (2003:8) then illustrates that a questionnaire can provide three types of data

about the respondents: factual questions, behavioural questions, and attitudinal

questions. Factual questions help to find out the background information of the

respondents that may be relevant to interpreting the findings of the research. This could

include, for example, demographic characteristics, level of education and the number of

years teaching. Behavioural questions then help to find out what the respondents are

doing or have done in the past, focusing on actions, life-styles, habits, and personal

history. Finally, attitudinal questions help to find out what people think, covering

attitudes, opinions, beliefs, interests and values. Therefore, we can, for example, ask

teachers to prioritise the criteria for selecting a textbook. As a result, factual questions

and attitudinal questions were used for this structured questionnaire.

Step 4 Developing and piloting the questionnaire

The questionnaire aimed to investigate what criteria are important or unimportant for

the Taiwanese teachers when they select a textbook for a General English course. In the

quantitative phase of the study, a closed-ended and structured questionnaire was

developed based on the three-phase interview data. I will illustrate how I developed and

piloted the questionnaires.

 Developing the questionnaire

The questionnaire items adopted all the preliminary set of Taiwanese teachers‟ criteria

established in the qualitative data analysis. The questionnaire was written in Chinese as

the interviews were conducted in Chinese. I combined and/or refined some of the

criteria in this preliminary set of teachers‟ criteria with a view to presenting the

questions more clearly to the respondents. T3 and T6 commented that a textbook should „introduce new vocabulary in preparation for reading texts‟, and T7 commented that a textbook should „introduce new vocabulary in preparation for conversations‟. I

123 summarized these statements as a textbook should „introduce new vocabulary in

preparation for other activities, e.g. reading texts or conversations‟. Second, a definition for numbers or length is given by one of the teachers‟ statements to help respondents answer the question. For example, „a textbook should not introduce too many new words per reading text (e.g. no more than 10 new words)‟. Third, some examples from the teachers were given to the same criterion to clarify the questions for respondents.

For example, „a textbook should have rich resources and supplementary information (e.g. cultural background, new vocabulary or a variety of optional activities) in the

teacher‟s manual‟. Finally, to make the oral statements clearer and more coherent, I added some linking words or rearranged the order of the words where it was necessary.

I was aware that it is a dilemma to design the questionnaire items precisely according to

the interview data because some statements from individuals might be vague and

subjective; or the examples might not have been given by the teachers; for example the

definition of the number of large classes, the length of reading texts, or the number of

new items of vocabulary introduced in a reading text; or the fact that the information

was given by only one teacher. Indeed, I attempted to find out which criteria are more

important, rather than prejudge the teachers‟ views; therefore I decided to stick with the teachers‟ perceptions, instead of putting my words into the questionnaire as I wanted to retain the non-judgemental nature of my research.

 Pilot Study

A pilot study was conducted to refine the questionnaire.

Aims

The aims of piloting this questionnaire were to clarify the instructions, to check any

124 Manion and Morrison (2003) suggest, a pilot study could enhance the reliability,

validity, and practicability of the research instrument.

Setting

The questionnaire was uploaded on a survey Website (my3q.com) for participants to

complete. The revised version of the questionnaire was also placed on the same Website

(see Appendix 7).

Procedure

There were six teachers involved in developing the questionnaire prior to the main large

scale study. The pilot study for the questionnaire was conducted with three teachers

who work in similar Universities to those for whom the questionnaire was designed and

three teachers who are professionals in designing questionnaires in the University of

Warwick. They went through the online questionnaire I sent to them and gave me

valuable feedback.

Changes to the questionnaire

I revised the questionnaire according to the comments from the pilot study as follows.

First, the instructions needed to be clearer. I added „non-English major department‟ to the title. Second, the questions in the „Differentiation‟ section were combined with the

„Students‟ Needs‟ section. Also the ideas that „a textbook should meet students‟ needs, e.g. to pass the GEPT exam‟, „a textbook should match the difficulty level of the target exam, e.g. the GEPT exam‟, and „a textbook should match the task types and question patterns of the target exam, e.g. the GEPT exam‟ were combined into one question as „a textbook should meet students‟ need to pass the GEPT exam, e.g. match its difficulty

level, task types and question patterns‟. Third, the ideas that „a textbook should give

activities for fluency practice‟ and „a textbook use sufficiently large font size‟ were deleted as they could be identified in other questions. Fourth, the ambiguous and

125 unclear questions were revised, for example, the idea that „a textbook should give short

dialogue exercises‟ was revised as „a textbook should give short dialogue practice activities‟. The idea that „a textbook should provide exercises to match students‟ learning styles, e.g. provide exercises with close-ended questions, sentence drills or

making full sentences‟ was revised as „provide controlled exercises to match students‟ learning style, e.g. provide exercises with close-ended questions, sentence drills,

matching or True/False‟. The idea that „a textbook should provide learning

opportunities for teachers‟ was revised as „provide learning opportunities for teachers, e.g. new knowledge or information‟ and the idea that „a textbook should be chosen according to students‟ feedback‟ was revised as „a textbook should have been positively evaluated by students in previous years‟. Finally, the participants reported that the questionnaire could be completed in 15-20 minutes.

The final version of the questionnaire was developed and refined back and forth in the

process of qualitative analysis, questionnaire development, and questionnaire piloting.

As a result, the questionnaire consists of 88 items. First, it asks teachers to prioritize the

criteria. The items are grouped into eight categories by following the format in the

interview results in order to analyse the results later: Overall Construction (17 items), Students‟ Needs (13 items), Teachers‟ Needs (16 items), Authenticity (6 items), Self-instruction (6 items), Cultural Issues (3 items), Visual Design (5 items) and

Practical Concerns (13 items). Second, it asks for teachers‟ background information (7

items). I was aware that some criteria might not be identified from the teachers in the

six selected universities. Therefore, two open-ended questions are also provided at the

end of the questionnaire to invite teachers to list any other criteria which have not been

mentioned with associated reasons for their choosing those criteria. This also provides

126 selection or on this questionnaire. The final point is that the rating scales in this

questionnaire contain four points to avoid respondents selecting the non-committal

central point, according to my experience of conducting surveys.

Step 5 Administering questionnaires: Phase Four

I decided to send the questionnaire to teachers by e-mail invitation. The drawback of an

online questionnaire is that a population list cannot always be fully provided, and

contact details are not always updated, correct or provided. It is also difficult to identify

which contacts are most qualified to participate in this type of research, as the e-mail

addresses do not provide details on whether the contact is responsible for teaching

English to non-English major students. The situation varies from university to

university because of University self-government as I have explained in section 1.1. It is,

however, known that all contacted people are EFL teachers in a Department of Applied

Foreign Languages, Language Teaching Centre, or Centre for General Education.

Accordingly, I can only claim that all the (138) teachers who filled in the questionnaire

were qualified to participate in this research, not that all the (893) teachers I contacted

were in fact qualified according to the criterion I had set up (see Step 1).

I managed to collect teachers‟ e-mails from 73 out of 78 Universities and Institutes of

Technology in Taiwan. The remaining five universities did not provide any online

information on individual teachers, although I did try to contact the departments or the

centres. This could be attributed to the fact that some Institutes of Technology were in

the process of being upgraded to Universities of Technology at that time so they were

updating the information on their websites, for example. The news about the upgrading

were widely reported in the media later. In total, 138 out of 893 EFL teachers completed

and returned the online questionnaire. Now I shall describe the procedures used to

127