Measuring Perceptions of Ability with Questionnaires

Student and teacher questionnaires were developed to collect both quantitative and qualitative data from participants regarding their perception of English language ability. Questions asked were based on questionnaires used in previous research about students‟ perceived English language ability and faculty members‟ perceptions of their students‟ ability to cope with English-medium instruction (Byun, et al., 2011; Collins, 2010; Craig, 2007; Vinke, 1995; Vogt & Oliver, 1998). Previous research into teachers‟ perceptions of students‟ ability (Craig, 2007), students‟ study habits (Douglas, 1977), and recommendations for increasing language acquisition of adult second language learners (Ferris, 2003; Gass & Selinker, 2001; Krashen, 1995; Krashen & Terrell, 1995) helped me develop and frame the survey questions. Questions center on perceived English language ability and improvement, and the ability to perform tasks related to academic study in English. The teacher

questionnaire has parallel items, along with questions related to the delivery and adaptation of materials due to teaching in the learners‟ second language. The

teacher‟s questionnaire also includes several questions with demographic components to get a better idea of who is answering the survey and how their previous experience may affect their responses. The questionnaires include various types of questions in

order to provide both quantitative and qualitative data in relation to both teachers‟ and students‟ perceptions of language ability (see appendices D and E).

By using a variety of item types on the questionnaire, different analysis techniques are possible which offer a range of ways to look at the data collected. The use of closed questions is a useful way to generate frequency of response information that can be statistically analyzed. They facilitate comparison between groups and are quicker to code and analyze than open ended questions (Cohen et al., 2007). The use of rating scales, which are commonly used in research, offer “flexible response with the ability to determine frequencies, correlations and other forms of quantitative analysis (p. 327). Ratings scales are “useful for tapping attitudes, perceptions and opinions” in a way that allows for statistical measurement (p. 328). They allow the researcher “to fuse measurement with opinion, quantity and quality” (p. 327). As with any item on a self-administered questionnaire, rating scales have limitations. There tends to be an avoidance of extremes; the actual meaning of the scale items may be different for different respondents; there is no assumption that the scale intervals are equal; and the scale limits the respondents to the choices presented (Cohen et al., 2007). In this research, triangulation of data sources and collection methods help to reduce these limitations.

The open-ended questions on the surveys offer a chance for students and teachers to more fully explain their choices if needed and are a “window of

opportunity for respondent[s] to shed light on an issue” (Cohen et al., 2007, p. 331) even though they are more difficult to handle as data and comparison between groups is difficult. The questionnaire for teachers has more open-ended questions including some asking about the adaptation of materials, the amount of support offered, and the need for support. As student writing and reading tend to be among the weaker skills in this region of the world, open-ended questions are limited in the student

questionnaire, but were explored more fully during interviews. 4.5.4.1 Questionnaire Administration

Using the software application SelectSurvey.NETv4.032.002, the

questionnaires were put onto a secure site where participants were able to take the survey by clicking on an email link. Advantages to using online questionnaires

instead of paper-based ones include a reduction in the amount of time to distribute and collect data, a decrease in the researcher effect, the ability to quickly transfer data

from one application to another for analysis with a decrease in processing and data entry errors, the ability for respondents to complete the questionnaire at a time and place that suits them, and fewer missing responses to items (Cohen et al., 2007). 4.5.4.2 Questionnaire Content

The first page of both the student and faculty questionnaires contains information about the purpose of the research and a consent form. It includes a statement about the research, expected benefits, risks, confidentiality, and then asks participants to indicate agreement to participate by clicking on the “next” button, which takes them to the first page of questions. Using Likert scale items, the

questionnaire requests participants to mark on a five-point scale of one ( poor) to five (excellent) how they would rate English-language ability in each of the four areas tested by the IELTS exam, how they would rate the ability to perform academic tasks (e.g., reading course materials, taking notes during lectures, asking questions, and writing academic papers), and whether they think they (or their students) have made improvement during the four years of study in any of the skill areas and why. It also asks if the participant would be willing to participate in an interview to further discuss their answers to the questions on the survey.

4.5.4.3 Increasing Response Rate

As Cohen et al. (2007) point out, there are various issues associated with response rate including timing, design and presentation of request, and follow up. Timing is an important factor in response rate. Being a part of the institution where the data was collected helped me to know when the best time was to request

participation to get the maximum response. The email requests to participate were sent in February right at the start of the semester, when both students and teachers are not overwhelmed with upcoming exams. A clear cover email was written that I hoped would appeal to participants based on their role as a teacher or student. My identity and work within the institution was highlighted in the emails asking for participation.

A follow-up letter is one of the most productive ways of increasing response rate (Cohen et al., 2007). Aware that some faculty members might be annoyed by repeated request emails, I sent only one reminder to teachers one week after the initial request. Email requests for faculty members were addressed at the college level, for example, “Dear College of Education Faculty Members.”

For students, emails were individually addressed with their name and a congratulations message about getting close to graduation. I sent several reminder emails to student participants as it was difficult to tell whether they had received the first request or not. Many were not participating in classes because they were on internships in their final semester, plus the university had switched to a new platform for email delivery at the beginning of the semester. Also, because students are of a different generation than the faculty members, they have different habits and levels of tolerance for electronic communication. For the students I had a clear target for collecting data. I wanted as many of the 75 students who had started the program in September 2007 as possible to respond and could identify who had responded because the online survey requested identifying information enabling me to target reminder emails for participation.

Clear instructions are important in response rate (Cohen et al., 2007). Thus in the email, the link was clearly highlighted in the text. If someone clicked on the link without reading the email, the purpose and instructions for answering the

questionnaire were again laid out on the survey welcome page. I also included my name and contact information in both areas.

Using the various techniques set forth by Cohen et al. (2007) to increase participation (e.g., flattering participants, personally addressing emails, indicating benefits and importance of research, and using follow-up emails), I had a response rate of 33% (53) from faculty members solicited and 60% (45) from students. Only 35 of the students‟ responses are included in this research as the other 10 did not have IELTS results for Test 2 at the end of the semester.