4.4. Research instruments
4.4.3. Piloting
The questionnaire was piloted to address any instrument weaknesses and to improve its reliability. The piloting stage went through three stages: initial piloting, online pilot study, and paper-based pilot study. As can be seen later, each stage of piloting informs the development of the quantitative instruments in many ways.
4.4.3.1. The initial pilot study
On 1 April 2012, the initial pilot study was conducted with four female participants. The final version of the Arabic questionnaire was piloted to check the wording, translation, and the meaning of the questionnaire items. The participants were asked to:
1. Make a note of any items whose wording they did not like, and to suggest improvements
2. Make a note of any items whose meaning was not 100% clear 3. Make a note of any item which they considered unnecessary 4. Provide any overall suggestions and recommendations.
The participants were chosen from different backgrounds. Two participants were English language teachers at a Saudi university, one was a university student in her preparatory year, and one was a postgraduate student in the UK. It was believed that having participants from different backgrounds would provide significant recommendations from different perspectives.
120
Based on the feedback of the participants, some questionnaire items were modified, reworded or deleted. For example:
Item 8 – ‘encourage students to select specific goals for themselves’ – was deleted as it appeared similar to item 5 – ‘encourage students to set learning goals’.
The word ‘curriculum’ in item 3 – ‘build the curriculum based on students’ needs’ – was modified into ‘lesson plan’, and then retranslated in the Arabic version. It was suggested by some participants that EFL teachers in Saudi cannot choose the curriculum, but they can control the lesson plan of each class.
Some items were reworded for the purpose of clarity: items 10, 17, and 48. For example, item 10 – ‘allow learners choices about the learning process’– was reworded into ‘allow learners choices about the English learning process’ to make it specific to English language learning.
Based on this initial piloting, the translation of the instrument was further modified, some items were reworded and one item was deleted (item 8). At this stage, the number of questionnaire items reduced to 65 and the questionnaire was ready to be administered for the main pilot study, which will be discussed in the next section.
4.4.3.2. Online pilot study
In April 2012, an online pilot study was conducted using ‘smart survey online software’. The primary purpose of this piloting stage was to develop a reliable questionnaire scale that is relevant to the Saudi context. The links to the teachers' and students' online questionnaires were emailed to the coordinator of the English language institute
121
at University A, who agreed to facilitate the pilot questionnaire, along with a message including information about the study and inviting teachers and students to participate. Arabic and English versions of the questionnaire were sent to teachers so that they could choose one version to answer. Students were sent an Arabic version. The coordinator of the language institution sent the questionnaire link to more than 100 EFL teachers and more than 250 students. After about a month, the number of participants was 72, of those 50 were teachers and 22 were students, as shown in the following table. The participants were all female.
Table 4.6: The number of participants in the pilot study Participants Number Teachers Arabic version 28 English version 22 Students 22 Total 72
Although students were encouraged by their teachers to participate in the study, the number of students participating in the pilot study was not enough to conduct statistical item analysis as suggested by Hatch and Lazaraton (1991) who suggests including at least 30 participants in each group. This shed light on the difficulty of receiving responses from students using an online questionnaire. Therefore, a decision was taken that the online questionnaire would not be statistically analysed and that a paper-based pilot study would be conducted. However, this stage of the pilot study was significant in many ways, as it assisted the researcher in taking important decisions before conducting the main study. First, it appeared that using an online questionnaire was not feasible in the context of the study, especially in the case of the
122
students; therefore, it was decided that a paper-based questionnaire would be used in the main study. Second, as one of the aims of this study is to compare teacher and students' perceptions about motivational strategies, it was believed that the questionnaire should be administered in Arabic to teachers and students to avoid any effects of language on the differences in their views about motivational strategies. Therefore, in the paper-based pilot study, it was decided that an Arabic version of the questionnaire would be distributed to both teachers and students. More information about this stage will be discussed in the next section.
4.4.3.3. Paper-based pilot study
A paper-based pilot study was conducted since the online pilot study failed to receive an acceptable number of student responses. Twenty-two students answered the questionnaire, while at least 30 responses were needed to perform the item analysis and to test the scale’s reliability. The paper-based questionnaire was conducted at about the beginning of May 2012. The number of participants was 194: 55 teachers and 139 students, as shown in Table 4.7.
Table 4.7: The number of participants in the paper-based pilot study Teachers Students Total
55 139 194
Data were coded using SPSS for Windows (version 17.0), and all negatively worded items were reversed. After the data cleaning, it was found that 11 participants’ responses were invalid because they were incomplete or had been answered unsatisfactorily, for example, they had chosen the same answer for all the items in the
123
questionnaire. These responses were discarded to avoid their negative effect on the data analysis.
Descriptive analysis of the data
In the following section, there will be a discussion of the descriptive analysis of the pilot study, including the mean, standard deviation, and the percentage of the undecided option and missing responses. The descriptive analysis of the pilot study showed that questionnaire items had a mean score ranging from 3.9 to 5.7. It appeared that most participants agreed with most of the questionnaire items. This might be because the items have a positive sense, but this should not cause a problem when analysing the data as we can still order the items from the most important to the least important. Furthermore, previous research of L2 motivation shows similar results, though they are using a different scale. For example, the study of Cheng and Dörnyei (2007) shows that the mean range of their questionnaire items is between 3.51 and 5.7 and this indicates that participants chose one direction of the scale. It should be noted that they used a different scale from the one used in this study, as they used six response options describing the importance of each item ranging from ‘not important’ to ‘very important’. The analysis showed that the standard deviation (SD) of the items was between 0.6 and 1.8 and this illustrated a fair variability of responses, though the responses tended towards the agreement side of the scale. As for the undecided option and missing responses, the percentage of ‘I do not know’ was between 0.5% and 8.2% and the missing data percentage was from 0.5% to 1.6%. The proportion of undecided options and missing responses was not high, and therefore it can be assumed that most of the questionnaire items were clear to participants.
124
The internal consistency reliability of the scale
The internal reliability of the multi-items scale was measured using Cronbach Alpha coefficient; the aim was that the Cronbach Alpha of a scale should be above 0.70 as recommended by researchers such as Dörnyei (2007b) and DeVellis (2003). However, it is suggested that it is difficult for short scales with 3-4 items to reach 0.70 (Dörnyei, 2007b; Pallant, 2010), and therefore, Dörnyei (2003, p.112) suggests that ‘if the Cronbach Alpha of a scale does not reach 0.60, this should sound warning bells’. In the case of scales with Cronbach Alpha less than 0.70, it is suggested that the mean inter-item correlation of each scale should be between 0.2 and 0.4 to ensure the internal reliability of the scale. Briggs and Cheek (1986, p.114) assert that ‘the optimal level of homogeneity occurs when the mean inter-item correlation is in the .2 to .4 range’. As shown in the following table, the Cronbach Alpha of three scales of the instrument was more than 0.70 which is considered a good ratio. On the other hand, eight scales had a Cronbach Alpha between 0.66 and 0.51. Based on the previous argument about internal reliability, ‘the warning bells’ sound for four scales in this study whose Cronbach Alpha does not reach 0.60. Therefore, the mean of inter-item correlation for each scale was checked to ensure the internal reliability of each scale. The mean of inter-item correlation of all scales was between 0.2 and 0.4, which is a good mean.
125 Table 4.8: The internal reliability of scales
The Cronbach Alpha of the instrument’s scales No. of cases No. of items Cronbach Alpha The mean of inter-item correlation Ideal L2 self 176 4 0.74 0.4 L2 related values 156 6 0.72 0.3 Teacher behaviour 164 5 0.62 0.3 Goals 164 3 0.56 0.3 Learner autonomy 153 4 0.66 0.3 Task 169 4 0.71 0.4 Classroom atmosphere 162 7 0.53 0.2 Learner confidence 157 7 0.63 0.2 Learner group 166 4 0.64 0.3
Recognise students’ effort 133 5 0.50 0.2
The questionnaire items after the item analysis
The item analysis resulted in reducing the number of scales and questionnaire items. As for scales, they were reduced into 10 scales (Table 4.9). After piloting, two scales ‘Presenting tasks in a motivating way’ and ‘Making learning stimulating and enjoyable’ could not statistically form reliable scales, and therefore they were combined to form one scale ‘Task’ to increase its reliability. In addition, study piloting showed that the scale of ‘Teaching materials’ did not work. Therefore, it was deleted as a scale, but the scale items were added to other scales.
126
Table 4.9: The development of the questionnaire scale after piloting (MS= motivational strategy)
MS scales in the first draft of the
questionnaire MS scales after pilot study testing
Teacher behaviour Teacher behaviour
Classroom atmosphere Classroom atmosphere
Learner group Learner group
L2 related values L2 related values
Goals Goals
Recognise students’ effort Recognise students’ effort
Learner autonomy Learner autonomy
Ideal L2 self Ideal L2 self
Learner confidence Learner confidence
Presenting tasks in a motivating way
Task Making learning stimulating and enjoyable
Teaching materials
In terms of the questionnaire items, 16 items were deleted because they lowered the reliability of the questionnaire scales: 14 items were negatively worded and two were positively worded (Appendix 9). As negatively worded items appear to have a negative effect on the scale's internal reliability, it was decided that all the items in the main study questionnaire would be positively worded.
Moreover, to increase the internal reliability of the scales, eleven items were added to increase the reliability of some scales. Six items were from an instrument used in previous research by Cheng and Dörnyei (2007). The other five items were positively worded and added to the questionnaire items, as shown in the following table:
127
Table 4.10: Items added to increase the reliability of the questionnaires scales
Scale Items Source
Teacher behaviour Establish good relationship with students Positively worded items
Learner confidence
Encourage students to try harder (Cheng & Dörnyei, 2007)
Design tasks that are within the students’
ability (Cheng & Dörnyei, 2007)
Learner group
Encourage students to share personal
experiences and thoughts (Cheng & Dörnyei, 2007)
Allow students to get to know each other (Cheng & Dörnyei, 2007)
Goals State the objectives of each class Positively worded item
Classroom atmosphere
Break the routine by varying the
presentation format Positively worded item
Bring in and encourage humour. (Cheng & Dörnyei, 2007)
Recognise students’ effort
Offer rewards in a motivational manner Positively worded item
Teacher should celebrate students’
success Positively worded item
Learner autonomy Involve students in designing and
running the English course. (Cheng & Dörnyei, 2007)
In addition, to check the validity of the questionnaire and to guard against the acquiescence effect, it was decided that five negatively worded items would be included but would not be analysed. These were added to increase the validity of the questionnaire. The following table shows the number of items in each scale during the different stages of the research.
128 Table 4.11: The number of questionnaire items
Before piloting After item analysis
After adding items to increase the scales' reliability
Scale (13 scales) No. of
items Scale (10 scales)
No. of
items Scale (10 scales)
No. of items
Ideal L2 self 5 Ideal L2 self 4 Ideal L2 self 4
L2 related values 7 L2 related values 6 L2 related values 6
Teacher behaviour 6 Teacher behaviour 5 Teacher behaviour 6
Goals 5 Goals 3 Goals 4
Learner autonomy 6 Learner autonomy 4 Learner autonomy 5
Presenting tasks in a motivating way 3 Task 4 Task 4 Making learning stimulating and enjoyable 5 Classroom atmosphere 3 Classroom atmosphere 7 Classroom atmosphere 9
Learner confidence 8 Learner confidence 7 Learner confidence 9
Learner group 5 Learner group 4 Learner group 6
Rewards 3 Recognise students’
effort 5
Recognise students’
effort 7
Feedback 6
Teaching materials 3 Negative items 5
129 The validity of the scale
For the scale to be valid, a number of procedures were followed. First, the questionnaire items are not only based on the previous literature, but are also drawn from the exploratory interviews conducted to make the instrument relevant to the context of the study. Second, as mentioned earlier, the questionnaire was administered for an initial pilot study to check the wording and the translation, as well as the meaning of the questionnaire items. After this pilot, the instrument was modified as recommended. Further revision of the questionnaire items was conducted after the item analysis.
The development of the scales after piloting
Though the pilot study went through different stages and took a relatively long time, it played a key role in the development of the quantitative instrument. The questionnaire was modified as a result of the item analysis. The following is an overview of the main changes of the instrument:
The undecided response option ‘I do not know’ was deleted from the instrument because a low percentage of participants chose this answer – less than 10%. It was believed that the questionnaire items related to the participants’ context, as all the items are about teaching practices which are related to teachers and students, and therefore the ‘I do not know’ option was not felt to be needed by participants very often. Additionally, research shows that the inclusion of undecided options in surveys does not have a positive effect on data quality (Krosnick et al., 2002).
130
As a result of the pilot study, the wording and translation of items were revised, and some items were deleted while other items were added; see Appendices 9 and 10.
Some of the items were moved into another scale if it related to the content of the scale in order to increase scale reliability. For example, the item ‘provide students with positive feedback’ was moved from the scale of ‘Recognise students’ effort’ to the scale ‘Learner confidence’.
It was decided that all the questionnaire items would be positively worded, as it appeared that the negatively worded items reduced the internal reliability of the instrument. However, five randomly negatively worded items were added throughout the questionnaire to ensure the validity of the instrument and to limit the effect of the acquiescence bias. These items are:
o Teacher should avoid giving students the opportunity to socialise.
o Teacher should advise students to use English in the classroom rather than outside classroom.
o Teacher should remind students of their duties to learn English.
o Teacher should be serious-minded in the classroom.
o Teacher should be the responsible about choosing the time of tests. After these modifications, the questionnaire items were randomised again,
Appendix 11.