Creating data collection tools

The data were collected by using the questionnaires that were developed based on the Critical Thinking Diagnostic Questionnaire (CTDQ) (Weiss, 2010) intended for measuring the college students’ perceptions of their HOTS. It was administered in the form of a mixed design that integrated 24 self-report, Likert style items and a number of qualitative questions. The CTDQ was elaborated by analysing a large number of the instruments intended for measuring critical thinking skills and dispositions and related to the two main domains of thinking skills: cognitive and affective (Thayer-Bacon, 2000; Ennis, 1990; Paul, 1990; Alvino, 1990; Bloom, 1956). In the study of Weiss (2010), it was administered to the students of an Arab college in Israel and was found to have high reliability: 0.874.

It was decided, therefore, that the CTDQ could serve as the basis for developing the new instruments in order to measure the perceptions of HOTS of teachers and the students in an Israeli Arab high school. Self-report questionnaires are frequently used in education research because of their utility (Cottrell & McKenzie, 2010; Wilcox, 2005; Mayer, 1999). Their disadvantage is that there might be reliability and validity problems, as participants may deceive themselves or others. A self-report questionnaire may actually measure respondents’ perceptions of their instruction or learning practices instead of actual behaviour during lessons (Cottrell & McKenzie, 2010; Wilcox,

68

2005; Mayer, 1999). These problems can be countered through the careful design and application of self-report measures which are addressed further in this section.

Two questionnaires were developed: one for teachers and another for students. The former includes twenty five closed items reflecting teachers’ perceptions of the methods they use for developing the HOTS of students. The latter contains twenty six closed items related to students’ perceptions of their cognitive skills and thinking dispositions. In the second part of the questionnaire, students are asked to provide written responses to four items (see Appendix 3, part A). These items refer to the level of metacognition in thinking (statement 2), problem-solving abilities (6), transfer of thinking skills (11), and teamwork abilities (15). Respondents rated the items by using a six-point Likert scale, ranging from 1 = ‘strongly disagree’ to 6 = ‘strongly agree’. Each participant was required to rate each item on the same response scale. Items in both scales were written in a clear manner to ensure that respondents fully understand the questions. The use of both positively and negatively worded items was intended to offset any affirmation/negation response bias (Fisher, 2004; White et al., 1993).

As to the second part of the questionnaire for students, it was considered an open-ended component of the questionnaire format (addressing open-ended questions). Drawing upon available empirical evidence, Ku (2009) recommends using instruments of mixed item format, both multiple-choice and open-ended, which will allow for a more complete representation of both the cognitive and dispositional aspects of HOTS. Such a format may help obtain the unique views of respondents and enhance the questionnaire’s internal reliability by clearing up possible misunderstandings and validating the quantitative survey (Boynton & Greenhalgh, 2004; O’Cathain &Thomas, 2004; Currall & Towler, 2003). It has been recommended that researchers determine the status of open- ended questions at the design stage of a study. A strategy can be used of treating them as a source of qualitative data, although, the depth of answers to open-ended questions tend to be more limited than with almost any other method of research (O’Cathain &Thomas, 2004; Bailey, 1994). In the current study, it was decided to conduct a preliminary analysis that involved reading the responses and considering whether they contributed to answering research questions (O’Cathain &Thomas, 2004; Boynton, 2004). Open-ended questions were included only into the questionnaire for students due to the amount of time needed for the analysis of a large amount of qualitative data.

69

A number of steps were taken to ensure the construct validity of the quantitative part of both tools. An analysis was performed of a wide body of research into students’ cognitive skills and thinking dispositions and pedagogical methods for developing these qualities in students, as can be seen in Appendix 4 (parts A and B). It was decided to enlarge the body of literature used for the construct conceptualization in the CTDQ. More attention is paid to the items that were considered important by the HOTS programme developers. These refer to students’ perceptions of their reasoning and problem solving skills, their abilities to establish connections between the previously obtained and new knowledge (metacognition), and application of knowledge in daily life (transferability of knowledge and skills) (Dean & Kuhn, 2003; Halpern, 2001; Csapo, 1999). In comparison with the CTDQ, the following items were added in accordance with the HOTS programme. As the programme promotes the constructivist approach to learning, the issue of solving complex problems was addressed. In addition, the HOTS intervention programme deals with the development of HOTS of low-achieving learners and the increase in social responsibility of students. It was concluded, therefore, to include these issues into the questionnaire. In both questionnaires, the area of thinking dispositions is enlarged by including the items referring to positive emotions that are expected to increase students’ learning motivation and responsibility towards school and community (Bailin, 2006; Hofstede & Hofstede, 2005; Goleman, 1995).

To assess content validity, researcher’s colleagues (a group of ‘critical friends’), who were well- acquainted with the field of HOTS, rated each item for its consistency with these areas and helped to decide whether the content of items pool are relevant and representative. The next phase of construct validation involved examining items for internal consistency. Cronbach’s coefficient alpha was used as the most commonly employed statistic used to measure internal consistency (Gleim & Gleim, 2003). The questionnaire for teachers was pre-tested with the help of friends - the Arab high school teachers with whom the researcher was acquainted through the internet forums and teacher seminars and workshops. Those teachers engaged some of their students to test the questionnaire for students. Together with ‘critical friends’, the researcher analysed the data and improved the questionnaire according to the remarks and comments received from the pilot. As to the results of the Cronbach's coefficient alpha test, Alpha levels below 0.7 are considered

70

acceptable (DeVellis, 1991). The results of testing show that both questionnaires have good internal reliability: all α-Cronbach scores range from 0.60 – 0.79 (see tables 3.2 and 3.3).

Table 3.2. The results of Alpha Cronbach coefficient test for the questionnaire for teachers

Variables Questionnaire items Likert scale Alpha Cronbach

Pedagogical methods for instilling HOTS in students.

1 - 14 1 - 6 0.757

Pedagogical methods for developing dispositions to HOT.

15 - 25 1 - 6 0.703

Total 1 - 25 1 - 6 0.789

Table 3.3. The results of Alpha Cronbach coefficient test for the questionnaire for students

Variables Questionnaire items Likert scale Alpha Cronbach

Students’ thinking skills 1 - 11 1 - 6 0.690

Students’ thinking dispositions 12 - 26 1 - 6 0.711

Total 1 - 26 1 - 6 0.794