For the purposes of the present study, analyses were conducted on the data collected first, from the administration of the CCQ-P, SEEQ and ICT Usage Survey. Given that the administration of the surveys were administered over two days, to avoid fatigue, only data from students who were present on both days was included. This provided a sample of 574 students.
The data analyses, carried out to address the objectives of the present study, can be divided into two parts. The first part involved the validation of the three newly- developed surveys, the results of which are reported in Chapter 4. The second part involved the analysis of data to answer research objectives 2 to 6, the results for which are reported in Chapter 5. Below, Sections 3.6.1 to 3.6.4 outline the data analysis processes used to address each research objective.
3.6.1 Research Objective 1: Validation of Surveys
It is widely accepted that the need exists for a thorough validation model of instruments used for research purposes (Leye, Himmelspach, & Uhrmacher, 2009). Verification and validation involve ensuring that a survey is constructed correctly and that it behaves with satisfactory accuracy, consistent with its objectives (Balci, 2003).
102
Validation of the three surveys developed for the purposes of this study was important to provide confidence in the findings of the remaining research objectives. The validation process was guided by Trochim and Donnelly’s (2008) construct validity framework (see Figure 3.1). Construct validity ensures that the internal constructs of any model accurately reflect the purpose and intention of the survey; that is, a causal relationship exists between the construct and what is being measured (Leye et al, 2009; Teglasi, Nebbergall, & Newman, 2012).
According to Trochim and Donnelly’s (2008) framework, a construct must meet the requisites of both translation and criterion validity. The following sections describe translation validity (Section 3.6.1.1) and criterion validity (Section 3.6.1.2) in relation to Trochim and Donnelly’s (2008) construct validity framework. Based on this framework, the present study used translation validity and criterion validity to ensure that the three surveys developed were valid measurement tools.
3.6.1.1 Translation Validity
According to Trochim and Donnelly’s (2008) construct validity framework, translation validity ensures that the operationalisation of the construct (in this case, the items used in each scale of the three surveys), accurately represents its theoretical foundation and can be comprehended by the respondents. Translation validity includes two elements: content validity and face validity. Content validity “focuses on whether the construct is theoretically sound and provides an all-encompassing representation of the construct” (Velayutham et al, 2011, p. 7). In the context of this study, content validity ensured that the scales of the surveys were based on research or theoretical grounds and were appropriate for the purpose of the survey, as recommended by Li and Sireci (2013). Face validity, on the other hand, was examined to ensure that the items were interpreted by the participants in ways that were intended by the researcher.
103
Figure 3.1 Construct validity framework (Trochim & Donnelly, 2008)7
3.6.1.2 Criterion Validity
The second part of Trochim and Donnelly’s (2008) framework focuses on criterion validity. Criterion validity examines the relationships between items within a construct and focuses on whether the operationalisation of the construct provides conclusions about these relationships that are expected, based on theory. According to Trochim and Donnelly (2008), there are four elements of criterion validity. The items of a construct should correlate highly with each other (convergent validity), and the correlations between items of different constructs should be relatively low (discriminant validity). Constructs should also be able to distinguish between groups that the constructs are theoretically intended to distinguish (concurrent validity) as
7 Reproduced by permission; see Appendix 6.
Construct Validity
Translation Validity
Operationalisation is an accurate detailed definition of the theoretical construct
Criterion Validity
Operationalisation gives relational conclusions that are expected based on
theory
Content Validity
Constructs are theoretically well defined and inclusive
Convergent Validity
Items of a construct are highly correlated to each
other
Discriminant Validity
Items from different constructs are not highly
correlated to each other
Concurrent Validity Distinguishes between groups it should theoretically be able to distinguish Predictive Validity Predicts something it should theoretically predict Face Validity Items of a construct are able to reflect clearly the theoretical
104
well as to predict things that they should, theoretically, be able to predict (predictive validity).
Convergent validity assesses whether the construct items correlate highly with each other. The factor structure and internal consistency reliability were examined to confirm the convergent validity of each survey. Principal axis factoring with oblique rotation was used to check the structure of each survey. As recommended by Pallant (2011), oblique rotation was utilised due to the overlapping nature of the learning environment dimensions. As recommended by Field (2009) and Thompson (2004), two criteria were used for retaining any item. First, the item must have a factor loading of at least .40 on its own scale, and, second, it must have a loading of less than .40 on all of the other scales. Cronbach’s alpha coefficient was used as an index of internal consistency reliability to assess whether the items within the same scale assessed the same construct. Two units of analysis were used to assess internal consistency reliability, namely, the individual and class means.
To confirm discriminant validity, the items of different constructs should not correlate highly with each other. An intercorrelation matrix generated during oblique rotation, as recommended by Brown (2014) and Field (2009), was used in the present study to provide evidence to support the discriminant validity of the survey scales.
To support concurrent validity, a given construct should be able to distinguish between groups that it is theoretically intended to distinguish (Trochim & Donnelly, 2008). In theory, students within the same classroom should have somewhat comparable perceptions of their learning environment whereas the perceptions of students from different classes should differ (Aldridge & Galos, 2017). Therefore, to examine the concurrent validity of each survey, an ANOVA was calculated for each scale, with class membership as the independent variable.
Finally, predictive validity focuses on the extent to which a given construct can predict something which it should, theoretically, be able to predict (Trochim & Donnelly, 2008). To provide evidence to support the predictive validity of each instrument in the present study, simple correlation was used.
105
3.6.2 Research Objective 2 – Differences between Actual and Preferred Learning Environment Perceptions
The second research objective sought to examine whether differences existed between primary school students’ actual and preferred perceptions of their learning environment and the extent of ICT usage within the classroom. As a first step, the average item mean and average item standard deviations were calculated separately for the actual and preferred responses for each scale. To examine whether these actual—preferred differences were statistically significant, Wilks’ Lambda (Wilks, 1935), was examined; the results of this examination led to the interpretation of the MANOVA for each scale. The scales of the CCQ-P and the ICT Usage Survey were used as the independent variables and students’ actual and preferred responses were used as the dependent variables. Finally, to examine the magnitude of the differences between students’ responses to the actual and preferred versions of each scale, the effect sizes were calculated (as recommended by Thompson, 2001).
3.6.3 Research Objectives 3 and 4 – Associations between the Learning Environment, Use of ICT, and Student Outcomes
The third and fourth research objectives sought to examine whether relationships existed between the affective outcomes of self-efficacy, enjoyment of class, and enjoyment of using ICT and (a) students’ perceptions of the learning environment (research objective 3) and (b) their perceived use of ICT within the classroom environment (research objective 4). Data analyses were conducted on the sample of 574 students (described in Section 3.4.3) using the actual responses. Simple correlation analysis was used to examine the bivariate relationships between each of the three outcome scales (from the SEEQ) and the CCQ-P and ICT Usage Survey scales. Multiple regression analyses (R) were used to determine the joint influence of the set of SEEQ scales (as independent variables) and the individual CCQ-P and ICT Usage Survey scales (as dependent variables), using the class mean as the units of analysis. To identify which of the CCQ-P and ICT Usage Survey scales contributed uniquely and significantly to the explanation of the variance in students’ self-efficacy and enjoyment (of class and use of ICT), standardised regression coefficients (β) were examined.
106
3.6.4 Research Objective 5 and 6 – Differences in Perceptions and Outcomes between Groups of Students
The fifth research objective sought to examine whether perceptions of the learning environment, ICT usage and outcomes (self-efficacy, enjoyment of class, and enjoyment of using ICT) differed for students of different gender. Similarly, the sixth research objective sought to examine whether perceptions of the learning environment, ICT usage and outcomes (self-efficacy, enjoyment of class, and enjoyment of using ICT) differed for academically at-risk and not-at-risk students.
Data analyses were conducted on the sample of 574 students using the actual responses. To examine whether differences existed between the perceptions of these groups of students, MANOVA was once again utilised. Separate MANOVA analyses were conducted for all three instruments (CCQ-P, ICT Usage Survey, and SEEQ) using the scales of each survey as the dependent variables and students’ gender or at- risk status as the independent variable. Multivariate tests using Wilk’s Lambda criterion were examined and, as a result, the univariate one-way ANOVA results were interpreted for each scale.
As there were different numbers of at risk / not-at-risk students within each class, the class mean was used as the unit of analysis. The one-way ANOVA provided an F value that compared the variability between groups to the variability within groups (Laerd Statistics, n.d.). The p value (or probability) of finding an F ratio as large as the one calculated by the one-way ANOVA was used to either reject or accept the null hypothesis, that is, that no differences exist in population means between the groups. Effect sizes were calculated to determine the magnitude of the differences between the scores of male and female students and between the scores of at-risk students compared with those students who were not at risk (as recommended by Thompson, 2001). Effect sizes were expressed in standard deviation units.
Given that at-risk and not-at-risk students reported different experiences of the learning environment, an analysis of covariance (ANCOVA) was used to examine these differences. The use of an ANCOVA allowed the preferred scores on the learning environment scales to be referenced against the actual scores and then
107
compared between the two groups of students (at risk and not at risk). This approach allowed for a comparison of the groups of students’ preferred scores. In this analysis, responses to the preferred version were used as the dependent variables, the corresponding responses to the actual version were the covariates, and the student type (at risk or not at risk) was the independent variable.
The results of the data analyses described in this section are reported in Chapters 4 and 5. Chapter 4 outlines the results that provide support for the validity and reliability of the three new surveys (research objective 1) and Chapter 5 outlines the results of the data analyses to address the remaining five research objectives.