Quantitative data

This section describes the instruments used to gather the quantitative data used in this study.

3.5.1.1 Raven’s Advanced Progressive Matrices (APM)

Measures of eductive ability, or otherwise known as Spearman g, lends support in answering Research Questions 1, 2 and 3 in Chapter 2 in terms of how deep learning and critical thinking may vary amongst learners with different levels of eductive ability within the CoLeCTTE framework.

Eductive ability, which was defined previously as a general measure of a person’s aptitude for complex problem solving was measured using Raven’s Advanced Progressive Matrices (APM) Sets I and II. It is used as a “well-validated measure of basic cognitive functioning) and used for different cultural, ethnic, and socio-economic groups on a worldwide and within country basis” (Raven, 2000).

APM requires persistence, logic and attention to detail and has been proven to test such things as success at computer programming and spreads the scores of the top 20% of the population (Raven, Raven, & Court, Advanced progressive matrices, 1998). As mentioned in Chapter 2, Raven continues to point out that measures of eductive ability is not a measure of intelligence, rather it measures the potential or aptitude to perform well in complex tasks. It is an individual measure of mental ability for analysing and solving problems, abstract reasoning, and their learning potential. It is a non-verbal measure which reduces any form of cultural bias. Based on this, the premise is that students with low eductive ability may or may not

perform well with problem-based learning tasks and activities that require complex critical thinking compared to students with higher eductive abilities, especially when they have to do multi-tasking such as using technology in context. So technology may or may not play a delimiting or facilitating role in the learning process.

Participants from each of the case groups were asked to complete APM Set I as a practice set and then APM Set II as the data set. The APM test was used in this study because of the professional licensing requirements to conduct the Watson- Glaser Test, and as discussed in Section 2.3.2, these two tests are significantly correlated. An alternative to Raven’s APM eductive ability test is to measure performance aptitude using the Wechsler Adult Intelligence Scale (House, 1996). The verbal IQ and performance IQ components of the WAIS-R have significant correlations with APM at .42 and .55, respectively. However, WAIS-R requires a trained psychologist to conduct the test and the researcher did not meet that criterion.

Individual raw scores of eductive ability in this study were converted into percentile scores based on the late 80s Australian normative values (1998a, p. APM96) and classified according to Raven, Raven and Court’s (1998a, p. APM72)

definitions. A summary of these classifications is provided in Appendix D.

Interpretations of APM results are further discussed in the analysis Chapters 4, 5 and 6. Identifying low or high eductive abilities amongst the participants assisted with analysis of outliers.

3.5.1.2 Online Technologies Self-efficacy Scale (OTSES)

The Online Technologies Self-efficacy Scale (OTSES) questionnaire developed by Miltiadou and Yu (1999) was used unchanged in this study to

determine technological self-efficacy, with questions added to acquire demographic information from the participants (see Appendix E). It measures self-efficacy with four technology types: Internet; Synchronous (i.e., chatting "live" via a synchronous chat system such as CourseInfo, First Class, NetMeeting, or IRC); Asynchronous I (i.e., using an e-mail system such as Pine, Netscape Mail, or Outlook; and

Asynchronous II (i.e., posting a message to a newsgroup, a bulletin board, or on the

discussion board of a conferencing system such as CourseInfo, FirstClass, etc. where participants are not online at the same time). Miltiadou and Yu (1999) reported that content validity, construct validity, and reliability were established through factor analysis and correlational analysis and revealed that all items could be collapsed into one scale with a Cronbach coefficient alpha (measure of internal consistency or reliability) for the whole instrument at 0.95. This questionnaire lends support in answering Research Questions 2 and 3 in Chapter 1 in terms of how deep learning and critical thinking may be affected by a learner’s perception of their self-efficacy within the CoLeCTTE framework.

3.5.1.3 Study Process Questionnaire (SPQ)

The Study Process Questionnaire (SPQ) was used in this study to determine a learner’s predisposition or approach to learning and motivation, and lends support in answering Research Questions 1, 2 and 3 in Chapter 1 in terms of how cognitive learning styles may influence deep learning and critical thinking performance within the CoLeCTTE framework.

Since the CoLeCTTE framework was based on socio-cognitive principles as discussed in Section 2.3.3, Bigg’s Learning and Study Process Questionnaires were considered for use in this study. The Learning Process Questionnaire (LPQ) (Biggs, 1993) consists of 42 questions to measure strategies to learning and motivation for learning. There are three strategies to learning and three types of motivation: surface, deep and achieving. Seven questions are dedicated to each type of strategy and motivation. However, the wording of the LPQ items is appropriate for a school setting (Kember, Charlesworth, Davies, McKay, & Stott, 1994). For this reason, the Study Process Questionnaire (SPQ) (Biggs, 1987) which also contains the achieving and motivation components and which has been designed for use in higher

education, was used in this study instead of the LPQ.

There are two dimensions to learning approaches in the SPQ, strategy (S) and motive (M), both of which can exist to varying degrees in a surface (S), deep (D) or achieving (A) learning approach. Results of the SPQ are calculated as individual raw scores (RS) and equivalent deciles (Dec) subscales of surface motive and strategy (SM and SS), deep motive and strategy (DM and DS) and achieving motive and strategy (AM and AS) as shown in the sample SPQ record form in Appendix F. These are drawn from the normative standards appropriate for the male and female participants in this study, Tables 9 and 10 of the SPQ Manual, respectively (see

Appendix G). These raw scores and deciles are further combined into scales of surface approach (SA), deep approach (DA) and achieving approach (AA) as shown in the sample SPQ record form in Appendix F. The raw score and decile equivalents are used to define the learning style profile which represents a learner’s general orientation towards learning. A deep approach (DA) is considered to be the ideal model for learning. It is associated with high levels of intrinsic motivation, pursuing new ideas and materials and the use of a variety of strategies in the search for

understanding. A surface approach (SA) is associated with minimal efforts to avoid failure, a focus on assessment requirements and strategies limited to rote learning or memorisation. An achieving approach (AA) is focused on the processes to achieve high grades rather than an intrinsic motivation to learn in-depth, therefore,

competition is a key motivator (Entwistle N. , 1981; Entwistle, Hanley, & Hounsel, 1979).

The sample SPQ record form (Appendix F) shows how deciles are used to assist teachers to judge a student’s raw score in broad terms and are translated into +, 0 and – symbols to draw up a learning approach profile and classification which are always given the following order: SA, DA and AA scales, and SM, SS, DM, DS, AM and AS subscales (see Appendix F). The resulting profiles give an indication of the strengths and weaknesses in terms of a learner’s approach and motivation to learning, and therefore, will assist in determining the type of instructional

intervention that might be recommended. Achieving a positive profile on the deep and achieving scales and subscales is widely interpreted as an indication of good approaches to learning. Furthermore, by getting a composite of the deep and achieving approach scales (DA + AA), the deep achieving approach (DAA) is determined and as these characteristics are combined, learners with a high DAA are

usually high performers who will usually require little or no learning support and counselling.

Table 3.1 summarises the SPQ‘s internal consistency or reliability measures amongst Australian students based on Cronbach’s coefficient alpha estimates as reported by Watkins (1998), Biggs (Kember & Leung, 1998), and Hattie and Watkins (1981), O’Neil and Child (1984), and comparatively analysed by

Richardson and Newby (2006). Although it would seem that alpha coefficients of less than 0.7 (as seen for the surface strategy, surface motive and deep motive estimates) may not be considered highly reliable, however, goodness of fit measures for all subscales are considered satisfactory. Goodness of fit determines the

discrepancy (0.05 residual value is standard) between the observed and expected values, so with a well-fitting model, these will be close to zero and evenly distributed among all observed variables (Byrne, 2001). Biggs (1993) used LISREL (linear structural relations) generalised least squares model to measure goodness of fit and found that all subscales had estimates greater than 0.97 (the convention is to accept higher than 0.95 as satisfactory), however, the root mean square residuals

(differences between individual values) were 0.072 for Surface Motive (0.072) and 0.044 for Deep Motive.

In addition, Kember and Leung (1998) conducted confirmatory factor analysis (the extent to which the items can actually measure a particular dimension in the instrument) and reported that the SPQ had the best fit with two- and three- factor models (Burnett & Dart, 2000). In the two-factor model, deep/achieving and surface with strategy and motive subscales are matched to meaning and reproduction learning orientations (comparative fit index of 0.987). In the latter three-factor model where covariance between surface and achieving approach is considered, best

fit (comparative fit index of 0.822) was also found, that is, surface learners may also have an achieving approach. A higher order factor was also considered by

combining deep and achieving scales. These results align with Biggs’ clarification that interpretation of SPQ measures were relational, non-orthogonal (not taken as independent measures) and contextual.

Table 3.1

Cronbach Coefficient Alpha Estimates of Internal Consistency Reliability

Dimension Watkins (1998)

Kember and Leung (1998)

Hattie and Watkins (1981), O’Neil and Child (1984) in Richardson and Newby (2006)

Surface Strategy 0.25 to 0.66 (median of 0.55) 0.57 0.66 Surface Motive 0.37 to 0.67 (median of 0.55) 0.60 0.61 Deep Strategy 0.47 to 0.76 (median of 0.69) 0.71 0.75 Deep Motive 0.44 to 0.70 (median of 0.64) 0.63 0.65 Achieving strategy 0.56 to 0.77 (median of 0.72) 0.74 0.77 Achieving Motive 0.48 to 0.77 (median of 0.68) 0.71 0.72

In 2001, Biggs, Kember and Leung (2001) developed a revised two-factor version of the Study Process Questionnaire (R-SPQ-2F) in light of major insights relating to higher education (i.e., student population, merging of disciplines) and its use as a classroom evaluation tool (i.e., accountability and role of teachers in evaluating student learning) and was found to be reliable using confirmatory factor analysis. Furthermore, population characteristics, attitudes, age, use of words in the

questionnaire to fit the respondents and response bias have provided reason to question the content validity of learning process questionnaires (Biggs, 1987; Richardson J. T., 2004; Zeegers, 2001). However, the shortened 18-item SPQ was stable and consistent with the 42-item SPQ based on a longitudinal study of British students to determine predictive validity and found to be useful for measuring large sample groups (Fox, McManus, & Winder, 2001). Richardson (2004) also

emphasised the need to interpret within the context from which the measurement was used while Turner (2004) recommends the use of follow-up investigations such as interviews to clarify results, particularly in the case of outliers. Importantly,

numerous studies across a number of contexts have established reliability of the SPQ for the measures intended in this study, including its appropriateness for evaluating innovations in technology, and predicting academic performance based on

correlations with deep learning (Burnett & Dart, 2000; Kember, Charlesworth, Davies, McKay, & Stott, 1994; Snelgrove & Slater, 2003; Zeegers, 1999).

Based on the review of reliability and validity studies undertaken above the researcher was confident that the 42-item SPQ could be used for the purposes of this study. Internal validity was maintained because this study was comprised of small numbers of participants per class; the groups were homogenous and investigated within their own contextual discipline and learning environments; and semi-

structured interviews were conducted in Case Groups 2 and 3 to gain more in-depth data. The investigation was also based on a comprehensive framework which was tested through within-group and within-class settings for Cases 1, 2 and 3.