RESEARCH DESIGN AND METHODOLOGY

1.5.1 Evaluation research

This study is located under the broad heading of evaluation research, sometimes referred to as programme evaluation (Rossi, Lipsey, & Freeman, 2004:2). In this study the purpose of the evaluation is formative and developmental in nature (improvement-orientated) (Babbie & Mouton, 2001:338,345; Mertens, 2005:232;; Rossi et al., 2004:44) as well as summative (judgment-orientated) (Babbie & Mouton, 2001:357) and takes place while the programme is being implemented (so-called ongoing evaluation) (ibid.:xxxi).

The theoretical perspective that forms the framework in this evaluation research is an interpretive-constructivist approach with a pragmatic focus typified by the following key features: the insiders’ perspective is emphasized, the research is conducted in the natural setting of social actors, the researcher is seen as the ‘main instrument’ in the research process, the product of the inquiry is richly descriptive and the research process is primarily abductive (Creswell, 2003:4,9; Denzin & Lincoln, 2000:20; Morgan, 2007:48).

1.5.2 Study implementation

The study commenced in 2005. Phase I focused on a needs assessment, followed by Phase II the pilot study phase during 2006 whereas Phase III, IV and V focused on the case study phase during 2007. The pilot study phase focused on the programme theory (design and plan

of the Hybrid PBL approach) during Phase I and Phase II; the case study phase focused on the programme process (implementation of the Hybrid PBL approach) during Phase III; Phase 1V focused on programme outcomes (impact) and sustainability (maintenance); and Phase V focused on programme improvement. Given the evaluative nature of this study the researcher started off with a broad evaluation plan and moved it around as the research progressed.

1.5.3 Research design

This evaluation study could be framed as a single case-study (Green & McClintock, 1991:14). In this single case-study the researcher explored in depth the process of conceptualizing, designing, implementing and evaluating a Hybrid PBL approach in a single programme at the University of Stellenbosch where the researcher used purposive sampling by observing the participants involved in the SciMathUS programme during 2007 which consisted of 42 adult students (ranging between the ages of 17 and 22) and three lecturers (two mathematics lecturers and one physical science lecturer).

1.5.4 Research methodology

A mixed-method approach to programme evaluation that involved collecting and analyzing both qualitative and quantitative data (Babbie & Mouton, 2001:368; Cresswell, 2003:14; Greene & McClintock, 1991:19) was used in this study. The combination of qualitative and quantitative methods emphasized the largely pragmatic approach adhered to in this study (Morgan, 2007:48-53). The main rationale for using the mixed-method approach was to converge or confirm outcome findings from different data sources (Creswell, 2003:210) which could enhance the validity of the findings (Merriam, 2002:12).

Within this study the researcher selected a concurrent nested mixed-method strategy which seeks to elaborate on or expand the findings of one method with another method. The concurrent nested strategy within this study could be identified by its use of one data collection phase (the pre- evaluation phase at the beginning of the study), during which both quantitative and qualitative data were collected simultaneously followed by a predominantly qualitative data collection phase during the implementation process of PBL followed by a post-evaluation phase at the end of the study where both quantitative and qualitative data again were collected simultaneously. A nested approach has a predominant method that

guides the research. Given less priority, the quantitative method in this study was embedded or nested within the predominant qualitative method. This nesting meant that the embedded method sought information from different levels. The data collected from the two methods were mixed during the analysis and interpretation phase of the research (Cresswell, 2003:14).

Multiple data construction strategies were used in this study (Mertens, 2005:16) most of which resulted in qualitative data that were generated from the programme evaluation of the study whilst quantitative data were generated during the pre- and post evaluation stages of the study. Whilst the qualitative data provided rich descriptive materials the quantitative data gave more precise numerical measures. Within the mixed method approach the researcher based the inquiry on the assumption that collecting diverse types of data best provided a better understanding of the research problem (Creswell, 2003:21).

Data was collected at key points over a sustained two-year time period. Different sources of information were used. Primary data were collected through the application of the Inventory of Learning Styles (ILS) on students’ regulation of learning processes in higher education, semi-structured focus group interviews and classroom observations. The Inventory of Learning Styles (ILS) (Vermunt & Vermetten, 2004:364) was used in this study as a pre- and a post evaluation instrument. The ILS questionnaire was constructed by Jan Vermunt (2004a; 2004b) in the context of a research project at Maastricht aimed at measuring four components of student learning, whilst providing an integrative learning theory focusing on the interplay between self-regulation and external regulation of learning processes (Vermunt & Vermetten, 2004:359). Secondary data were collected through document analysis (including a literature review) and records of meetings in order to satisfy the information needs of stakeholders1 (Babbie & Mouton, 2001:76; Capeling-Alakija, Lopes, Benbouali & Diallo, 1997; Claessens & Jochems, 1993:51; Mertens, 2005:71, 390). The primary and secondary data that were collected formed the case study data base.

1 _{Stakeholders: An actor that has vested interest in a given project. In the case of SciMathUS} stakeholders included the lecturers, the learners, project managers, the funders, the researcher herself and the University of Stellenbosch (Capeling-Alakija et al., 1997).

1.5.5 Data analysis

In choosing data analysis procedures the researcher considered the evaluation questions, approaches, data collection techniques, and kinds of data collected. Data collection and data analysis occurred simultaneously as an interactive, continuous and on-going process (Cocklin, 1996:94; De Vos, Strydom, Fouché & Delport, 2002:341). The data analysis within this mixed method research occurred both within the quantitative approach and the qualitative approach, and also between the two approaches (Creswell, 2003:220). The quantitative and qualitative data generated from the evaluation research required vastly different competencies of the researcher in the analysis process. The quantitative data focusing on changes in students’ learning patterns gave precise numerical measures whereas the qualitative data provided rich descriptive materials (ibid.:222).

Analysis of the quantitative data consisted of identifying the variables that the ILS

questionnaire measured. The ILS questionnaire was used to measure the relationship between the dependent variable (Y), namely student learning patterns (in other words self-regulated learning) and the independent variable (X), namely introducing the students to the Hybrid PBL approach. The pre-post evaluation design involved observations and measurements before commencement or implementation of the PBL intervention (O1) followed by the PBL intervention (X). After completion of the PBL programme, another set of post-evaluation observations and measurements was administered (O2). Standard quantitative analyses, such as a comparison of mean scores between O1 and O2, a t-test and an analysis of variance (ANOVA) then indicated whether there was a statistic significant difference between the pre- evaluation and post-evaluation participants. The key question, namely did the programme participants change, was answered through this pre-evaluation-post-evaluation design (Babbie & Mouton, 2001: 349).

The qualitative data from the various observations made by the tutors, the researcher and students were analysed using the thematic and content analysis procedure of open coding as described in Berg (1995:185) and Frank and Barzilai (2004:46). The researcher grouped data from the text under the components and patterns (themes) of student learning identified in the ILS questionnaire (Ely, 1991:150). These patterns were given names (codes) and were refined and adjusted as the analysis proceeded (Merriam, 2002:14; Mertens, 2005:423; Miles & Huberman, 1994:9). The units of data which were coded into themes were clustered into the

relevant components (categories) identified on the ILS questionnaire. The categories reflected the purpose of the study and they were exhaustive in allowing all the data to be categorized.

1.5.6 Limitations to research design and methodology

Constraints on this envisioned programme improvement and evaluation process were:

• Programme circumstances and activities could change during the course of the evaluation. • Finding an appropriate balance between scientific and pragmatic considerations in the

evaluation design proved to be at times difficult.

• Every improvement and evaluation plan must negotiate a middle way between optimizing the situation for research purposes and minimizing the disruption caused to normal operations.

• The distinction between programme development and evaluation could become increasingly blurred (Mertens, 1998:235, Rossi et al., 2004:21-22, Terre Blanche & Durheim, 1999:224).

1.5.7 Ethical Issues

The following ethical requirements were met during this study:

• Participants were provided with the opportunity to give their consent to participate freely in the study.

• The purpose, goals and objectives and various supporting or competing agendas were openly discussed and everyone had the opportunity to influence and shape events (in the interests of transparency).

• Participants had the right to remain anonymous in documentation of the research results. • The researcher ensured that the evaluation was conducted with honesty and integrity in its

entirety.

• The evaluator respected the security, dignity, and self-worth of all the stakeholders.

• The researcher worked towards a true participatory and democratic relationship with stakeholders in a spirit of collaboration, capacity-building and co-ownership (Babbie & Mouton, 2001:359; Capeling-Alakija et al., 1997; Mertens, 2005:81; Miles and Huberman, 1994:291-292; Tuckman, 1978:16).