F ORMATIVE EVALUATION , CURRICULUM CONSISTENCY AND RESEARCH RIGOUR

Formative evaluation is conducted (amongst others) "to inform the decision- making process during the design, production and implementation …. of an educational program with the purpose of improving the program (Flagg, 1990, p. 23). The evaluation methods during a design, development and implementation process differ, depending on the 'design rationale' (Maslowski & Visscher, 1999). Visscher-Voerman et al.(1999) describe four design approaches or 'rationales' (see section 4.3.2) that can be related to formative evaluation approaches (Maslowski & Visscher, 1999). The question what design approaches were used in case of the development process for the Faculty of Education will be addressed through the reconstruction and analysis of the design, development and implementation process (see chapters 6, 7 and 8).

In a deliberative design approach (cf. Walker, 1990) negotiation between stakeholders is thought crucial, characterised by continuous deliberation of means and ends, possibilities and alternatives (Schwab, 1970). After consensus on the outcomes of the program, formative evaluation may be used to compare the preliminary program and intended outcomes. Quality is perceived by the stakeholders as 'extrinsic quality' or 'worth', comparable to external consistency as described by Kessels (1999b), and evaluation serves to assess the 'context- determined' value of the programme (Maslowski & Visscher, 1999) or the congruence between the real and intended outcomes of the programme. The designer should ensure intrinsic quality, that is, the 'merit' of a programme or its 'context-free value', complying with professional standards. Evaluation is described as assessing the degree of fit of aspects of the programme with the design criteria as agreed upon by the stakeholders.

Kessels (1999b) describes quality in corporate education with help of the concept of curriculum consistency and states that this concept leads to an integrated approach to curriculum design that may also be beneficial to general education. He introduces internal consistency as the logical relationships between the elements of a curriculum, such as aims of the curriculum, competencies of students, learning environments and output or outcomes. This is comparable to the ideas of internal quality (Maslowski & Visscher, 1999) and alignment of the curriculum (Biggs, 2002). Internal consistency is best served by a 'systematic approach' to curriculum design (Kessels, 1999b). External consistency involves a coherence of perceptions of all stakeholders of what 'the problem is' and 'how it will be resolved' (Kessels, 1999b). External consistency can be reached through a 'relational approach' (Kessel 1999b), corresponding to the deliberative approach of Walker (1990) named the 'communicative paradigm' by Visscher-Voerman et al. (1999).

Research design

157

Formative evaluation has taken place throughout the design, development and implementation of the curriculum in the Faculty of Education. It involved both staff and students.

The arguments between the quantitative and qualitative research paradigms concerning rigour, validity, consistency and truth have also involved development research. Van den Akker (1999) mentions the "…apparent lack of rigour and control in methodology" (p. 11). Design and development processes have a dynamic character that sometimes hampers controlled experimentation and leads to research designs that are emerging during the design and development process (cf. Thijs, 1999). Development research could be placed under what Creswell (2003) calls 'mixed methods research', that is, using a mix of quantitative and qualitative methods. The related knowledge claims are, according to Cresswell (2003) pragmatic knowledge claims arising " …out of actions, situations, and consequences rather than antecendent conditions (as in post-positivism)" (p. 11). Shavelson, Phillips, Towne and Feuer (2003) mention the recommendations of a US National Research Council committee that concluded that:

" …design studies are creative endeavours that hold promise especially for developing and elaborating conjectures and addressing issues of research-based practice; and

….principles of research that apply to all scientific work must apply also to knowledge claims from design studies' (p. 26)

They emphasise the importance of a comprehensive record of the evolving design process (cf. Van den Akker, 1999). Three generic research questions should drive the choice of research methods (Shavelson et al., 2003):

1. What is happening? This involves a rich description using qualitative and quantitative methods.

2. Is there a systematic effect? Shavelson et al. (2003) recommend the use of randomised experiments, when feasible. "The use of experimental studies combined with case (and other) studies of implementation seems appropriate to test the generalisability (and limits) of the effects" (p.28). In development research the use of multiple, randomised experiments is often not possible and Shavelson et al. seem here too rigorous in their attempts to put all (educational) research on a strictly scientific footing. In development research evaluation is done against the criteria or desired outcomes and generalisation is more directed towards usability and practicality than towards unravelling 'the Truth'. 3. Why or how is it happening? Here Shavelson et al. (2003) recommend the use of

Shavelson et al. are referring to design studies in the area of instruction technology with an important role for formative research. In this study of the design, development and implementation of a competence-based curriculum for the Faculty of Education the reconstructive aspect is reflected in Shavelson's generic question one. Shavelson's second question is, as argued above, less relevant for this study because in the reconstructive research approach of this study comparison with other, similar, programmes is not aimed for. The case study approach leads to a generalisation to theory -analytic generalisation- rather than to populations or universes -statistical generalisation- (Yin, 1994). The question of a systematic effect could also be interpreted as a question about the impact of a programme. Can the impact be causally explained by the programme activities and (immediate) outcomes, or are other factors playing a role? In the third question the importance of a formative approach is emphasised in various cycles, until the product has the required characteristics. In case of educational programmes something similar can be done through a systematic, continuous formative evaluation of curriculum elements, e.g. courses or modules, or aspects (e.g. the integration of generic competencies) and the revision of these elements and aspects. This implies a flexible curriculum that can change from year to year as a result of the formative evaluations. The 'why or how is it happening' question is also the guiding question in the analysis and reflection on the reconstructed curriculum development process in case of this study. Technical-professional and socio-political 'lenses' are used for this analysis and reflection (see Section 5.7).

The use of various data sources and various methods in collecting data signified triangulation through "multiple occurences or representations of the processes being studied" or "a different picture and slice of reality" (Denzin, 1994, p. 6464- 6465). This means that the perception of the researcher was shaped by multiple sources of information representing multiple perspectives, thus preventing bias in interpreting the data during the reconstruction and analysis of the curriculum development process. Triangulation, in this sense, does not primarily aim for convergence or completeness (cf. Knafl & Breitmayer, 1989; Massey, 1999). Data triangulation is especially important to avoid a bias in results and conclusions because of interference of the different roles of researcher, designer and member of staff. Therefore, data have been collected in as many ways as possible. A research diary during the implementation of the curriculum also served to keep awareness of which activities could be attributed to which role. A further reflection on the different roles of the researcher is given in Section 9.2.2.

Research design

159