Formative evaluation

The vast range of data collected during the iterative design and enactments of the ePlay MakerSpace process, posed a potential challenge with regards to critical and meaningful analysis. Scholars such as van den Akker (1999) and Parsons et al. (2013, 2016) propose the use of formative evaluation within research to evaluate iterations in a consistent and standardized manner. Subsequently, formative evaluation is included as a research tool to

97

holistically evaluate each of the e-Play MakerSpace enactments from social and technical perspectives.

Formative evaluation allows the researcher-designer (and participants where appropriate) to gauge social and technical progress in the research and intervention process (van den Akker, 1999). Socially it provides insight into participants’ satisfaction, enthusiasm, their commitment to and ownership of the intervention process (van den Akker, 1999).

Technically, formative evaluation gauges whether the intervention is on track and/or what changes should be made to achieve the intended outcomes (Haslam, 2010). Formative evaluation also exposes potential implementation problems that may cause frustration, leading to redesign and refinement in order to reduce these problems (van den Akker, 1999). In addition, formative evaluation determines whether the different forms of learning are taking place as planned and what (if any) new knowledge and skills teachers are

acquiring and how they applied this in their classrooms (Haslam, 2010). Formative

evaluation may also shed light on any related changes to the school culture or organisation that may result from teachers’ participation in the intervention (Haslam, 2010), or that may be required to facilitate change to pedagogy. Therefore, formative evaluation is a key research activity to inform the iterative refinement and elaboration of intervention designs if these are to have sustained, widespread real-world impact.

The primary purpose of formative evaluation is quality improvement (van den Akker, 1999). Quality is emphasized during the research in terms of validity, practicality and relevance, sustainability and effectiveness (van den Akker, 1999; van den Akker et al., 2006). Improving content and construct validity improves the quality of the intervention: whereas content validity refers to “the extent that the design of the intervention is based on state-of-the-art knowledge”, construct validity refers to the extent to which “the various components of the intervention are consistently linked to each other” (van den Akker, 1999, p. 10). Achieving quality in terms of practicality refers to the extent to which users (and external experts) find the intervention usable and appealing (van den Akker, 1999) and whether the intervention is realistically usable in everyday, naturally occurring test-bed contexts (Cobb et al., 2003; van den Akker et al., 2006). The design should be relevant to those in the target setting and

98

address participants’ needs and wants, and its use should be sustainable over time (van den Akker et al., 2006). Achieving quality in terms of effectiveness refers to the extent to which experiences and outcomes of the intervention are aligned with the intended goals and aims (van den Akker, 1999) and whether the time, effort and resources expended in the

intervention is worth the investment (van den Akker et al., 2006).

The use of a standardised formative evaluation tool serves to improve the quality and validity of the research, as well as the practicality, effectiveness, relevance and sustainability of the design. The formative evaluation foregrounds shortcomings, implementation

problems, and/or design or procedural concerns. Results from the formative evaluation can then inform the generation of alternative solutions or the refinements of existing designs to improve weaknesses (van den Akker, 1999). A formative evaluation process was

subsequently developed to evaluate e-Play MakerSpace iterations.

Design-based research is situated in unique contexts and is dependent on the participation and engagement of individuals in particular roles as they work to develop and enact the design intervention. McKinney et al. (2006) frame these areas for evaluation as context, participants and intervention design, concurring with Edelson (2002, 2006). Each of the areas illuminate particular aspects of the design that need to be evaluated.

3.2.4.1

Formative evaluation of the context of the intervention

In order to produce warranted theory, the DBR researcher has to develop a thorough and in-depth contextual understanding (van den Akker et al., 2006) of the context of an intervention. The context of DBR interventions are characteristically situated within

authentic natural test-bed education settings (Cobb et al., 2003; van den Akker et al., 2006). Viewing each context as unique and situated within a wider system of fields, DBR projects work to understand how and why an innovation works within a particular context (Design- Based Research Collective, 2003).

Contextual factors shape the goals of each design. This includes local factors such as the school culture and climate, learners’ profiles and access to resources, as well as systemic

99

factors like policies, regulations and curricular demands (van den Akker et al., 2006). The design context is subsequently elaborated by identifying challenges and constraints in the test-bed context, as well as identifying opportunities that may be exploited therein (Edelson, 2002).

A myriad of unpredictable local and systemic factors may potentially change in the course of the intervention’s iteration cycles. Institutional changes, learners’ interests and

circumstances, access to resources, or the school culture may change and positively or negatively impact the implementation. To develop a holistic understanding of the context such changes should ideally be documented and included in the formative evaluation of each implementation cycle; as well as decisions made to address any problem or exploit any opportunities that this may bring about in subsequent design(s).

Collaborating with different institutions, teachers and learners across varying contexts, and often different languages and cultures, may potentially pose logistical challenges to the research. The DBR researcher may need to make “trade-offs between refinement of a particular innovation to maximize success and generalization of findings from an ultimately highly refined enactment” (Design-Based Research Collective, 2003, p. 7). Trade-offs need to be recorded and included in the formative evaluation.

The following framework was developed to evaluate the context of each e-Play Makerspace enactment:

• Identify systemic level factors that may impact educators’ participation, implementation and/or application of newly learnt skills and knowledge.

• Identify institutional level factors such as resources, culture, leadership or value afforded ET integration, that may impact how participants implement and/or apply newly learnt skills and knowledge.

• Identify classroom level factors that may impact how participants implement and/or apply newly learnt skills and knowledge.

100

3.2.4.2

Formative evaluation of the participants in an intervention

Participants’ insights at the different levels and stages of the DBR study augment those of the researcher-designer (van den Akker et al., 2006), being central partners in the

intervention process and implementation. The extent to which teachers may choose to participate in the DBR varies, and while some may start enthusiastically and end tired and not-involved, others may start the process slowly but become deeply involved. The varying types of participation may be influenced by participants’ beliefs and attitudes towards the intervention, for instance, whether they believe it has benefit to them and/or their

students, or whether the learning enacted in the intervention resonates with their own approaches to learning.

It is essential to the success of DBR projects that a collaborative relationship is developed and nurtured between the researcher-designer(s) and participants. A collaborative relationship may be achieved by creating situations where teachers mutually benefit, whether emotionally or strategically (van den Akker et al., 2006). If teachers enjoy what they do, the emotional reward for all stakeholders is likely to encourage collaboration. Equally, if teachers derive benefit from the intervention and specifically the data collection, this may further enhance collaboration. The building of trust is vital to the development of a collaborative relationship (van den Akker et al., 2006). Trust starts with the researcher- designer building empathy with participants’ contextual circumstances, their goals and problems, and actively engaging all participants in the various aspects of DBR for

collaboration to be democratic and meaningful. In turn, this may transfer to participants taking ownership of the process, being committed to its outcomes and enthusiastically working towards achieving its targets. Participants’ satisfaction with the process and its outcomes therefore become an important indicator of these factors.

The following framework was developed to evaluate the participation of participants in ePlay MakerSpace enactments according to the extent to which they:

101

b) Actively engage in the process and in building / making structures;

c) Are committed and enthusiastic about achieving the outcomes (ET Integration and changing their practices);

d) Are empowered by their participation in the process, like taking responsibility for their own learning and reflexivity, their willingness to learn and be guided

(reciprocity), and collaborating in relational learning in small and large group interactions;

e) Display creativity, like developing original ideas and/or behaviours, their flexibility in generating novel ways to use tools they were exposed to and/or fluency in

identifying or developing relevant ideas;

f) Engaged in the reflective processes, interrogating their own subjective principles and/or creating space for objective evaluation of their own and others’ world views. Adhering to the convention of comparability (1956), the various elements of the formative evaluation were described in a rubric to standardised evaluation. Additionally, the formative evaluation rubrics limited researcher bias and increased interpretive validity. The formative evaluation tool as included as Appendix C.

3.2.4.3

Formative evaluation of the ePlay MakerSpace design

The design of each iteration of the ePlay MakerSpace, needs to be formatively evaluated to ascertain to what degree the design practically and effectively addresses the requirements and goals it set out to achieve.

A formative evaluation for the design of each iteration was consequently developed by combining two types of evaluation:

• Solution design: the design of the ePlay MakerSpace itself

• Implementation procedure: the way in which the ePlay MakerSpace process unfolded and/or how the design was implemented

102

Criteria for the formative evaluation of the Solution Design draws from van den Akker’s (1999; 2006) work and is aimed at improving the quality of the design. This relates to the practicality, relevance and sustainability of the design, as well as its effectiveness and validity. Descriptions of each evaluation item is included in the rubric (see Appendix D) as well as potential questions developed from these.

The formative evaluation for the Implementation Procedure evaluates the Strengths,

Weaknesses, Opportunities and Threatens of each iteration of the ePlay MakerSpace makes using a SWOT analysis frame. The SWOT analysis formatively evaluates how each iteration was enacted and the different elements implemented. As can be seen in the rubric (see Appendix D), potential questions are included, corresponding to the description of the evaluation criteria.

In keeping with the principle of comparability (Benney and Hughes, 1956), the design solution and implementation procedure of both iteration of ePlay MakerSpace are evaluated using this formative evaluation.

3.2.4.4

Formative evaluation in terms of design principles

In addition to the formative evaluations of ePlay MakerSpace participants, the design solution and its enactment and implementation for each iteration, a formative evaluation of the design in terms of the DPS included and used as a parallel analysis of the process. This serves two purposes: firstly, to evaluate the extent to which the design and enactment of each ePlay MakerSpace iteration meets the DPs, and secondly to identify potential

refinements to the DPs. The formatively evaluation as parallel analysis therefore allows for the iterative review, refinement and elaboration of DPs following each iterative cycle of the ePlay MakerSpace.

103

3.2.4.5

Capturing Formative Evaluations

A G/Form was developed to capture the different formative evaluations for each iteration:

Formative Evaluation of ePlay Design (link embedded). This is available on the website on page W1.2) Research Instruments.

Individual participants are evaluated using the G/Form: Formative Evaluation for

Participants (link embedded). Results are immediately available on a Google Sheet (G/Sheet) once forms are submitted. Teachers’ individual formative evaluation results are then

collated to provide an average formative evaluation of the participation in each ePlay MakerSpace session. This is achieved by analysing the individual teachers’ results on the G/Sheet of each ePlay MakerSpace, and establishing an average for each session. These averages are then used as the ‘Evaluation of Participants in general’ for each of the iterations.