Based on the stage model of evaluation (Patton, 1997), the next step in the research process was to conduct a formative evaluation of the draft manual and its implementation during a pilot workshop. Formative evaluation is “intended to improve performance, [and is] most often
conducted during the implementation phase of projects” (OECD, 2010, p. 23). It aims to understand issues and refine projects to make them more effective, efficient, and useful and is beneficial for situations in which the project is new or the approach is novel (Hall & Hall, 2004;
Patton, 1997; Tessmer, 1993). This phase responded to the second research sub-question, “What form might a relevant, accessible, and effective conservation-focused conflict sensitivity
framework take?” It helped my PDP team collaborators and I understand if the manual, which we revised in response to the needs identified during Phase I, and the workshop based on it were relevant, accessible, and effective for conservation practitioners. We sought to answer questions related to whether the manual and workshop in their current forms sufficiently addressed the needs of conservation practitioners and, if not, what changes should be made so that the manual and any associated workshops would effectively address the needs of the widest variety of practitioners.
Formative evaluation is focused on improvement and thus fits well within the action research approach, which is concerned with action for problem solving as well as the
development of participants’ capacities (Boog, 2003). By evaluating the draft manual and pilot implementation workshop, the PDP team and I were simultaneously able to develop the
capacities of workshop participants in conflict sensitivity and collect information that we used to
“troubleshoot” and refine the manual prior to its finalization and launch (see Chen, 2005). And because my collaborators and I went through this process together—defining evaluation
questions; collecting, analyzing, and interpreting the data; and making determinations on how to use the results—the PDP team had opportunities to develop its evaluation capacities.
The steps of formative evaluation vary (Patton, 1997; Tessmer, 1993). Tessmer (1993) lays out a comprehensive process for the formative evaluation of educational products, which I
found useful for guiding this research phase, as the manual and pilot workshop are similar to an educational curriculum for conflict sensitivity. In particular, I incorporated two parts of
Tessmer’s process: an expert review and a pilot implementation. Expert reviews assess the accuracy, technical quality, usability, and appeal of the material portions of the project, i.e. the Environmental Peacebuilding Training Manual (Tessmer, 1993). To get the most relevant and useful feedback possible, it is important to seek out experts “who have special knowledge, skill, or experience with regard to the content, features or audience of the [material]” (p. 48). With my CI collaborators, we selected four people to participate as expert reviewers.8 Due to the short time period in which we needed to conduct the expert reviews (the PDP team faced an institutional deadline for the publication of the manual), experts were purposefully selected based on their skills, expertise, and personal connections. They represented the fields of environmental conservation, conflict sensitivity, evaluation, and training methodology, which were the areas of knowledge most important to us because they covered the key components of the manual.
Each expert reviewer received an electronic copy of the manual to review, after which they participated in semi-structured, in-depth interviews that focused on both the content and structure of the manual. The interview protocol was designed with my PDP collaborators’
feedback.9 Most reviewers also returned a copy of the manual in Word format with comments and track changes as another form of feedback, which the PDP team and I reviewed. I compiled
8 Two other people participated in the expert reviews but did not consent for their feedback to be used in this dissertation. Their feedback was used in the final evaluation provided to the PDP team but is not in the discussion here.
themes from both the interviews and the hard copies for analysis and discussion. These findings are discussed in further detail in the Chapters 5 and 6.
For the pilot implementation, the PDP team used the manual to outline a workshop in order to understand how the manual would work in practice and whether its implementation was effective, accessible, and relevant for conservation practitioners needs vis-à-vis conflict.
Conducting a pilot implementation in the form of a training workshop in the Philippines most closely matched with Tessmer’s (1993) field testing, or a “situated evaluation” process in which
“instruction is evaluated in the same environments in which it will be used when it is finished”
(p. 137). As Chen (2005) notes, “the key to designing pilot tests is to mimic exactly the program activities and processes planned for the full-scale implementation” (p. 121). The setting for the workshop was chosen by the PDP team based on country programs’ interest in the manual and available resources, so the selection process was likely similar to how future workshop locations might be selected. Over two and a half days in Palawan, Philippines, members of the PDP team, CI Philippines staff, selected partners, and myself covered topics in the manual including environmental peacebuilding concepts, conflict analysis tools, collaborative consensus building, facilitation, conflict-sensitive programming, and monitoring and evaluation for conflict
sensitivity.
Keeping with the action research paradigm, the PDP team and I worked together prior to the pilot implementation workshop to develop an interview protocol for a select group of
workshop participants who would complete pre- and post-workshop interviews.10 CI Philippines staff and the PDP team then identified four workshop participants that could participate in those interviews to provide a contextualized look at participants, their experiences, any conflict-related
challenges they faced as part of their work, their familiarity with conflict sensitivity frameworks, what they hoped to and did get out of the workshop, and how their perceptions and knowledge had changed as a result of the workshop. These people were purposefully selected based on their availability as well as their long-term and varied experiences with CI. None of those selected were interviewed during the needs assessment, so their perspectives provided new information on the conflicts and needs that conservation practitioners face as well as detail that would help the PDP team and I better understand the specific context of the manual’s implementation.
In addition to the interviews, all workshop participants were given pre- and post-tests to assess changes in their knowledge of and self-perceived capacities related to key concepts presented during the workshop. The PDP team and I developed the tests together to address the research questions as well as additional information required by CI (such as participants’
feedback on the workshop location and logistics). I also conducted participant observation throughout the workshop, taking notes on peoples’ reactions, feedback, and questions. These observations provided me with first-hand experience of the workshop process and atmosphere, and I was able to record participants behaviors, statements, and reactions in real-time (see Creswell, 2009). I kept my observations in the form of field notes, careful to distinguished between what happened and my own thoughts, questions, and reflections using different colored pens, styles of writing, and thought bubbles (see Gibbs, 2007). I also had debriefing sessions with my PDP team collaborators when possible, noting their observations and reactions as well.
Using the data collected from the interviews, pre- and post-tests, and my debriefing sessions with the team, I developed an interim evaluation document that constituted the last round of feedback for the Environmental Peacebuilding Training Manual prior to its
finalization.11 The evaluation document provided information and analysis to the PDP team on how a conflict sensitivity framework could be used for environmental conservation and included recommendations for improving the manual based on the feedback from the expert reviewers as well as those who participated in the workshop. The PDP team and I then discussed the
document and what final changes would need to be made to the manual. I was responsible for making some of those changes myself, particularly those related to monitoring and evaluation.
After the many rounds of iterative feedback based on a cycle of design, implementation, review, the manual was then made available to all CI staff at the end of June 2017.12