7.1
Overview and context
This chapter sets out the elements for an evaluation framework for exemplar projects, consistent with the fourth aspect of the project requirements. This element of the project raises some complex issues. Although the features of potential exemplar programs are identified in chapters 5 and 6, the scope and budget of any such programs are at this stage undefined. For this reason, the framework set out in this chapter outlines the principles that should be applied in any such evaluation and identifies more specific potential approaches that may be applied in any particular instance. Further, any evaluation that takes place within the parameters outlined here needs to take account of evaluation activities occurring under the Domestic and Family Violence Reform Evaluation Strategy in It Stops Here.
As discussed in chapter 2, the field of violence prevention is in its developmental phase. Limited empirical evidence exists on the kinds of approaches that are effective, and significant challenges arise in assessing such approaches. There are several challenges that are relevant in this context, some of which are conceptual and methodological and others that are practical. From a conceptual standpoint, it is acknowledged that the measurement of the impact of prevention initiatives is inherently difficult because what is being measured is the absence rather than the presence of certain phenomena (DFV), and there is no certainty that the phenomena would have occurred in the absence of the prevention measures.
Further, the intended impact of such programs is often wide and relatively non-specific. Some programs are intended to have shorter-term effects, but the impact of other programs is intended to be long-term and attitudinal. Any effects of the program might well not be evident for a significant amount of time. Even if they do become evident, it may be difficult to determine the extent to which effects consistent with the intention of the program are attributable to the impact of the program or other developments, including individual circumstances and broader social influences. The corollary of this is that the absence of effects inconsistent with the intention of the program may also be attributable to issues other than the failure of the program to achieve its objectives. These issues are particularly relevant in the current environment in Australia, where a number of prevention initiatives are being implemented at federal, state and local level, including those such as The Line and others identified throughout this report and listed at Appendix 1. From a practical perspective, the other challenges for the evaluation of family violence prevention programs arise from the funding and organisational context in which these programs are embedded. As spelt out in section 5.2.3, often such projects are funded out of specific-purpose grants emanating from a variety of sources, including federal, state and local government grant programs and grants tied to specific purpose organisations, including philanthropic organisations. Often these grants have a short lifespan.
The value of and need for evaluation of violence prevention programs is well-recognised (WHO, 2010; VicHealth, 2007; Flood, 2013) because of the emergent nature of the policy, practice and the body of knowledge in this field. Such evaluations not only support sound
funding decisions but they add to the body of evidence on violence prevention by developing practice-based knowledge. This is one of the arguments for building evaluation funding into program funding packages.
The level of grant funding and the life of the program may provide limited scope to do evaluation and any such evaluation may occur on a very modest basis (Flood, 2013). So called “gold-standard” evaluation approaches involving external evaluation experts and methods based on experimental design (based on two groups—one that receives the intervention and one that doesn’t) are often out of reach, not just for financial reasons but also because a control group may not be readily identifiable for some programs (Flood, 2013). For this reason, the value of smaller scale evaluations has been emphasised in the family violence prevention literature, along with recognition of the importance of supporting program providers to develop the capacity and expertise to evaluate their own programs. Such approaches are recognised to have a number of advantages including supporting the development of empirically informed reflexive and self-critical or self-aware practice.
A significant direction in the literature on evaluation of family violence prevention programs, notably in recent reports produced by VicHealth (Flood, 2013; Kwok, 2013), is the endorsement of “empowerment evaluation” where the agencies and staff implementing programs are supported to perform their own program evaluations. The main justifications for this are twofold: first, that prevention initiatives are often implemented by community- based organisations with limited resources, including very limited resources for evaluation. Thus prevention is seen as a community-driven responsibility and the context and purpose of evaluation activities are based on self-reflective practice and ongoing improvement in program development (Flood, 2013). Second, empowerment evaluation is seen as a capacity-building exercise in which program staff become skilled not only in delivering their programs but in assessing and evaluating program effectiveness, thus supporting “self-determination” in the family violence prevention field.
This direction has emerged in a policy and funding context where an agency—VicHealth— has implemented a focused primary prevention program supported by a comprehensive framework (VicHealth, 2007) for the past seven years. VicHealth has thus developed a philosophy and infrastructure to support a direction of this nature and worked intensively with the agencies delivering programs to equip their staff to conduct these kinds of evaluations. This approach to evaluation is thus embedded in a particular policy and organisational context designed to support this direction. As will be discussed in more detail below, internal evaluations have their limitations, and questions regarding their validity have been raised (Dyson, 2014). In light of such concerns, internal evaluations may often benefit from structural support from a larger organisation or policy and practice framework.
In the absence of these conditions in the NSW context, and in light of the developing nature of practice and knowledge in the prevention field, there are several strong rationales for supporting external, rather than internal, evaluations of new programs or programs that are being funded but have not yet been evaluated, depending on the size of the funding package. External evaluation is independent and this will ensure the evaluation is informed by the exercise of objective judgment. This is important for three reasons. First, the judgment exercised by the evaluators will not be influenced by any interest in whether or
not the program continues. Second, the evaluators have a professional distance from the program and are thus able to consider professional practices, attitudes and dispositions from a neutral standpoint. Nonetheless, working with professionals in the program to develop an informed understanding of professional practices, attitudes and dispositions is important. Third, distance from the program and the professional and client relationships within the program means that data from professionals and clients will be gathered by a neutral third party, and will not be affected by any existing relationships and dynamics within the program and its client group. It is important that such dynamics are examined from an external rather than internal perspective. Having said that, it is acknowledged that the programs in the areas being considered may raise issues of particular sensitivity arising from the circumstances of the client group and it is important that the evaluators work with the program professionals to ensure that these sensitivities are dealt with appropriately.
For new programs, planning for the evaluation should begin with planning for the implementation of the program. External evaluation should be implemented in a collaborative manner with the evaluation team working closely with the program implementation team.
The nature of evaluation strategies adopted will depend on the aim of the evaluation. Formative evaluation refers to the process of examining a program or initiative in its pilot or developmental phases with the intention of using the evaluation information to refine the final form of the program or initiative. Summative evaluation refers to examining the impact of a program of initiative. Further, evaluations may focus on processes or outcomes. Process evaluations examine the impact and effectiveness of the processes applied in a program. Outcome evaluations focus more specifically on the result achieved by the program. Decisions in relation to the nature of the evaluation approach applied are informed by which of these foci is the core purpose of the evaluation. Some evaluation designs may include all of these elements.
7.2
Principles and implications
Careful consideration should be given to the question of whether evaluation is carried out internally or externally. It may be that external evaluation is the preferred approach in light of the current policy and funding context in NSW, at least in the short to medium term.
Planning for the evaluation should commence with planning for program implementation. The evaluation should be planned and implemented in a collaborative approach with the program staff. A collaborative approach will not only ensure the program aims and context are appropriately reflected in the evaluation approach but can support capacity building for reflective practice in the future.
The scope and nature of the evaluation should be proportionate to the funding package for the program. Larger funding packages require a more rigorous evaluation approach. Internal evaluation may be an appropriate approach for programs that are being funded
to expand and have previously been externally evaluated.
Internal evaluation on a regular basis should be supported when a program is out of its establishment phase.
7.3
Evaluation methodology
The following sections set out the steps and approaches to developing appropriate evaluation methodology for the kinds of exemplar programs discussed in chapter 6. This discussion draws on the literature on evaluation generally, as well as evaluation literature in relation to family violence programs specifically. To illustrate the steps in the process, examples have been drawn from the evaluation of the St George Migrant Resource Centre key program, discussed in chapter 9. Other documents and reports that have informed this approach include:
Evaluation and Innovation in Family Support Services (Child Family Community Australia [CFCA], 2013a);
Planning for Evaluation I: Basic principles (CFCA, 2013b); Planning for Evaluation II: Getting into detail (CFCA, 2013c).
Additionally, the experience of the researchers at AIFS more broadly in conducting evaluations and developing evaluation frameworks across a range of areas, informs this discussion.
In broad terms, evaluations have four main elements. These are an initial conceptual element that involves identifying the objectives of the programs and developing an understanding of how the program aims to achieve these objectives. The second involves developing a series of evaluation questions to identify whether the program meets its objectives. The third entails identifying what information can be collected to answer the questions. The fourth entails implementing the data collection strategies. The fifth involves analysing the data and using them to answer the evaluation questions. These steps are discussed in more detail in the next sections.