7.9 Interpretation of the findings
7.9.4 Does it matter that we do better?
Given the issues associated with public sector evaluative reasoning practice
evident in this study, this section considers the question: Does it matter that we do better?
As described above, the five elements are interconnected and together build a coherent case from which an evaluative conclusion can be drawn that is defensible (Fournier, 1995; Fournier & Smith, 1993). Therefore the absence of, or weakness in one or more elements undermines the evaluative claim and therefore weakens the evaluative conclusion/judgment. More fundamentally, it compromises the logic underpinning the evaluation and therefore the evaluation report. Such flaws in logic are illustrated in some of the reports that contain either no evaluation objectives or questions (three reports) or only non-evaluative evaluation objectives
(seven reports). Of these 10 reports, seven end with one or more conclusions/ judgments that use evaluative language. Put simply, such reasoning is logically impossible and therefore seriously flawed.
Further as described in chapters 4 and 5, I assert that evaluative reasoning is evaluation Theory (the emphasis being on the capital T). It is what makes reasoning from values to an evaluative judgment valid according to informal logic. It is what distinguishes evaluation from other forms of systematic enquiry. If evaluation Theory (aka evaluative reasoning) is at the core of evaluation, it is critically important that those of us who conduct evaluations practice rigorous evaluative reasoning. Reports based on poor evaluative reasoning have the potential to discredit evaluation as a legitimate and useful tool for government agencies. Referring to the question in the title of this sub-section, based on the snapshot offered by the meta-evaluation findings it appears that some evaluators need to do better, and it does matter that they do. Given my premise that evaluative reasoning is a critical aspect of evaluation quality, there appears to be need for greater diligence about reasoning to an evaluative conclusion.
7.10
Limitations
There are a number of limitations associated with the meta-evaluation. Firstly, the sample is not, nor does it claim to be, representative of public sector evaluation reports. Only evaluation reports in the public domain were able to be considered for inclusion in the sample. There is no requirement in New Zealand for public sector agencies to make evaluation reports and other official documents publicly available, other than via a formal request made under the Official Information Act 1982. As noted above, of the 63 agency websites examined, only 22 websites contained evaluation reports (not summaries) dated 2010-2013. No reports were available on the websites of some large agencies.Secondly, while there are many programme evaluation reports posted on websites, there are noticeably fewer policy evaluation reports. The reports of some large-scale, national policy initiatives posted on agency websites (such as Working for Families and KiwiSaver) are summary reports of high-level findings written for a general audience. Such reports were excluded from the sample because they lack sufficient detail about
the evaluation approach required for the meta-evaluation. Lastly, the review focuses only on central government agencies as there were an insufficient number of reports available on local government agency websites.
A further limitation is associated with desk-based examination. There are a range of influences that determine the design and conduct of an evaluation and a desk-based examination is unable to collect such information. As noted above, political influences are both inherent and significant in the public sector context. What is presented in an evaluation report represents the requirements of the commissioner irrespective of whether the report has been produced internally or externally. An evaluation may have been poorly scoped by the commissioning agency, leaving the external evaluator to do the best they can with a poorly considered brief. An evaluation may have had a restricted budget, resulting in an argument that is unable to be supported by expert or literature-based warrants. These and other important contextual factors are not visible in an evaluation report, unless they are included as information about limitations. Lastly, the contents and format of an evaluation report may reflect the requirements of the commissioner. A commissioner may request a report presenting high level findings only, and may place lesser value on a robust argument supporting such findings. A final limitation is that the examination of the reports was undertaken by one researcher who knew some of the report authors, creating the possibility of researcher bias. A larger meta-evaluation could be undertaken involving a small team of evaluators thereby allowing for inter-rater reliability to be established.
7.11
Conclusion
This study seeks to understand how evaluative reasoning is understood and practised in a public sector context in Aotearoa New Zealand. While the Q study reported in chapter 6 provided insights into how evaluative reasoning is understood in this context, this chapter has presented a different perspective by examining (using meta-evaluation) evaluative reasoning practice as demonstrated in evaluation reports written or commissioned by public sector agencies. The meta-evaluation indicated significant variability of practice among a non- representative sample of 30 reports written by New Zealand-based authors in the period 2010-2013 for 20 public agencies. The snapshot of current practice
provided by the meta-evaluation findings suggest there is scope for evaluators working in or commissioned by public sector agencies to improve their evaluative reasoning practice, particularly in respect to building a strong evaluative claim through warranted argument. This chapter has argued that improving the practice of evaluative reasoning is not an option for evaluators. Rather, robust evaluative reasoning is at the core of what it means to undertake evaluation, particularly in the public sector context. This argument is discussed further in chapter 9.