Structure

This section orients the reader to the structure and main points of each of the thesis chapters.

Chapter 1: Introduction - presents the main focus of the thesis on power/data/knowledge relations in NGO impact evaluation, as part of today’s development 2.0 landscape. The thesis

motivation, approach, core themes and research questions are described. Key audiences and final contributions are outlined, and the historical and contemporary context is sketched, before this thesis structure preview.

Chapter 2: Literature review - describes two contrasting areas of development impact evaluation literature. The first literature domain is normative “technical impact evaluation knowledge” or TIEK. This literature concerns goals, methods and prescriptive discourse to support aid organisations in performing impact evaluations. The second domain concerns critical perceptions on development evaluation. This literature contrasts with TIEK, and deals with problems related to prescriptive discourse, how evaluations, NGOs and power relations unfold in practice and change over time. These issues are termed “critical configurations”.

Chapter 3: Theory - builds on problems identified in the literature around the lack of theoretical grounding for power, data and knowledge relations in TIEK. The “DIKW Pyramid”, a widespread and implicit model of data, information, knowledge and wisdom is found to implicitly underpin contemporary evaluation models. An alternative view of impact data/knowledge relations involving products, processes, power, political participation and practice, the “6P Sensitivities”, is proposed because DIKW and TIEK elide the diffusion of power and practice in evaluation networks, aid chains and critical configurations.

Chapter 4: Approach and Design - outlines a critical and engaged research stance, explains the rationale, and justification for using Cultural Historical Activity Theory (CHAT) to operationalise critical engagement in the study. CHAT’s framework for engaging research partners and for critiquing power and practice via activity systems, contradictions and temporal chains of activities is described. The chapter explains the research design, why the author selected two NGO cases for the study, access to research partners, and how data was analysed. The result is a comparative and qualitative CHAT study of two NGO case collaborations between 2013 – 2016, and a framework for exploring the ground between scholarly critique and pragmatic engagement.

Chapter 5: NGO Case 1 Rural India - features the first NGO, who are expert evaluators with a long-term philanthropic funder, and who run female farmer livelihood programs. The sequence of evaluation activities is described in the empirical section, and the analysis section articulates the CHAT activity systems, contradictions and temporal activity chains. Two kinds of impact are evident in the case, one local, unclear, and changeable kind of impact closely

associated with farmer lifeworlds (Impact-1), and one digitally documented and represented kind of impact deployed in marketing efforts (Impact-2). Power relations are generated as impact data/knowledge is incrementally edited along a temporal chain of evaluation activities, with illegible local voices and contexts edited out or submerged, and managerial, marketing and data management concerns expertly elevated by the NGO and philanthropy.

Chapter 6: NGO Case 2 HTSG Thailand - features the second NGO as a novice impact evaluator, lacking guidance and resources. The empirical section describes the NGOs learning, struggling and transformations. The analysis section articulates the activity systems, contradictions and temporal chains. However, in this case two contradictions are pertinent, between two kinds of impact as in the first case, and two forms of the NGO. HTSG are transforming from an old version of themselves into a new data/knowledge intensive NGO. The temporal chain illustrates the changes, elevations and submerging of different notions of impact, and the transformation towards, but not reaching yet, impact expertise. HTSG’s confusion and lack of data/knowledge capacity are key case results.

Chapter 7: Discussion –this chapter builds on the CHAT analysis and results concerning how power is generated in small NGO evaluation data/knowledge construction activities, to arrive at three contributions. The first contribution is to the evaluation body of literature and concerns the big picture narrative of the Impact Iceberg, where certain impact results become legitimate, and others illegitimate, silent, submerged under the waterline. The governmental and developmental processes in todays’ development 2.0 landscape identified in the literature are revisited, and seen to configure the iceberg. The second contribution is conceptual and pragmatic, labelled as Audit 2.0 devices, and challenges TIEK models and the DIKW legacy. The devices are: firstly, data/knowledge chains and networks; secondly, the 6P Sensitivities; thirdly, the Impact Spectrum; and fourthly, the contrasting notions of Datamentality and Datamateriality. The third contribution is Critical Engagement. This is reviewed as a way of taking critiques of power into the wilds of evaluation practice. Together, the contributions are advocated are elements in seeing evaluation-as-practice, which contrasts with scientific, business pragmatic, technology-centric, or participatory views of evaluation.

Chapter 8: Conclusion - reviews the key goals and core arguments of the study, describes a series of limitations, and lists future research options. It recaps how the study has articulated

the impact of evaluations in data/knowledge intensive development 2.0, rather than the normative “how to” foregrounded in technical evaluation discourse. Implications for

evaluation, data/knowledge models and development policy and practice are re-iterated at the end.

2

Literature: From technical to