NGO Case 2: context, access, timeline, methods and data

This section introduces the research context, how accessed evolved, a case timeline, methods used to generate data, and details of the data used in the study. In this case, research centred on Hill Tribe Support Group (HTSG), a local NGO in northern Thailand with normally around 20-25 staff. HTSG were founded in 1996 when two staff left a large international NGO during a period when it was withdrawing funding from Thailand. HTSG provides healthcare and education services for minority ethnic communities and children in isolated rural settings that lack transport or telecommunications infrastructure. Unlike Rural India, HTSG’s funding comes from diverse sources, including direct public fundraising, local and national government, national and international private foundations, international NGOs, the UN, and the EU.

In terms of access, the study began when HTSG responded to an email enquiry about collaborative research on impact evaluation in May 2014. A series of email exchanges between the author, two office staff (Susan and Orr), and the Director of HTSG (Khun) led to a one- week site visit to HTSG in Thailand in August 2014. This visit included discussions and conversations with key office and field staff, group meetings, formal and informal social activities such as lunches and car rides, a presentation by the Director, a meeting with a board member, and an evaluation needs analysis workshop run by the author.

Between August 2014 and February 2015, discussions continued over email and included a proposal for further evaluation research and advice as well as a second trip to build HTSG’s impact evaluation capacity. The second trip was for 10 days in February 2015, and featured further conversations, group discussions with field and office teams, meetings, and a series of workshops lead by the author that focused on evaluation approaches and implementation. Collaboration continued through 2015 and 2016. During this time, it focused on distant support for the design of an evaluation baseline study as part of a healthcare project extension to two new villages, advice and support for a new Migrant Project funding application to international donors, and a baseline evaluation to start the Migrant Project.

Figure 4.8: Research timeline for NGO Case 2 HTSG Thailand

In terms of both data generated and methods, this case was more extensive than case 1, involving two field trips to Thailand by the researcher. The methods and techniques used to generate empirical data during the study included: one-to-one conversations, small group discussions, large group meetings, presentations, workshops, field notes, recordings, email exchanges, reports, spreadsheets, photographs, diagrams, online conference calls, and the co- production of evaluation documents and plans created during workshops and online collaborations. Conversations, meetings and workshops generated data about specific NGO activities, not an “entire culture and social life” (Hannerz, 2003: 208). Conversations were not interviews structured in advance, but open-ended to encourage participants to discuss their contexts and needs (ibid: 209).

As with case 1, CHAT informed the interactions and engagements. Probes or stimulations by the researcher were used to explore how evaluation was perceived by or could be arranged by the NGO, to further understand participant views, plans, goals, and limitations. A key feature of the research was the workshops held during visit two, in which the author led sessions on

evaluation approaches and techniques, and learned about the novice status of the NGO in terms of their impact evaluation experience, expertise, resources, tools and attitudes. The workshops themselves had a dual focus – to offer consulting support about TIEK, but also to test and probe NGO concerns, expectations and needs. In this sense the researcher was not a “passive recorder who avoids provoking responses” (Salancik, 1978: 638), but was an active researcher “playing the role” (ibid: 639) of a consultant. This kind of probing admittedly followed the author’s interests (Gaver et al, 1999: 29) in testing critical insights, but it also allowed the researcher to build dialogue and opportunities for learning around the limits and problems faced by the NGO. Examples of this included how needing to do marketing with impact stories surprised staff, or how exploring evaluation voices and networks in the first field trip prompted the NGO to develop data brokering and sharing relations with partners. These responses and others are described in more detail in the case chapter.

The author provided advice and support for HTSG on how to plan, approach and conduct impact evaluations. In return, HTSG allowed access to their staff, work processes and project information. In retrospect, the author influenced HTSG’s approach to impact evaluation. As with Rural India and Imagine, only by offering a “service” as a contribution, and by engaging with the NGO’s needs, was access and the testing of critical insights possible.

In terms of data generated during the case, Table 4.3 gives an overview. Appendix 4 adds detailed information on all empirical interactions and sources. Key participants included: Khun (NGO CEO), Susan (Donor liaison and US intern), Orr (Finance Officer), Pang (Healthcare Project Manager), Cherry (Evaluation Specialists and US Intern), and Mia (Migrant Project Coordinator and US Intern).

Type of data # Details and focus

Emails 100+ Arrangements, requests, advice; often multiple-page messages Site visits 2 Visit 1: August 2014 (5 days). Visit 2: February 2015 (10 days). Documents 34 E.g. annual reports for funders

Spreadsheets 7 Project data on beneficiaries, staff and volunteers Co-designed

documents

8 Evaluation plans, brainstorms, tools (e.g. survey or interview questions designs), funding application sections

Group meetings 12 In offices, meeting rooms, restaurants; 3-10 people, 30-120 mins 1-1 conversations 10 With one staff member or director, 15-25 mins on average Presentations 2 By director (1) and by researcher (1)

Workshops 6 Focus on learning aspects of impact evaluation

Group calls 2 Online group conference calls for evaluation advice/planning Website pages 20+ HTSG homepage, about, project pages

Field notes 100+ pages

Hand-written in notepads, digital on laptop, text & diagrams

Table 4.3: Overview of HTSG Case 2 data, May 2014 - December 2016

Study participants included HTSG’s director, one board member, office and field staff, the healthcare project team, and three US interns who volunteered at the NGO for one to two years each during the period of the research. Twelve out of 25 staff members took part in the research. The author speaks Thai to work level, although with some grammatical inaccuracies, and a staff member / translator was available for all group events. Recordings were made of workshops. Field notes were taken throughout, using a notepad and pen or a laptop during or shortly after events.

Interactions centred on HTSG’s impact evaluation goals, plans, knowledge and capacity, with the content of conversations, group calls, and workshops oriented around supporting their evaluation learning and a funding bid. The author’s input included: asking if and how evaluations were being carried out already; asking what resources were available (e.g. human, technological, time); probing how to improve the NGO’s awareness of and performance of

impact evaluation; and probing how to use impact and evaluation knowledge in planning, communications and funding. Overall, the empirical data and analysis are used to describe HTSG’s evaluation work and how their learning about impact evaluation was implicated in changes to their organisation and work.