Chapter 4: Epistemological Stance and Research Methodology
4.4 Validity and reliability, ethics, and limitations
4.4.1 Validity and reliability criteria
The validity of the research consists in its trustworthiness. I used three strategies to ensure its highest quality and reliability (Yin 1998: 32-36). First, I collected data from multiple sources of evidence, combining secondary and primary data, and diversifying the sources for primary data. This technique serves to validate the construct of the research, for it implies a triangulation in the data collection. Second, I used consistency checks between responses of informants, as well as with my own experience and with informal conversations with consultants and staff from both organisations. I had worked for nearly five years at the World Bank before undertaking the PhD thesis (see Section 4.4.2 on ethics and positionality). This experience was very valuable as a way to immediately react to responses, asking for further explanations when I was surprised by the answer and making it possible to deepen the conversation more quickly. Since the IADB has a similar mission and structure as the World Bank, and thanks to previous informal conversations with IADB staff and consultants, I benefitted from a similar additional check for the information acquired from interviews at the IADB. Thereafter, I joined the IADB as staff member - after having completed the data collection for this thesis. I worked at the IADB for two years and a half, and then decided to take a leave of absence to be able to complete my dissertation. The revision of the thesis therefore benefitted from this additional experience. Finally, the detailed description of the protocol of the research, including identification and contact of resource persons, way to record data, and questions asked, aimed to minimise biases in the research and to provide enough guidelines so that another researcher could reproduce the same study and reach similar findings.
4.4.2 Research ethics and positionality
The ethical standards of the research consisted in asking for permission to take notes at the beginning of the interview, guaranteeing the informant’s anonymity (never quoting them by name in the text), asking permission to include their name or title in the list of interviewees, and of course, answering any questions regarding the research. Ethical conduct also entailed transcribing the interview as quickly as possible after each meeting to ensure accuracy and precision. When notes were unclear and I could not remember precisely, I always mentioned it in the transcript.
Another key ethical consideration that relates to my unique positionality in this research dealt with my work experience in both MDBs, in particular with the IADB that I joined during the course of the PhD. As previously mentioned, I had worked for five years at the World Bank, mainly in the Human Department of the Africa Region, when I decided to apply for a PhD. During that time, I designed a project to support women victims of violence in Colombia, called the ‘Golondrina project’ that received a USD one million grant from the Japanese Trust Fund. My research for this project reinforced a long-standing reflection on the main challenges and injustices in the developing world and motivated me to embark on the PhD journey, for I wanted to research in more depth what major aid agencies like MDBs could do to push the agenda forward in violence prevention from a gender perspective. I thus left the World Bank to go to the United Kingdom to begin the Doctorate. After I had completed the coursework, passed the upgrading seminar and conducted all the interviews in the United States and in Colombia, I received an offer to join the IADB as a Young Professional. Aware that such positions were rare and prestigious, and in consultation with both of my supervisors at the Bartlett Development Planning Unit (DPU) at University College London (UCL), I accepted the offer. I declared my research during the recruitment process and upon arrival, and informed the Ethics Office of the IADB. I also informed the Director of the Office of Evaluation and Oversight to which I had been assigned. After six months, I received an offer to become Specialist at the IADB. At the Office of Evaluation and Oversight, I first co-led an evaluation on implementation processes of five citizen security projects in Central America and the Caribbean, and then led a thematic evaluation on citizen security at the IADB looking at its contribution in relation to major challenges in the sector. Both evaluations were in line with my research but did not overlap. The first study involved five case studies and looked at implementation variables that facilitated the execution of particular projects. The second evaluation focused on looking at how the IADB had responded to the complexity, risks, and knowledge gap that characterise citizen security in the region. My PhD research has fed and is enriched by this work on citizen security at the IADB. It has fed it because my research gave me a solid understanding on violence prevention and the role of MDBs in that respect, understanding that is reflected in my analysis in both studies. It has also been enriched because I spent the past two years working on citizen security and getting to know
from the inside the operational and strategic discussions surrounding the issue. During that time, I interviewed all Citizen Security Specialists at the IADB. This unique positionality as PhD researcher and MDB staff highlights key ethical questions in the research process and data analysis. I build upon a rich body of feminist literature that reflects on power relations and the particular positionality of the researcher, and subscribe to England’s words when she argues that ‘the researcher’s positionality and biography directly affect fieldwork and fieldwork is a dialogical process which is structured by the researcher and the participants’ (England 1994:80). In the particular case of my research, I am aware interviewees might have responded differently to the Evaluation Officer and to the PhD researcher. I thus used the knowledge gained through the 19 interviews undertaken in 2012 as background information only. With the agreement of my Director and the Human Resources Department at the IADB, I decided to take a leave of absence to complete my dissertation. As such, my role as a researcher and an Evaluation Officer has always been transparent and fully disclosed throughout the process. Beyond the question of transparency, it is undeniable that the final thesis is the result of this dual positionality and a dialogical process constructed in an iterative manner between the researcher (myself) and the researched, with the specificity that some of the ‘others’ being researched, as England (1994:87) name them, were sharing and reflecting part of my own experience as former MDB staff member.
4.4.3 Research limitations
This research presents three main limitations. These limitations do not affect the validity of findings and could serve as the basis for future research. First, the research does not present a detailed mapping of interventions in the six sectors (in addition to Citizen Security/ Violence Prevention) where interviews were conducted. I started such a mapping, systematically examining more than 150 projects at the World Bank in six sectors to assess the extent to which violence prevention and gender were explicitly referred to in the diagnostic, components, and result frameworks in projects documentation. When I began doing a similar exercise for the IADB, I quickly realised I would not be able to undertake a similar analysis and compare results, for lack of access to information. As Evaluation Officer, I then commissioned such an analysis during my work at the IADB, but results were still unsatisfactory because of varying levels of available information across projects.
Even at the World Bank, interviewees pointed out how project documentation only partially reflected the interventions developed in countries and how levels of detail regarding those interventions varied significantly PAD to PAD. I thus stopped the mapping exercises. I considered them interesting pieces of analysis and information, but they were not indispensable to conduct the research and reach meaningful findings. In fact, none of the banks has officially adopted a mainstreaming approach to violence prevention. What mattered was to collect and analyse participants’ responses about the opportunities and constraints affecting the mainstreaming of inclusive violence prevention.
The second limitation of this research lies in the single country case study. As mentioned in Section 4.3.1, single case studies have been criticised for the lack of generalisation of the findings (Bell 1993). In the present case, and in the context of aid, case studies in other countries might have found different results, for donor-recipient relations vary with the level of economic development of the country and the political position of Governments towards MDBs. As such, findings from the Colombian case do not aim to be generalised. They are nonetheless interesting for the characteristics highlighted in the selection criteria for Colombia, in particular in terms of readiness of the context for mainstreaming inclusive violence prevention and levels of reflectiveness on the role of MDBs in that respect.
Finally, this research shares a common limitation to all research projects relying principally on data collected through in depth interviews: findings depend on the particular experience of interviewees in the study populations. Even though I contacted as many staff members, managers, and Board members in both MDBs as I could, I am aware that I might have missed relevant resource persons either because I did not identify them, they did not reply, or they were not in Washington when I was there. Not having done these interviews does not jeopardise the findings of the research. In fact, in most cases, I interviewed other persons who had a similar position. I intentionally contacted as many persons as I could identify, aware of the fact that in any research project there is always a number of persons who cannot or are not willing to participate, above all considering the time constraints of a PhD field work. All the interviews were conducted in depth and their number were equal or higher compared to those undertaken in other thesis similar in scope (Satterthwaite 1999; Gulrajani 2005). They also covered a wide range of positions, sectors, and experience that
allowed me to gain an extended understanding of both organisations from an internal and external perspective, and test all the hypotheses and variables of the research. Another limitation of primary data lies in the semi-structured format of the interviews. I chose to adopt a conversational approach to collect as much information as possible and obtain the genuine interpretation and perception of staff members. This technique facilitated contact and brought a wealth of informative anecdotes to the research, but I could not always ask all the questions I would have liked. This was nevertheless the exception rather than the rule, and I consider it as the price for opting for flexibility instead of rigidity.
4.5 Conclusion
This chapter has described the interpretivist stance and the qualitative methodology used to conduct the research. The findings are presented in the following chapters. Chapter 5 provides the context in which to interpret the results from the in-depth interviews with MDBs’ staff members, managers and Board members, and the country case study. It presents the situation of violence in LAC with a special focus on Colombia, and identifies the main challenges in policymaking to prevent violence from a development and gender perspective in LAC. Chapter 6 presents the findings from the process tracing analysis. It reconstructs the evolution of violence prevention at the IADB and the World Bank from 1998 to 2011. Chapters 7 and 8 analyse responses from MDBs’ interviewees to explain why violence prevention has only emerged slowly in the agenda of both banks. For this, it discusses the analytical categories developed in the conceptual framework. Chapter 9 presents the findings form the case study in Colombia, and Chapter 10 concludes by reflecting on the main findings and proposing directions for future research.