Rigour, corroboration, triangulation and validation 108

In addressing the methodological rigour against the criteria of: truth value (internal validity), applicability (external validity or generalisability), consistency (reliability) and neutrality (objectivity) (Guba & Lincoln, 1981), it is emphasised that requirements for assessing claims of evidence in the constructivist approach adopted in this study differ from the rules of evidence in logical positivistic frameworks. The absence of such rules does not affect the required rigour and this study was conducted in accordance with ethical and intellectual responsibilities in dealing fairly and objectively with the data. iREACH employees were given opportunities to comment on drafts of the non-academic reports and the few changes resulting from these comments, clarified minor misunderstandings, rather than changed any fundamental issues. Time and funding constraints limited the research. Not covered by any grant, it relied on iREACH resources, which was positive in terms of capacity building and ownership of the process, but it limited the scope to the time staff could allocate. While the research could have been an imposition on the busy schedule of staff members, they also appreciated the external involvement, as this assisted them with evaluations. Staff considered the study constructive, but also had to contend with the pressure of other duties.

This study achieved rigour through multiple sources of evidence, an open research process, maintaining accurate and transparent records and encouraging stakeholder participation in all research phases.

6.2.5.1 Triangulation and corroboration

The FG structure with participants from different interest groups provided multiple sources of information, enabling us to listen to perceptions from different community sectors. Feedback

from participants is one of the most logical sources of corroboration (Miles & Huberman, 1984) and it is also ethical to return the results to those who generated the information, giving them an opportunity to comment. Where this is not done, participants may feel disappointed, as was the case following research at the Women of Uganda Network (WOUGNET, 2003):

‘In July 2001 we received two Norwegian students who were doing their Masters degree by then and they wrote their dissertation basing on the findings from our association unfortunately they have never given us a copy of these findings’ (p.24).

In line with the view of Reilly & Gomez (2001), that results of evaluation studies should be made public in ways that are appropriate for different stakeholders, the iREACH research plan incorporated participant validation, enabling FG participants to offer feedback at meetings and a summary of the findings was prepared for this purpose. Such fora were arranged after the 2009 study and following heated discussions, there was general consensus that the summary reflected the participants’ views. Due to reduction in iREACH staff numbers, no feed-back sessions were arranged for the 2010 study, but the report and data will be included in a village open data system, currently being implemented.

Another important data source for triangulation was the 2010 survey. It was intended that further triangulation take place through the collection of statistics for indicators defined in the FGs. As the establishment of such a system would be beyond the scope of this thesis and in anyway had to be anchored in the community, it required the full support of and resources from iREACH, but these were not forthcoming. IDRC has recognised the importance of such a local statistical system and provided a grant in late 2010 for iREACH to establish an open commune data system, for the collection and dissemination of statistical information of interest to villagers and local authorities. Even with data of this nature, caution must be exercised in attributing improvements to iREACH, as only some of the impacts arose from its activities alone. The allocation of weights to contributions by various parties would be a time-consuming and probably futile exercise, but by acknowledging them, attention is drawn to the difficulty of attribution in an environment exposed to inputs from several agencies.

6.2.5.2 Applicability, construct and content validity and generalisability

As informants did not answer questions directly to fit into the model constructs, the extent to which their expressions validated these is to a considerable extent inferred. Such inferences were particularly challenging for empowerment, which is more abstract than capabilities and sustainability (e.g. how does one determine whether a certain expression reflects empowerment). With no pre-determined factors to measure the constructs, content validity (i.e.

ensuring that what is intended to be investigated is what is actually investigated) has only limited relevance. However, findings from other studies (interleaved with findings from the iREACH research) in the chapters presenting the field research results, were useful in an indirect way for content validity. Comparing and contrasting findings between other projects and iREACH will be even more essential in a potential gradual evolution of the model towards theory building. Although the aim of this research is to develop a model for more general applicability, results from this field study would most likely not be generalisable, as its contributions were contextual. The level of generalisation required for a theory would therefore have to await further field research from this and other initiatives.

6.2.5.3 Replicability

Replicability (i.e. anyone conducting the same research would get the same results) is a common validation method in the positivist epistemology, but in constructivist approaches results cannot be reproduced, as the mere participation in research would affect informants. Also, the very reason for a longitudinal study is to explore changes over time, so a study undertaken at a different time would most likely yield different results, as was found when comparing the 2009 and 2010 studies. So the focus should be on whether the study is believable, rather than replicable.

6.2.5.4 Neutrality, objectivity and bias

While in non-positivist frameworks, ‘objectivity in its pure form is an unattainable state’

(Lincoln & Guba, 1995, p.108), there could nevertheless be justifiable concerns about biased outcomes and subjectivity in light of the involvement of iREACH employees, as their presence could have influenced participants to express only positive views. In this research, the general risk of relying on interpreters was magnified by their status as iREACH employees. The reliance on staff for invitations of FG participants could also have introduced strong bias, as

could the venues for the FG sessions in iREACH hubs. But, ‘independence does not mean

isolation; those people involved in, responsible for and affected by the delivery of the projects and programmes being evaluated should make an active and meaningful contribution to the process’ (DFID, 2009, p. 17). As unfavourable views of some iREACH issues surfaced, it is

unlikely that staff exerted undue influence (e.g. by stacking FGs or manipulating interpretations). There were actually a few unsolicited negative comments about certain staff. Common themes and diverse views in the different groups appeared to be spontaneous, providing a high degree of confidence that genuinely held opinions were expressed. There was no apparent “kowtowing” — by participants to staff or staff to participants. Although the

research was not about evaluating staff performance in a way that could result in chastisement or reward, it is plausible that being associated with positive outcomes would be advantageous. In any case, there are not many alternatives for research of this nature, as fly-in-fly-out outsiders operating independently could also be subject to employee influence, neutralising the potential objectivity of total reliance on external resources. As the thesis is testing the feasibility of this type of evaluation, it would probably have been unrealistic and too expensive for an “outsider” to spend sufficient time at the pilot sites to arrange the total study independently. Anyone associated with a project for a sufficient time to gain the necessary trust would probably cross the boundary towards becoming an “insider”, with a stake in seeing positive outcomes, making the researcher sympathetic to the topic of the study (Bailey, 1994). In the same vein, reciprocal influences stemming from the interaction between the inquirer and others could also lead to bias.

My pro-bono participation in other iREACH activities could be a case in point. While such participation is common when entering a new site as a researcher (Stake, 1994), it can present problems of role dilemmas when conducting research in settings where the researcher is also an employee or has some affiliation. This issue has not been sufficiently addressed in the literature (Hurworth & Argirides, 2005). In this case, the pro-bono nature avoided such role dilemmas and there was congruence in both roles, which aimed at contributing positively to the project. Declaring a high degree of sympathy towards shared access facilities, should make the reader cautious when interpreting the results of this study. The rigorous analysis method adopted should guard against this sympathy colouring the findings.

Balanced against the risks of reduced data integrity are benefits, gained from involvement by employees and the richness and understandability they added. Local staff assisted with explaining certain concepts (e.g. the wider meaning of the term “walk the streets”, meaning gallivanting around). The presence of local staff also enabled the sessions to incorporate the exchange of information and impressions in an unplanned manner. Only staff members would have been in a position to respond directly to the variety of questions from participants and such exchanges are compatible with a naturalistic form of inquiry (Guba & Lincoln, 1981).

To conclude, although some findings might be questionable, we do not consider the limitations and the somewhat unconventional methodologies and nature of conducting the research to have distorted the data or even rendered it unsafe for analysis in the context of its purpose and should not undermine the findings with respect to the research questions. The study does not purport to present objective truths, but rather the perspectives of participants and respondents as presented in the FG sessions and surveys. While similar potential biases could

affect the surveys, these nevertheless serve as a backdrop against which to assess the risk of overstated claims of benefits arising from iREACH by FG participants.