Validity

The following sections address issues of ethics and bias in the research methods, as well as the reliability and potential for generalization of the findings and analysis.

4.5.1 Ethical Issues

There were certain challenges with this research that may have either biased the sample of participants or could have biased the results of the interviews. The potential biases were checked with the research advisory committee and understood prior to beginning the data collection.

First, because the TRCA collaboratively developed the list of stakeholders to be interviewed, the names of individuals they decided on might not be the truest representative sample of their stakeholders. For example, there were names removed by TRCA practitioners (during one iteration of developing the list) who might have been too ‘unfriendly’ according to the TRCA and there was concern about a possible backlash toward the TRCA from interviewing these certain stakeholders.

Throughout the interviews it was discovered that a few of the interview participants were friends of the TRCA CEO or had previous personal relationships with him. This could have affected their responses in terms of being overly sympathetic to the TRCA and its endeavors, or very positive and enthusiastic about the TRCA.

Participants from different stakeholder categories were either more willing or less willing to interview. For example, there was a lot of rejection of the invitations from the ‘private industry’

category and more or less complete participation from the ‘municipal government’ category,

even when replacements had to be suggested. While the researcher attempted, and somewhat successfully, to find enough new names to replace the initial rejections, it is important to note that this happened with the original sample.

The researcher’s level of experience as a novice interviewer could have affected the outcome of the interviews. Interviewing requires performing multiple different tasks and the ability to hold many different thoughts and respond quickly and appropriately all at the same time and while attempting to ensure the participant is comfortable (Legard et al. 2003; Warren & Karner, 2009).

The ability to do this requires practice and experience, both of which the researcher did not have much of at the beginning of the data collecting process. To counter this, several pilot interviews were conducted with the questions to gain experience (see section 4.3.3 Pilot Interviews), and the lead faculty member of the core research team attended the first few interviews in order to for the researcher to learn through observing him.

Finally, though the representation of categories of stakeholders was diverse, the actual diversity of participants that were interviewed was lacking. The participants were overwhelmingly of senior positions and usually older and male. This was partly due to the contact names given to the researcher by the TRCA. This group, being somewhat homogenous, could have affected the interview outcomes.

4.5.1.1 Participatory Action Research and Ethical Issues

The homogeneity of the group of stakeholders that was interviewed is arguably a consequence of a participatory action research approach, in which the TRCA research team was involved in all steps of the research and collaboratively developed this list. Participatory action research approaches are often thought to be beneficial to research participants because the participatory aspect of the research is considered democratic in that it increases participants’ voices and

agency and thus challenging existing power structures (Kindon, et al., 2007). However, this depends on which voices are being represented in the process. A structural research approach can often serve to reinforce the knowledge and ideas of elites in the system (Cornwall, 2003; Kesby et al., 2007). Often the framework of the research project helps to reinforce existing power hierarchies (Mosse, 2001). This is an ethical concern with the participatory action research approach that is highly relevant to this research. In this case, the research participants involved in the collaborative development of the list of stakeholders to interview were almost entirely senior staff persons and majority male. Not surprisingly, the final list of stakeholders was also almost entirely senior staff persons in their organizations as well as homogenous in terms of gender and race. It seems that the research participants represent a level of power and influence in their organization and this resulted in a list of stakeholder names that reflected a narrow, powerful and homogenous group that mirrored the research participants’ own identity and influence.

A group of stakeholders that represented women, junior staff persons, younger age groups or ethnic diversity may have led to very different results, for two reasons. First, simply due to the diversity of voices there may have been issues or concerns about the TRCA raised that did not emerge in the more homogenous group of stakeholders. Second, senior staff persons and CEOs, who mostly represented the stakeholder group, spend less time in daily, on-the-ground interactions with TRCA staff and TRCA programming. They are often overseeing activities and observing things at a broader scale and at more of a distance than staff below them in their organizations. This means they have a different perspective which might translate to their answering of interview questions, which could be less in-touch with the realities of their organizational relationship to the TRCA or more sympathetic to the power structure of the TRCA because they benefit from a similar structure in their own organizations.

4.5.2 Data Reliability

The interviews were completed over a five-month time period, and because of this there are things that may have influenced interview responses or even willingness to interview.

The current municipal political climate at the time skewed the answers of some participants when talking about their evaluation of Toronto, and there was a tendency to sometimes perseverate on the municipal politics with veiled references to Rob Ford. Rob Ford was mayor of Toronto from 2010-2014 and was likely one of the most polarizing and talked-about mayors the city has ever seen. This was due to his hardline right-wing policies, and in the later years of his mayoral post, a very highly publicized drug scandal and subsequent messy and public handling of the situation by Mayor Ford (Doolittle, 2014). There was also an upcoming municipal election lead-up (in which he was replaced by mayor John Tory) during the interviews, which might have swayed conversations to mention this (The Canadian Press, 2014a). Finally, some interview participants had had recent political encounters with Ford that were publicized, or were very familiar with mayoral politics and this explains some of the mentions of this in the transcripts.

Overall, the municipal politics of the time was generally a more popular topic of conversation among everyone, especially those whose work related to Toronto or municipal policy, which included almost every participant who was interviewed, and this very likely skewed the emphasis of many discussions toward the political environment and the current leadership.

A summer rainstorm and subsequent flooding of much of the city of Toronto and the GTA along the lakefront and in the valley systems happened in July of 2013, causing major infrastructure and transportation problems (The Canadian Press, 2014b). This was a flooding event that was uncommon on the scale that it happened, and, as it was a storm water flood event that involved areas of TRCA’s jurisdiction, it is possible that this event was fresh enough in interview

participants minds during interviews that it directed some of their discussion about the organization.

4.5.3 Generalization

Jane Lewis and Jane Ritchie (2003) discuss generalization in qualitative research as involving three distinct concepts: representational generalization, inferential generalization and theoretical generalization (p. 264). Representational generalization refers to whether the results can be generalized to a larger size of the same sample, inferential generalization is the degree to which the results can be generalized to another context, and theoretical generalization means whether the conclusions of the research can be applied to theory more broadly (Lewis & Ritchie, 2003).

A case study approach makes inferential generalization more difficult to begin with, mainly because case studies are meant to provide analytic or theoretical generalization rather than statistical generalization to other contexts (Yin, 2009). While the TRCA case might be able to provide some further insight for the literature concepts that are used in the analysis (systems thinking, organizational change, etc.), the uniqueness of the TRCA context makes it difficult to generalize to other settings. Although it is one of many conservation authorities in southern Ontario, the TRCA is quite the anomaly among those authorities given its substantially larger budget, organizational size and capacity and its densely urban environment. The TRCA faces different issues than many of the other authorities and for this reason it may be difficult to achieve an inferential generalization from this research.

There are certain aspects of this particular research that mean a representational generalization might be hard to achieve as well. Due to the way the interview participants were selected (partially by the TRCA staff themselves) the sample is likely biased and in that way might be less representative of a truer sample of TRCA’s stakeholders.