Research Methods
4. Accurate Representation: Analysis and ‘evidence’
While it is certainly the case that data analysis, along with systems that frame data gathering, can in some cases generate what might broadly be designated as evidence368, at the community level information that results in a fuller
engagement with the locale must be concerned with the intricate interplay of individual and communal forces.
In relation to the primary research that informs this work, I took the position that research outcomes that flowed from various and diverse contexts could not be conflated under the rubric of ‘evidence’. Rather, they constituted a representation of individual contributions. As indicated in Part One, agency is an emergent and contingent quality. For that reason, ‘coupled human and natural systems’ must be considered in their very real complexity. By implication, research outcomes are context-specific and do not, in my view,
form the basis of any claims with regard to a more generalizable representativeness, or indeed to transferability.
With regard to qualitative primary research more generally, the business of representing participant contributions and research outcomes can become contentious, especially when research data are subjected to analysis from which ‘evidence’ is (allegedly) constructed. This issue goes to the heart of what constitutes evidence in qualitative research and warrants some closer examination.
In an extensive review, Freeman et al369 address the complex interplay of
various paradigmatic and theoretical-conceptual challenges that apply here. Their overarching challenge addresses the shift currently underway in many academic institutions that strives to reinstall ‘science’ in the conduct of social research. The largely positivist frame that applies here has been mandated into law by the US Government, and is supported by the National Research Council. As they note:
With the NRC prepared to define evidence and the American Educational Research Association imposing standards for reporting on research methods in its publications, qualitative researchers may feel under siege. Top down efforts such as these to legislate scientific practice and mandate research design threaten to harden the boundaries of what counts as science, to devalue many qualitative research endeavours, and to limit creative research practice of all kinds.370
As the authors establish, there is a high need for the various disciplinary streams that engage in qualitative research to ‘…avoid consensus about or prescribe standards of evidence in this diverse field. Such prescriptions…amount to disciplinary action that constrain the generation of knowledge rather than improve it’371.
Beyond the specific case example provided here, qualitative research has from time to time been engaged in processes that were directed at legitimising its place in the broader practicum of social research. While the means by which
369 Freeman, M., de Marrais, K., Preissle, J., Roulston, K. and St. Pierre, E. (2007), “Standards of
Evidence in Qualitative Research: An incitement to Doscourse”, Educational Researcher, Vol. 36, No. 5, pp. 25-32
370 ibid., p. 25 371 ibid., p. 25
this was attempted are variable, the example cited above echoes the turn to ‘science’ that has activated some very real problems for social research generally. Grounded Theory exemplifies the point, to the degree that its claims-making posture rests on an avowedly positivist frame. In particular, Grounded Theory is deployed in qualitative research in order to generate ‘theory’ based on ‘data’ drawn from small population samples that are usually non-representative. Widely deployed, yet now more frequently critiqued, the notion that a ‘theory’ can be developed from such small-sample research raises issues of context and meaning, in addition to the more usual challenge about the meaning of theory itself.
In an extensive review, Thomas and James372 follow the trajectory of
Grounded Theory, which began in the 1960’s, at a time when qualitative research was suffering a decline ‘ … principally due to pincer-like pressure from …the ‘‘hard’’ methods such as statistical method and structural functionalism on the one hand, and competition from the ‘‘soft’’ side in the form of ethnomethodology ... on the other’373
. In many respects the project was successful and they note that through the foundational work of Glaser and Strauss … ‘there can be little doubt that it has been a major - perhaps the major - contributor to the acceptance of the legitimacy of qualitative methods in applied social research.
Yet the limitations of Grounded Theory are seldom as closely scrutinized by those who deploy it and extend to four key areas. The first ‘…concerns the effect of highlighting the immediately apparent and observable at the expense of attending to the interweaving of structural features of social situations with activities’. The second, concerns the ‘focus on the here and now of everyday encounters’ (which) limits the concept of power that is possible in the approach.’ The third challenges the development of theory in a grounded theory approach, which should be guided by data rather than limited by it. Finally, ‘…the insistence that grounded theory should be recognizable to the people studied (encapsulated in the notions of ‘fit’ and ‘relevance’) places unhelpful constraints on analysis because it rules out features and
372 Thomas, G. and James, D. (2006) ‘Reinventing grounded theory: some questions about
theory, ground and discovery’, British Educational Research Journal!, Vol. 32, No. 6, December, p. 767
interpretations which they could not be expected to have considered’374.
As the authors note:
…far from providing the epistemic security promised by grounded theory, these notions - embodied in continuing reinventions of grounded theory - constrain and distort qualitative inquiry, and that what is contrived is not in fact theory in any meaningful sense, that ‘ground’ is a misnomer when talking about interpretation and that what ultimately materializes following grounded theory procedures is less like discovery and more akin to invention. The procedures admittedly provide signposts for qualitative inquirers, but…researchers should be wary, for the significance of interpretation, narrative and reflection can be undermined in the procedures of grounded theory375.
What is clear from the examples cited above is that while the place and prominence of ‘data’ and ‘data analysis’ in social research is well established, it often comes at the expense of a full and accurate representation of participant contributions. In that regard, these salutary cautions take us back to the very ‘source’ from which we purportedly draw our data, and while it is necessary to generate a framework within which research can be structured and analysed, the claims attributable to data alone often times stand in isolation from the people and places that ground our research.
With these things in mind I was less inclined to work toward ‘data analysis’ as a feature of this research, instead opting to ensure the representative capture of individual responses to questions about beliefs, intentions and activities. On that basis I chose to elicit themes that emerged from interview.
5. Themes
In acknowledging that interview content in social research is characteristically descriptive I sought to develop a framework within which I could capture any significant iterations of sustainability, along with the meaning and values that drove particular activities or initiatives. Accordingly, the process that was designed to elicit themes involved five stages and took place after all interviews had been completed376. It also followed standard social research
practice377.
374 op. cit., p. 769
375 op. cit., p. 768
376 Note that all interviews were sound recorded
Stage One: Editing Sound Files
All interviews were voice-recorded and transferred to sound files.
Editing complete sound files was undertaken so that they contained relevant interview material that related to all elements of the research. That is, responses to direct questions posed by the researcher, along with any other relevant information that arose in conversation.
Stage Two: Transcribing interview content
This process was extensive and involved listening to and transcribing interview content.
Stage Three: Coding
Coding the content of interviews involved the identification of responses to specific questions, as well as any significant additions that were provided by participants. Content was recorded in initial notes that supported the development of coding categories. For example, specific questions about what sustainability meant from an individual contained a lot of detail that was framed in conceptual categories such as connection to place, human relationship building, and so on
Stage Four: Themes
The themes that emerged from the coding process constitute an array of conceptually grounded areas of interest and involvement. As such, the themes are directly concerned with explaining and accounting for responses to questions. In addition, the themes enable a level of interpretation with regard to values, beliefs and intentions and the ways in which they inform initiatives or activities in the community setting.
Stage Five: Final Edit
A final sound edit was specifically designed to capture the themes that emerged from the coding process. Original, un-edited voice recordings were retained in the event that any content escaped scrutiny.
Stage Six: Participant Feedback
Participant feedback involved checking that the actual voiced content of interviews was an accurate representation of contributions.
This practice is not always considered in the conduct of primary research but I regarded it as an element of ethical practice that should be installed wherever possible. Quite apart from the obvious need to ensure accuracy, people who contribute to our research endeavours do not always get to access them.
(Note: It was not possible in all cases to reach participants and confirm the accuracy of the final cut of the sound edit. For that reason I ensured that I began by editing the sound files of people who were available to review them. This ensured that I would develop a more critical level of practice before commencing on edits of those who could not be contacted. In those cases where respondents were available all affirmed the accuracy of the final-edit sound files.)
6. Researcher Commentary: Reflexivity