Interpretive research quality evaluation

Commonly accepted positivist criteria for assessing research are limited in their

applicability when it comes to interpretive research. These criteria are most appropriate, and their logic is clear for natural scientific research. Nevertheless, there are researchers like e.g., Yin (2018) who continue to use the classical concepts of reliability, (external) validity, and generalizability (internal validity) also for qualitative research that are in general seen as only applicable to positivist, quantitative research. The validity and reliability of e.g., a survey strategy depend on the number and selection of credible

samples (representativeness) as well as on providing a good approximation to the variables of interest (precision). Participants need to be identified regarding the precise criteria (sample selection). These criteria should characterize the participants in terms of e.g., their professional background and experience, or the sector and companies they are working for.

Researchers like Lincoln and Guba (1985) formulate new names for versions of the previously stated criteria in order to recognize the nature of qualitative research.

Dependability replaces reliability, credibility replaces generalizability, and transferability replaces external validity. A third group of researchers has moved further away from the original criteria towards new concepts in order to ensure and judge qualitative research’s quality. Guba and Lincoln (2005) develop a range of authenticity criteria replacing validity.

Validity is not like objectivity. There are fairly strong theoretical, philosophical, and pragmatic rationales for examining the concept of objectivity and finding it wanting. Even within positivist frameworks it is viewed as conceptually flawed. But validity is a more irritating construct, … it points to a question that has to be answered the one way or another: Are these findings sufficiently authentic

(isomorphic to some reality, trustworthy, related to the way others construct their social worlds) that I may trust myself in acting on their implications? (Guba & Lincoln, 2005, p. 206)

Those authenticity criteria – so called because we believed them to be hallmarks of authentic, trustworthy, rigorous, or “valid” constructivist or phenomenological inquiry – were fairness, ontological authenticity, educative authenticity, catalytic authenticity, and tactical authenticity. (Guba & Lincoln, 2005, p. 208)

Picking up the idea of different quality criteria for interpretive research, Schwartz-Shea and Yanow (2012) carve out the limitations of positivist standards for interpretive research and formulate the criteria for designing trustworthiness. As stated in the decision on the research approach (see section 3.1), situated and contextualized meaning-making interpretive researchers focus on a bottom-up concept development, constitutive understandings of causality, the relevance of the researcher’s identity, and the need for improvisation considering the data co-generated in field relationships. Even though the research process is expected to be flexible, procedural detail planning is necessary. Its discussion is important for evaluating the quality of interpretive research.

Systematicity

Schwartz-Shea and Yanow (2012) identify procedural details such as the relation of

researcher identity to the selection and access to research participants, the researcher’s role and degree of participation, the mapping of research conditions for intertextuality, and the anticipation of evidence forms and the analysis of their relationship to the research

questions as quality evaluation criteria.

Reflexivity

Schwartz-Shea and Yanow (2012) emphasize especially the presence or absence of the following criteria for evaluation:

• reflexivity as the most important criterion acting as a counterpoint to positivistic objectivity,

• data analysis strategies, and • member-checking.

All three criteria are practices in which researchers participate as controls of their own sense-making, and they can be considered as standard criteria for evaluating interpretive research.

Reflexivity refers to the different ways a researcher deals with his own sense-making and influential circumstances in different phases of the research. Thereby, reflexivity takes the researcher’s own characteristics, his scholarly community, and his social setting into consideration. Reflexivity varies at different stages of the research. At the beginning of a research project (design stage), reflexivity means a systematic consideration of the research setting’s characteristics and how this might impact the researcher-participant interactions. Although reflexivity cannot be predictive, an interpretive research

construction depending on the researcher’s identity is dynamic and interactive. Thinking ahead of time about possible identity issues can help when these issues materialize in one way or another.

Being reflexive continues while performing research. This includes the questions on how the researcher’s presence and characteristics consciously or unconsciously influence the

participants and the outcome. Furthermore, the adequacy of the initial research setting including the conceptual framework and possible revisions in the research design during the field phase are also part of the reflective process.

Reflexivity continues after collecting information and impressions from the research implementation phase. Especially gaining impressions and keeping them in notes (so- called field notes) helps with conserving the impressions of conversations, settings, events, or interactions with participants to embed and thus consider these impressions for the overall picture and its interpretation.

In all cases, reflective notes need to be self-consciously tagged as researcher sense- making (as opposed to description, even as interpretive presuppositions mean that “description” is never a mirror but itself a theoretically-informed interpretive act). … What makes reflexivity interpretive – some call this critical reflexivity – is the link to the epistemological matters. This includes the self-monitoring of the

researcher’s own “seeing” and “hearing” in relation to knowledge claims …. This seeing, hearing and feeling produces researcher understandings. (Schwartz-Shea & Yanow, 2012, p. 101)

Methodologically, reflexivity is important for two reasons. Firstly, the researcher can analyse how his own position affects the research presentation and the knowledge he claims to have. Secondly, critical reflexivity leads the researcher to think deeply about how his own historical background and social-cultural context shape the formulation of the research questions and the conceptual framework generated. In these processes, reflexivity is a methodological value that underlines interpretive criteria like sense-making during data generation and analysis, and transparency of knowledge generation. Making reasoning transparent invites to evaluate to what extent the choices of research might have affected the researcher’s knowledge claims.

As the researcher himself is the major instrument for conducting interpretive research (Schwartz-Shea & Yanow, 2012), it is questionable whether he does cherry-picking or not. To disprove this guess, various strategies and techniques were developed to check the sense-making process during data generation and analysis. What these various strategies have in common and what is crucial in the end is the general idea that the researcher

consciously searches for evidence that forces a continuous, self-challenging re-

examination of the initial impressions, theories, and favoured explanations. Moreover, these strategies share the continuous test and revision of initial expectations, and the identification of inconsistencies. Demonstrating awareness of these strategies marks the awareness of the general issue of concern of sense-making during the data generation and data analysis phases.

Because interpretive researchers do not seek to mirror the world, their primary concern in checking their own meaning-making is not focused on “getting the facts right,” as if there were only one version of social reality. Rather, they are looking to articulate various experiences or viewpoints on the topic under investigation, in order to be able to understand the nuances more fully. (Schwartz-Shea & Yanow, 2012, p. 105)

Member-checking

The third criterion for the quality evaluation of interpretive research is member-checking. Member-checking refers to the presentation of written material to the research participants to let them check if the researcher captured their perspective the right way. The sense of member-checking in interpretive research is more than checking whether the researcher captured the facts right, which would imply that there is a single social reality, at least the individual social reality of the participant. Member-checking is rather used in the context of complex research settings that might lead to complex sense-making involving tacit knowledge or situated meanings. Against this background, the researcher may check the accuracy of his situational understanding of the participants’ shared experiences.

Trustworthiness

Interpretive research seeking for sense-making must also deal with a central concern, its trustworthiness. From a methodological perspective, dealing with this concern is

implemented differently depending on the epistemological perspective. Positivism makes use of falsifiability as a powerful design concept. However, the purpose of interpretive research is not model testing and erasing ambiguities, but also understanding ambiguous human meaning-making in a context. Schwartz-Shea and Yanow (2012) denote the

coherence of the analysis, rather than the “goodness” of the model”(p. 108). In order to address the explanatory coherence, which means the adequacy of an explanation or analysis, according to Schwartz-Shea and Yanow (2012), the researcher needs to focus on three points: the consistency of evidence that is from different resources, the way

conflicting interpretations are engaged, the logic with which an argument is developed.

Methodological practices of positivism consider the researcher’s presence and his judgement as problematic, because they might impact on the objectivity of the results. Therefore, these methodological practices include controls that seek to avoid a researcher’s contamination and bias. In contrast, interpretive research methodologies consider the researcher the central instrument of data generation and sense-making. Positivist controls cannot be applied to interpretive research because they contradict each other

epistemologically. But that does not mean that interpretive researchers are limited to their perceptions. Being conscious about partial knowledge and multiple perspectives enables the interpretive researcher to analyse these fragments. Moreover, a reflexive researcher can also think about what is not said – the silence in interviews – and try to investigate its meanings. This makes the difference of interpretive research.

No one can be fully transparent to herself …, and all research endeavors proceed based on some set of presuppositions. The interpretive commitment is to increase understanding of the ways in which the characteristics of individual researchers and their academic communities affect the production of knowledge in the human sciences. Research design that discusses the role of reflexivity in the project communicate this commitment to reviewers and other readers. (Schwartz-Shea & Yanow, 2012, p. 112)

3.3.1.2 Implementation of trustworthiness, systematicity, reflexivity, and