Quality criteria

The question on how the quality of qualitative research can be determined is frequently asked and debated. How can subjective perceptions, everyday knowledge and matters or similar objects in qualitative research be assessed reliably? What criteria should it satisfy? There is a heterogeneous literature on this topic, depending on the research field and paradigm, giving a number of different stances taken by qualitative researchers in relation to this issue. Steinke (2004) identifies three different positions for the evalu- ation of qualitative research:

1. Quantitative criteria for qualitative research 2. Independent criteria of qualitative research 3. Postmodern rejection of criteria

In the first position, which is closely linked to the QCA of Mayring (see Section 4.6.1), Steinke (ibid.) states that it is characteristic for criteria from quantitative research to be transferred to qualitative research. Main criteria, such as reliability, validity and (even though less) objectivity, are adapted and operationalized. For example, “inter-coder re- liability” (Mayring, 2010), suggests that codes should be coded by several people or at least on several occasions. Such a ‘re-test’ should preferably be conducted by others, as well as the researcher (to make a ‘comparison over time’ as suggested by Schreier, 2012). The analysis process is then performed a second time and checked whether it leads to the same results. Even if this was not a main part of the used quality criteria for this study, such a re-test was conducted (by myself) as well to readjust coding (for more information, see Section 5.5), but no major statistical value is given to this issue, since in my opinion, the changing context is vital for the coding. I myself noticed that my own perception of the data changed during the research process.

In the second position, Steinke (2004) identifies adherents with fundamental doubts about the transferability of quantitative criteria to qualitative research. As a starting point when forming appropriate criteria, these adherents take the particular theoretical, meth- odological and procedural character of qualitative research. They discuss aspects like communicative validation, triangulation, validation of the interview-situation and authen- ticity. This is intended to compensate for any one-sideness or distortion, which may result from individual methods or researchers, as well as aiming to create the lowest power distance in research relationship. Within this position, there have been various

attempts to develop alternative criteria, such as those suggested by Lincoln and Guba (1985): trustworthiness, credibility, dependability, transferability and confirmability. To- day, the most used criteria within qualitative research (with differences within different disciplines and research paradigms) nevertheless continue to be validity and reliability, where “validity typically receives more attention than reliability” (Flick, 2014).

In the third position that Steinke (2004) highlights, postmodern rejection, she identifies adherents who generally argue against the possibility of formulating criteria for qualita- tive research. For example, Richardson (2011) postulates that it is impossible to relate criteria to a fixed referential system. Furthermore, Denzin (1990) states that the division between the observer and the observed reality is overcome by the researchers writing in the first person singular, and questions about reliability and validity are not necessary. Steinke’s criteria have been criticized (Lüders, 2010) for not being comprehensive enough, even if they come within a framework for guidance for implementation and “de- rive from a special understanding of qualitative research” (oriented towards grounded theory) (ibid, p. 81).

My own stance is colored by the two last positions that Steinke highlights. From my perspective, research should be comprehensible for other people and it should be in- tended to compensate for one-sideness. One main aim of quality research in social science is to produce descriptions of a social world. Such descriptions facilitate in-depth insights in another way than most quantitative research. All descriptions are, however, bound to a particular perspective and therefore subjective. It can never be disregarded that the social interaction that takes place in the research may create different stories, depending on the interacting people and the context. The researcher gets to observe different things depending on who they are: for example, a 50-year old researcher might not get the same story told by a teenager, as would a 20-year old person. Further, the researcher’s understanding of the observations and issues discussed in interviews is filtered by their previous personal experiences.

Drawing on the second position identified by Steinke, this study was focused on docu- mentation and making the research process intersubjectively comprehensible. This in- cluded a detailed documentation and description of the research process, after trying out the research method in a pilot study (see Section 4.4). During the research process, a research diary (see Section 4.3.5) was written for preparation, follow-up and as a

memory aid. The researcher’s own reflection has always been present, in field notes, memos during coding, transcription rules, and further documentation.

As an observer in a classroom, it is important for the researcher not to try to assume all possible roles, which is often intended in the beginning. Instead, the ethnographic ob- server should try to change attention in the classroom. He or she needs practice in making the unknown and invisible (the implicit) known and visible (Kullberg, 2014). In this study, the pilot study in Germany allowed me to practice in the role as observer and interviewer in the European countries. Prior visits in schools in Chile also had made me familiar with the settings in that country.

Data collection and evaluation has a communicative character (Riemann, 2010) and the interpretation of the data can therefore be conducted in a better way as a communicative process. This is something that I noted early in the research process and therefore for- mal and informal discussions with various people were imperative for the process of reflection and used as a validation strategy (Flick, 2004b). Such discussion supported the entire process, taking place in research groups as well as outside of such groups. The discussions were mainly conducted with other PhD students or academics, but also with other people, in order to overcome my own bias from my previous experiences and knowledge.

In this study, in addition to my own analyses and interpretations, interpretations took place in a research group at the PH Weingarten for the chosen analysis method (For-

schungswerkstatt Qualitative Inhaltsanalyse) in a peer-debriefing25 process. In this

methodological colloquium, mainly the analysis method and the coding of the data were discussed. The focus was on an exchange on data analysis in different content analysis approaches. Parts of the coding were interpreted in the group and alternative interpre- tations other than my own were given space, adding a verbal validity to the analysis. The colloquium was visited especially in phases when I was deeply active with coding, primarily during the 1st _AP.

Presentations were further given in another research group at the PH Weingarten, at- tended by masters and PhD students (Nachwuchskolloquium). In the two last years of

25 _{Drawing on Lincoln and Guba (1985, p. 308): “It [peer-debriefing] is a process of exposing oneself to}

a disinterested peer in a manner paralleling analytical sessions and for the purpose of exploring as- pects of the inquiry that might otherwise remain only implicit within the inquirer's mind”.

the thesis, seminars were given at Stockholm University, which were attended by mas- ters and PhD students and other academics.

The input from the mentioned groups of people proved to be very useful for the analysis procedure, providing external perspectives on things that I might have taken for granted from my own internal perspective. This tweaked the methodology resulting in some changes of the coding of the materials.

Further actions in quality matters were the mixed methods design approach as well as detailed descriptions of the steps taken. This approach in the data collection was part of the validation strategy, applying different perspectives to the research process and the collected data. The sampling strategy used in this study is described in detail, as well as the individual methodical decisions (see Sections 5, 6 and 7).