Reliability, Validity, Reflexivity, Generalisability

Whilst these concepts: reliability, validity, generalisability and reflexivity have contrasting meanings and implications for the research, Silverman (2013) grouped them as indicative measures of the research quality and credibility. Each concept is now discussed in relation to this research project.

4.13.1. Research Reliability

Reliability is defined as “…the extent to which research is consistent in what it measures” (White and Denholm, 2011, p. 235). In this sense, validity and reliability are inextricably linked. Silverman (2013) recognised that reliability and validity can often be confusing as terms for a researcher so offered a clear distinction between them: reliability refers to the degree of consistency from the interview participants and validity refers to the accuracy of data collected and the interpretation of that data.

“A first requirement is that researchers do everything they can in terms of methodological rigour to ensure that their accounts are accurate and that their interpretations are reasonable”

(Buchanan and Bryman, 2009. p.424) Literature Review Oct 2008- Dec 2009 Data Collection Feb 2009- Oct 2011 Interrruptions of Study Aug 2011 - Jan 2012 July 2012 - Dec 2012 Analysis Dec 2012 – Feb 2013 Writing Up Mar 2013 – Sept 2013 Viva Voce Mar 2014 Correction Period April 2014 – March 2014

Silverman (2013) warned against using measures and scales for qualitative research as it contradicts the very purpose of a method which benefits field notes or transcripts, where deeper explanations of causes and effects may be presented. The research model presented at the end of Chapter II suggests such measures and scales could be used. Whilst the research model was used for planning the interview questions, the interview questions were deliberately kept broad in approach (semi-structured) and all qualitative data collected from the main research interviews between 2009 and 2011 was later matched against the variables in the model, such as commitment, leadership and trust.

In reflection of the decision not to record interviews, this meant that there was greater pressure as a researcher in ensuring all facts were correct and important points were not missing. Emailing the notes back to participants was a form of verifying and triangulating the data. This method of triangulation was only one of several approaches used in ensuring interview data was reliable.

4.13.2 Triangulation

Rozemeijer (2000) noted that method triangulation originated from shipping navigation where several, as opposed to one, location points were used to calculate the exact location of a ship at sea. The author identified four ways for adopting triangulation across empirical research: method triangulation, source triangulation, data triangulation and research triangulation. Each approach is classified in Table 4.14 and deemed relevant for this research except for research triangulation. Several researchers need to be involved in data collection which makes it unsuitable for doctoral research where an entirely individual research approach is taken.

TRIANGULATION METHOD DESCRIPTION

Method triangulation Combining face-to-face interviews with observations and industry documents

Source triangulation Combining several sources of data taken from more than one interviewee

Data-triangulation Handling large amounts of data taken from different perspectives Research triangulation Individual observations from a number of different researchers are

compared

Table 4.14: Four Approaches to Triangulation (Developed from Rozemeijer, 2000)

Data triangulation and method triangulation have been widely supported in the social sciences by authors such as Mason, (1996), Silverman, (2000), Robson, (2002), Bryman and Bell, (2007), Matthyessens, (2007) and Yin (1994; 2003; 2011; 2014). Glaser and Strauss (1999, p.17) argued that method triangulation led to “richer constructs” for “the road ahead”. Robson (2002) argued that method triangulation reduced the threats associated with research validity and Danermark et al. (2002)

claimed that combining different research methods and instruments was a powerful approach for improving the reliability and validity (internal and external) of research, otherwise termed as triangulation. Both, method and data triangulation approaches have been used for this research. Yin’s (2014) case study research included guidance on source triangulation. The author distinguished between convergence and non-convergence of multiple sources of evidence. Yin stressed that a key point for researchers was ensuring evidence was collected from as many sources as possible, such as site visits, interviews and surveys. The company documents may then be merged for the overall findings and should not treated separately. Taking these arguments forward, with the aim of improving the reliability of the data collected, source triangulation and data triangulation were utilised for this research project.

Triangulating of interview documents ensured reliability and validity of the information taken but it also gave participants opportunity to add further comments. This approach allowed interview participants to challenge areas which may be inaccurate or skewed, in particular this was relevant when extracting information from company reports.

4.13.3 Research Validity

Yin (2014) classified research validity in four ways: construct validity, internal validity, external validity and reliability. He stressed that research validity would become more of an issue when applying multiple case studies. The methods, research questions and research purpose needed to be consistent throughout. White and Denholm (2011) distinguished between internal and external validity; internal validity ensures constructs are measured appropriately: “the extent to which a test appears to measure what it is supposed to measure” (ibid. p. 234) and external validity is otherwise known as generalisability and is “a measure of the extent to which the results can be extrapolated to another setting” (ibid.).

4.13.4 Generalisability

As already noted the terms generalisability an external validity are often used interchangeably to mean the same thing. Silverman (2013) noted that qualitative case study findings can be less generalisable than quantitative methods due to the differences in sample size. Dismissing this potential problem Silverman (2013) emphasised that quality research should be a result of generalising theoretical propositions, not judged by the research population. The nature of this research has been to examine strategic orientation in the context of manufacturing. Whilst the UK touring caravan manufacturing industry offers empirical evidence, the main contribution offered through this research are conceptual developments for understanding and measuring a firm’s supply chain orientation (SCO).

For this research, by probing deeper, almost to an ethnographic level, rather than gathering large amounts of data from a broad sample it was hoped more meaningful insights for conceptual development could be revealed. The research model featured in Chapter II, Figure 2.18 and Figure 2.19 is proposed in Chapter VII as generalisable outside of the UK touring caravan industry and even away from manufacturing.

Interview data collected in December 2014 and January 2015 was aimed at meeting two objectives: triangulating the findings from 2009-2011 and a preliminary test of the research model in preparation of a quantitative study post PhD.

4.14 Research Ethics

Mason (2007) emphasised that research needs to be conducted as morally as possible to protect both the researcher and the participants. Ethical considerations were crucial and were discussed at the opening of all interviews. This prompted wider discussion about participant anonymity and also helped to clarify the role of the researcher. Offering consultancy was a frequent misconception by many participants during the initial introductions. An ethical approach to quality research ensured that interview participants were approached with sensitivity and interviews were conducted confidentially and managed appropriately, recognising that interview participants were competitors.

Ethical approval was sought in adequate time for conducting the research in 2008 and again in 2010. Each actor received a formal consent form prior to the study which provided an overview of the research, the supervising team contact details, the researcher’s contact details and it reminded them of the opportunity to withdraw at any time. One of the key challenges of mixed methods researched was highlighted by Brown and Hedges (2009) and Merten, (2011) as new knowledge being created that further challenged the researcher’s ethical responsibilities of data handling. This point was confirmed when given confidential data sets of caravan manufacturing output, both historical and current. This information was held in a password protected file in ensuring confidentiality.

Gatekeeper - one of the important learning points for planning this research project was the importance of identifying the appropriate gatekeeper. This was a previously unexplored industry for academic research which meant gatekeepers needed to be identified; their approval was crucial to this research (Robson, 2002; Oliver, 2003, Saunders et al. 2007). In this research the gatekeepers were selected as chairmen, senior directors: the key players or those with higher strategic interest in the industry. Some difficulties were faced in 2010 when a letter was received detailing a complaint about the research process. Fortunately, this matter was quickly resolved as the company had previously been contacted but the incident quickly reinforced the need and importance to check for more than one gatekeeper in ensuring the research process was conducted in a smooth and ethical manner. However,

this confirmed Goulding (2002) and Mason’s (2007) view that the qualitative researcher is merely a research tool who needs to be continually highly and actively engaged.

4.15 Research Limitations

Three main research limitations have been identified for this research. Each is now explained:

This research has focused on a single country and industry: the UK touring caravan manufacturing industry. Given more time and resource it would have been useful to make a comparison across industry sectors in the UK and across nations such as touring caravan manufacturing in the UK and Germany. The timing of the main data collection (2009-2011) was during an exceptionally economically challenging period which was severely affecting the operations and strategic management of UK manufacturing. The UK touring caravan manufacturing industry was no exception being affected by these challenging conditions; both the manufacturing sector and supplier market was contracting at a concerning rate. Due to this sensitive timing a decision was made to not record the interviews. Whilst the content of the written reports was confirmed by interview participants after each meeting it is inevitable that not all interview discussions were noted in the reports which were based on memory or industry reports and typed up following each visit.

Researching from a critical realist perspective has required a more in-depth, zoom lens examination rather than a wide-angled approach which may have limited the generalisability of the research outcomes.

4.16 Summary

This chapter has presented and evaluated a four phased structured approach to the research design (adapted from Ho et al. 2010). The research philosophy and research design (methods, strategic choices and tools) adapted to support the research project has also been presented and evaluated.

Figure 4.13 portrays how the research questions are argued as central in determining the research methods, the conceptual framework under examination, the research goals and the research validity (Maxwell, 2012).