Reliability and validity are important criteria in evaluating the quality of quantitative research and have been adapted for evaluating the quality of qualitative research in very similar ways (Bryman, 2004). Reliability refers to the reproducibility and consistency of measures. Validity refers to the issue of ‘whether an instrument measures what it aims to measure’ (Bowling, 2002:147) or ‘the extent to which an account accurately represents the social phenomena to which it refers’ (Hammersley, 1990:57). Reliability affects validity. An unreliable instrument has low validity (Bowling, 2002).
Reliability
In this research, measures were taken to improve reliability. All the survey samples were randomly selected using the computer programme Research Randomiser V3.0 (Randomiser, 2006) to ensure that every member of the population of interest had a calculable chance of being selected in the sample and each of the eight groups of service users in the two Integrated Locality Teams in Cambridge City PCT was included to make the sample more representative and to reduce random error (Litwin, 1995) or sampling bias (Bowling, 2002). As I was the only researcher doing the coding there was no problem of ‘inter-observer consistency’ (Bryman, 2004:71),
which refers to lack of consistency in different researcher’s different decisions. I had one of my supervisors check the coding of my first five interviews and compare how she would code my analysis of the same data in order to improve the reliability of the qualitative data analysis. She agreed with my understanding of the data, although she would have given the codes different names. However, from a constructivist point of view, people from different social and cultural backgrounds may have different ways of understanding the world (Burr, 1995) and it is very possible and reasonable for different researchers to have different understandings and interpretations of the same data (Welsh, 2002).
Validity
Several measures were taken to establish the validity of the study:
Face validity, which refers to casual review of an instrument by untrained individuals (Litwin, 1995), was established by asking a staff member of the Integrated Locality Team to look at the questionnaire to see whether on the face of it she thought it made sense.
Content validity, which refers to more formal and systematic assessment of an instrument than face validity by experts (Litwin, 1995), was established by asking two members of staff, one a research coordinator and the other my practice supervisor at Cambridge City PCT, to check the content of the questionnaire. One staff member suggested that I could ask service users whether they had used the service before the organisational change and to compare the services if they had used them both before and since the change. I accepted this suggestion and added it to the questionnaire. My practice supervisor pointed out that service users might not know who their care coordinator was, even if they had one. So I changed the question ‘do
you have a care coordinator?’ to, as she suggested, ‘Who is responsible for your care?’
There are some threats to reliability and validity in research. Some common biases and errors such as sampling bias, acquiescence response set, evaluation apprehension and interviewer bias were constantly eliminated (Bowling, 2002). When I piloted the questionnaire I found that some interviewees answered my questions before I
finished the whole question. In the questionnaire, questions 11, 14, 15, and 18 offered the same set of satisfaction level response: very satisfied, satisfied, neither satisfied nor dissatisfied, dissatisfied, very dissatisfied and not applicable. Question 19 had a different quality response set: excellent, good, satisfactory, poor, very poor and can’t say. Some interviewees immediately answered ‘satisfied‘ to questions 11, 14, 15, and 18 when I had just uttered the first two responses for them to choose from - ‘very satisfied? Satisfied?’. It was the same with question 19. Some immediately answered ‘good’ when I had only uttered the first two responses for them to choose from - ‘excellent? good?’. There may be two reasons for this: first, the interviewees might feel that the word they have heard express what they really feel and so they give their true response, second, because of the ‘acquiescence response set’ or ‘yes-saying’ bias, research participants ‘more frequently endorse a statement than disagree with its opposite’ (Bowling, 2002:153). In order to eliminate the bias I changed the order of the two response sets (see Appendix 4 Questionnaire) and printed two cards in large font with the two reversed response sets (see Appendix 7 Satisfaction level responses card and Appendix 8 Quality responses card). I presented these two cards to every respondent who was to be interviewed to help them to complete the
questionnaire. I found the cards very effective in encouraging them to read the response sets carefully before pointing out their answers to me (Litwin, 1995).
Before the interviews, I told the staff who were to be interviewed that this research was neither funded nor organised by the PCT and that I was an independent
researcher, and I told the service users who were to be interviewed that I was not a member of their care team and that the research was not a part of the care they received. I did not give them any hint of what my own opinion was or what I stood for. I made them comfortable and told everyone that taking part in this research was voluntary and that they did not have to answer any question if they did not want to. I did all this to try to reduce any anxiety which may be generated in people when they are tested, because the anxiety might have caused them to try to give the answer they thought I wanted to hear rather than their true response. This is called evaluation apprehension (Bowling, 2002). During the interviews I tried not to ask any leading questions or to reveal my own opinions in order to reduce the possibility of
interviewer bias.
4 Findings
4.1 Introduction
This chapter presents the findings of the study – the contextual conditions,
implementation processes, causal mechanisms, planned goals and achieved outcomes of the integration of health and social care services for older people in
Cambridgeshire, and explains why some of the intended aims were not achieved. The contextual conditions, implementation processes, causal mechanisms and the
intended outcomes of the integration programme were derived from analysis based purely on qualitative data and the achieved outcomes were derived from analysis of both qualitative and quantitative data (see Figure 4.1).
Figure 4.6 Methods of data analyses
Policy