In addition to considering the purpose of data and its relationship to research objectives and any hypotheses, another key issue in data collection relates to what we have termed here the quality of data and in particular what constitutes effective data in the context of a research/consultancy exercise. Perhaps as you would expect, data can, and does, vary enormously with regard to quality and the researcher must understand, therefore, what constitutes the different dimen-sions or criteria for assessing the quality of data.
Overall, effective data provides the basis for the information required to meet the objectives of the research project. Underpinning the seemingly simple perspective as to what constitutes effective data are several potentially com-plex dimensions or criteria of quality when it comes to assessing data. Of particular importance with regard to dimensions or criteria of data quality are the dimensions of what researchers refer to as validity, reliability, and generalizability.
● Validity: Validity relates to the extent to which the data collection method or research method describes or measures what it is supposed to describe or measure. Ghosh and Chopra (op. cit.) define validity as ‘an absence of self contradiction’. Clearly, validity as a dimension or criteria of data quality is crucial. After all, if a research methodology or instrument does not measure or describe what it is supposed to, then at best it is possibly meaningless and at worst misleading. If we are attempting to measure attitude, for example, then it is important that our approach to data collection actually allows us to measure this dimension of behaviour. Although this sounds obvious, in fact validity is a complex concept and there are many different dimensions to, and types of, validity including for example: ‘content validity’, ‘predictive valid-ity’, ‘concurrent validvalid-ity’, ‘construct validvalid-ity’, ‘face validvalid-ity’, ‘internal and external validity’, and ‘statistical validity’ (Burns 2000; McBurney and White 2004). Most methodologies and instruments will enable us to assess the extent to which they are likely to be valid as a method of data collection, though
again, there are several dimensions to such assessments. Overall, as already stated, the researcher must assess the extent to which an approach to the collection of data will produce data that is valid.
● Reliability: Reliability relates to the extent to which a particular data collec-tion approach will yield the same results on different occasions. Perhaps we should point out, of course, that this assumes that there are no real changes in what is to be measured or the circumstances of such measurement. Where data is unreliable then we obviously have to be careful in carrying research results from one situation to another.
● Generalizability: Generalizability is essentially another dimension of validity quality in data and relates to the extent to which results from data can be generalized to other situations. Generalizability with regard to data is par-ticularly important with regard to two aspects of data.
The first is where data has been collected on the basis of a sample. As we shall see in later chapters, sampling is often used in the generation of data in the process of research and consultancy projects. The researcher must know, or be able, to measure/assess the extent to which results from the sample will also be present in the wider population from which the sample is drawn.
Generalizability, therefore, is related not only to the methods of data collection and the circumstances thereof, but also to issues such as sample design and sampling method, etc.
The second, and related aspect to generalizability, is the extent to which the data and results of a particular research project can be generalized to other sit-uations. This, of course, is crucial in developing theories and particularly in the deductive approach to research.
In fact, all three of these dimensions of data quality very much stem from the need within the deterministic model of research to produce results that are sci-entific, and reproducible. As Easterby-Smith et al. (2002) point out: ‘the notions of validity, reliability and generalisability were in fact originally developed for use in the physical sciences and particularly where quantitative data was being collected’. However, as they also point out, these dimensions of data quality are just as relevant and therefore can also be applied and interpreted in the con-text of more qualitative research methods and techniques. Their ideas are shown in Table 6.3 linked to the notions of inductive versus deductive research approaches which were considered in Chapter 3.
These are the three key criteria or dimensions for the quality and effectiveness of data. There are, however, a number of other dimensions that can also have an important effect on data quality. These include:
● Sampling and measurement errors: Both types of errors can, and do occur in data collection with perhaps obvious potential effects on validity, reliability and generalizability.
● Data recording, storage and retrieval: Issues in data recording, storage and retrieval can have important effects on data quality. The researcher must give careful consideration as to how data will be recorded and stored. Sometimes
for example, data recording, storage and retrieval can prove problematical.
An illustration would be where, say, the researcher is collecting data through personal interviews. An ideal way of recording, storing and retrieving data from the research might be to use, say, a voice recorder while conducting the interviews. However, the use of a voice recorder might inhibit some inter-viewees in terms of being more guarded about what they say, thereby affect-ing the quality of the data collected. On the other hand, dispensaffect-ing with the voice recorder and using, say, handwritten notes during the interview might encourage interviewees to open up more during the interview, but would be much more difficult to retrieve subsequently, and the act of recording hand-written notes might slow down and affect the spontaneous nature of the inter-view. Ethical issues would, of course, be a consideration if the interviewer was to use a voice recorder without letting the respondent know this before-hand. The researcher must, therefore, give careful consideration as to how data is to be captured and recorded and how it is to be stored and retrieved.
● Preparation for data gathering: Data quality can also be affected by the extent to which the researcher has prepared for the data-gathering process. In particular, is the preparation required to ensure that the researcher is famil-iar with the necessary research and measuring instruments being used. An otherwise faultless data-collection design methodology may suffer simply because the researcher has not prepared sufficiently in advance in terms of applying and administering the research instruments for data collection.
Deductive research Inductive research
Validity Does an instrument Has the researcher
measure what it is gained full access to the supposed to measure? knowledge and
meanings of informants?
Reliability Will the measure yield Will similar
the same results on observations be made different occations by different researchers (assuming no real on different occasions?
change in what is to be measured)
Generaliz- What is the probability How likely is it that ability that patterns observed in ideas and theories
a sample will also be generated in one setting present in the wider will also apply in other population from which settings?
the sample is drawn?
Figure 6.1 Questions of reliability, validity and generalizability in deductive versus inductive research methods
Source: Adapted from Easterby-Smith et al. (2002) p. 53.
6.5 Choosing between data collection methods
A key decision is the selection of the research methodology and techniques. The researcher needs to understand not only which alternative methodologies and techniques are available, but also the main criteria in selecting between them for a particular research project. This in turn requires an understanding of the characteristics, uses, advantages and disadvantages of the main alternative research methodologies and techniques of data collection. You might be won-dering then, since we have not yet considered the major alternative research methodologies in detail how we can at this stage usefully introduce the issue of how to evaluate and choose between these alternatives. In fact, although the final choice of research methodologies does indeed require the researcher to be familiar with the characteristics of the alternative methodologies, it is possible and useful at this stage to outline the key criteria when selecting between these alternatives. These are now outlined: