VALIDITY OF MEASUREMENTS

Validity operates on a completely different plane from reliability; it is possible to have perfectly reliable measurements that are invalid. Validity is defined as the accuracy of the measurement: it is an assessment of the exactness of the measurement relative to what actually exists. To illustrate this concept and its difference from reliability, think of a respondent who is embarrassed by a question about his income. This person makes under £12,000 per year, but he does not want to tell it to the interviewer. Consequently, he responds with the highest category –

‘Over £50,000.’ In a retest of the questions, the respondent persists in his lie by stipulating the highest income level again. Here, the respondent has been perfectly consistent, but he has also been completely untruthful. Of course, lying is not the only reason for invalidity. The respondent may have a faulty memory, may have a misconception, or may even be a bad guesser, which causes his responses to be inexact from reality.

How do you remember the difference between reliability and validity? Think of your wristwatch. If you set your watch incorrectly, it will save 1:00 every 12 hours (reliable), but it will be inaccurate (not valid). But how would you determine whether a person was accurate in his or her response? Naturally, you would need some means of verification. For income, an income tax form or verification from the employer would serve to determine the validity. Of course, either of these items would be very difficult to obtain, but what about an estimate of the person’s income from another family member? Regardless of the source used, this example demonstrates the basic concept in validation: the identical answer must be obtained from a different source of data collection. This approach is known as convergent validation, or assessing the

validity of information by using two different methods to obtain the information.

Convergent validation is described more fully later in this section.

Actually, there are a number of different types of validity involved in measurement.

The ones more commonly considered by marketing researchers are fact, predictive, convergent and discriminant.

Face validity is concerned with the degree to which a measurement ‘looks like’ it measures that which it is designed to measure. It is a judgement call by the researcher and is made as the questions are designed. Thus, as each question is developed, there is an implicit assessment of its face validity. Revisions enhance the face validity of the question until it passes the researcher’s subjective evaluation. Unfortunately, this method of assessing validity is the weakest. It can be strengthened somewhat by having other researchers critique the questions.

Predictive validity pertains to the extent to which a particular measure predicts or relates to other measures. For instance, we might ask respondents to indicate their purchase intentions for onion dip on a probability scale. Those who indicate high likelihoods of purchasing the product or brand should have high incidence of actual purchasing, whereas those with low likelihoods should have a low incidence of actual onion dip purchasing. So we ask them to tell us how many onion dips they have purchased in the last month to validate their responses. You would predict that a respondent who says she dislikes purchasing onion dip would not have purchased as much onion dip as one who likes purchasing onion dip. Those with strong positive attitudes towards a brand should have good things to say about that brand and so forth.

If your logical predictions are supported by the findings, predictive validity has been demonstrated for those measures.

When the researcher uses two different methods or sources of data collection for the same piece of information, convergent validity is being assessed. For example, field research companies often use a telephone validation technique to verify respondents who were interviewed by their interviewers in a high-street mall-intercept study. If the two different methods produce similar results, convergent validity is demonstrated.

In another example of convergent validity, a drop-off survey was delivered by college students to every tenth house in a medium-sized city. The students explained the survey and left the questionnaire with a head of the household. They returned later that day to pick up the completed questionnaires. The convergent validity was assessed by follow-up telephone calls. In the drop-off phase, 386 homeowners responded, and the average age of the home was 14.2 years. Thirty questionnaires were selected at random, and follow-up telephone calls were made to these homes. This time, the other head of the household was questioned, and the average home age was determined as 14.6 years. The closeness of the two averages (14.2 and 14.6) indicates convergent validity. (A statistical test revealed no significant difference.)

Discriminant validity means that dissimilar constructs should differ. In other words, questions that measure different objects should yield different results. When the researcher knows that real differences do exist, he or she should find that the responses differ. Otherwise, there is some doubt as to the validity of the measurements.

Discriminant validity was also assessed in the homeowners’ survey just described.

One question on the survey asked how concerned homeowners were about home

Marketing Research and Market Forecasting

59

security. They were to indicate concern using a scale from 1, meaning ‘not concerned at all’, to 7, meaning ‘very concerned’. Another question asked about their concern about home fire safety using the same 7-point scale. The averages were found to be 4.3 and 5.6, respectively. Thus, they were more worried about fire safety than about break-ins, and this difference seems reasonable because fires can happen any time, but break-ins are more rare. These are two different constructs, and their discriminant validity was demonstrated by the difference between the averages.

How to develop valid measures

What does the marketing researcher do about validity? Face validity becomes second nature to the researcher, and it should be constantly evaluated throughout the design of the questionnaire. Unfortunately, in most cases, face validity is the only assessment used to determine the validity of research measures.

There is very good reason to be vitally concerned about reliability and validity of measurements in marketing research: if a measure is unreliable or invalid, the entire study falls apart. Consequently, we have provided a quick reference summary of the various tests discussed in this section in Table 3.2.

Problem I

In conducting a survey for the Equitable Insurance Company, Burke Marketing Research assesses reliability by selecting a small group of respondents, calling them back, and readministering five questions to them. One question asks, ‘If you were going to buy life insurance sometime this year, how likely would you be to consider the Equitable Company?’ Respondents indicate the likelihood on a probability scale (0–100 per cent likely). Typically, this test–retest approach finds that respondents are within 10 per cent of their initial response, that is, if a respondent indicated that he was 50 per cent likely in the initial survey, he responded in the 45–55 per cent range on the retest.

Table 3.2 Criteria for reliability and validity tests used in marketing research

Test Criterion

Test–retest reliability Most respondents give an identical response to the same question administered with some time interval between.

Equivalent forms reliability Most respondents give an equivalent response to near identical questions administered at different times.

Split-half reliability The responses of one-half of the sample are highly similar to the responses of the other half when key questions are compared.

Face validity The researcher judges that a question ‘looks like’ it is measuring what it is supposed to measure.

Predictive validity One measure that logically should predict another measure is found to do so statistically (usually correlation).

Convergent validity Two completely different methods of measuring the same variable are found to yield similar statistical findings (usually means or correlation).

Discriminant validity One measure that logically should not be related to another measure is found not to do so statistically (usually correlation).

Problem II

General Food Corporation includes Post, which is the maker of Fruit and Fibre cereal.

The brand manager is interested in determining how many Fruit and Fibre consumers think it is helping them towards a healthier diet, but she is very concerned that respondents in a survey may not be entirely truthful about health matters. They may exaggerate what they really believe so they ‘sound’ more health conscious than they really are, and they may say they have healthy diets when they really do not.

The General Foods Corporation marketing research director provides a unique way to overcome the problem. She suggests that they conduct a survey of Fruit and Fibre customers in Pittsburgh, Atlanta, Dallas, London, Birmingham, Bristol and Manchester and Denver. Fifty respondents who say that Fruit and Fibre is helping them towards a healthier diet and who also say they are more health conscious than the average Briton will be selected, and General Foods will offer to ‘buy’ their groceries for the next month.

To participate, the chosen respondents must submit their itemised weekly grocery trip receipts. By reviewing the items bought each week, General Foods can determine what they are eating and make judgements on how healthy their diets really are.

Questions

1 The survey has been going on for four weeks, and it has two more weeks before the data collection will be completed. Respondents who are retested are called back exactly one week after the initial survey. In the last week, reliability results have been very different.

Now Burke Marketing Research is finding that the retest averages 20 per cent higher than the initial test. Has the scale become unreliable? If so, why has its previous good reliability changed? If not, what has happened, and how can Burke Marketing Research still claim that it has a reliable measure?

2 What is your reaction to General foods approach? Will General Foods be able to assess the validity of its survey this way? Why or why not?

(I) Problem 3.3 Computer-administered survey Introductory comments

As our introductory example illustrates, computer technology represents a viable option with respect to survey mode, and new developments occur almost every day.

Although person-administered surveys are still the industry mainstay, it is important to discuss computer-administered survey methods as we are certain that this approach will grow and become common in the foreseeable future. Computer-assisted surveys are in an evolutionary state, and they are spreading to other survey types. For instance, a computer may house questions asked by a telephone interviewer, or a questionnaire disk may be mailed to respondents for self-administration. Basically, a computer-administered survey is one in which computer technology plays an essential role in the interview work. Here, either the computer assists an interview, or it interacts directly with the respondent. In the case of Internet questionnaires, the computer acts as the medium by which potential respondents are approached, and it is the means by which respondents return their completed questionnaire. As with person-administered surveys, computer-administered surveys have their advantages and disadvantages.

Marketing Research and Market Forecasting

61