Non-Random Sampling Methods

In addition to the above methods, which are based to a greater or lesser extent on random probabilistic sampling, there are a number of other sampling methods which are not based on random sampling. The two principal methods are quota sampling and expert sampling.

Quota sampling, as the name suggests, is based on the interviewer obtaining responses from a specified number of respondents. The quota may be stratified into various groups, within each of which a quota of responses must be obtained. This method, for example, is often used when interviewing passengers disembarking from aircraft or other transport modes and for many types of street interviews where passers-by are stopped and asked questions. The major problem with quota sampling is not that quotas are used for each sub-group (after all, this is the basis of stratified sampling), but that the interviewer is doing the sampling in the field and this sampling procedure may be far from random, unless strictly controlled. Left to themselves, interviewers will generally pick respondents from whom they feel they will most readily obtain a response. Thus passers-by who appear more willing to cooperate, are not in a hurry, and are of a

social class comparable to the interviewer will more likely be interviewed. In a household survey, households which are closer to the interviewer's residence (and hence require less travel to reach), households whose members are more often at home, and households without barking dogs are more likely to be interviewed. Such preferential selection can often cause gross biases in the parameters to be estimated in the survey.

Expert sampling, on the other hand, takes the task of sampling away from the interviewer and places it in the hands of an "expert" in the field of study being addressed by the survey. The validity of the sample chosen then relies squarely on the judgement of the expert. While such expert sampling may well be appropriate in the development of hypotheses and in exploratory studies, it does not provide a basis for the reliable estimation of parameter values since it has been repeatedly shown that people, no matter how expert they are in a particular field of study, are not particularly skilled at deliberately selecting random samples. A more appropriate role for the expert in sample surveys is in the definition of the survey population and strata within this population, leaving the task of selecting sampling units from these strata to the aforementioned random sampling methods.

4.5 SAMPLING ERROR AND SAMPLING BIAS

Despite all our best intentions in sample design, the parameter estimates made from sample survey data will always be just that: estimates. There are two distinct types of error which occur in survey sampling and which, combined, contribute to measurement error in sampled data.

The first of these errors is termed sampling error, and is the error which arises simply because we are dealing with a sample and not with the total population. No matter how well designed our sample is, sampling error will always be present due to chance occurrences. However, sampling error should not affect the expected values of parameter averages; it merely affects the variability around these averages and determines the confidence which one can place in the average values. Sampling error is primarily a function of the sample size and the inherent variability of the parameter under investigation. More will be said about sampling error when techniques for the determination of sample size are discussed.

The second type of error in data measurement is termed sampling bias. It is a completely different concept from sampling error and arises because of mistakes made in choosing the sampling frame, the sampling technique, or in many other aspects of the sample survey. Sampling bias is different from sampling error in two major respects. First, whilst sampling error only affects the variability around the estimated parameter average, sampling bias affects the value of the

average itself and hence is a more severe distortion of the sample survey results. Second, while sampling error can never be eliminated and can only be minimised by increasing the sample size, sampling bias can be virtually eliminated by careful attention to various aspects of sample survey design. Small sampling error results in precise estimates while small sampling bias results in accurate estimates. The difference between these two sources of error is sometimes confused, with attention being paid to reducing sampling error while relatively little attention is paid to minimising sampling bias. In an attempt to underscore the difference between the two concepts, consider an analogy with rifle marksmanship as illustrated by the targets shown in Figure 4.8.

A C C U R A T E I N A C C U R A T E P R E C I S E I M P R E C I S E

Figure 4.8 The Distinction between Accuracy and Precision

These targets illustrate four essentially different ways in which rifle shooters may hit the target. The top left target shows a marksman who consistently hits the bullseye. The bottom left shows one who centres his shots around the bullseye but also tends to spray his shots; he seems to be able to aim at the right point but tends to suffer from slight movement of the rifle at the last moment so that his shots are not consistent. The top right target shows the results of a marksman who consistently misses the bullseye; he holds the rifle rock-steady but unfortunately he is aiming at the wrong point on the target, maybe because the telescopic sights on the rifle are out of adjustment. The bottom right shows a shooter who appears to be aiming at the wrong point, but because he also suffers from nervous jitters he sometimes hits the bullseye even though he is not aiming at it. These four situations may be categorised in terms of the precision and the

accuracy of the shots; precise shooters always hit the same spot, while accurate shooters aim at the right point on the target.

It is fairly clear which of the four shooters would be regarded as the best; the top left shooter shoots with both accuracy and precision in that he consistently hits the bullseye. It is also probably safe to say that the bottom left shooter is the second best in that he is at least on target (on average). However, it is not quite so clear which of the remaining two are the worst. Is it better to be consistently off-target, or inconsistently off-target (where at least you have some chance of hitting the bullseye)? This judgement of the quality of marksmanship is made more difficult when the bullseyes are removed to leave only the holes left by the rifle shots, as shown in Figure 4.9. In this case, it is difficult to say whether the top left or the top right group of shots came from the better marksman. Indeed, one may argue that both groups are equally good. In the absence of any knowledge about where the marksmen were aiming, one is more readily swayed by the precision of the shots in judging the quality of the shooter. Indeed, the top right group of shots is now vying for the best group of shots, whereas in Figure 4.8 it was vying for being the worst group of shots.

A C C U R A T E I N A C C U R A T E P R E C I S E I M P R E C I S E

Figure 4.9 The Confusion between Accuracy and Precision

The above description of the marksman can be applied, by analogy, to the design and use of sample surveys. A precise survey is one which displays repeatability; that is, if administered on repeated occasions under similar conditions it will yield the same answers (irrespective of whether the answers are right or wrong). On the other hand, an accurate survey is one which displays validity, in that the

survey is aimed at a correct sample of the correct target population. The precision of a sample survey can be increased by increasing the sample size so as to reduce the possibility of unobserved members of the population having, by pure chance, characteristics which are different to those observed. The accuracy of a sample survey can be increased by ensuring that, first, the sampling frame does not systematically eliminate some members of the population and, second, that the sample is obtained from the sampling frame in a truly random fashion.

Much attention is often paid to reducing sampling error (i.e. increasing precision) by means of elaborate sampling designs and large sample sizes. Relatively little attention, however, is generally paid to increasing accuracy by means of reducing sampling bias to ensure that the questions are being asked of the right people. We are often guilty of "Type-III Errors", described by Armstrong (1979) as "good solutions to the wrong problems". By simply increasing sample sizes, and not paying attention to the quality of the sample, we can always ensure that we will be able to spend enough money to get precisely wrong answers! Indeed, by analogy with Figure 4.9, when we do not know much about the true population we are trying to survey, then we assume that a precise answer is better than an imprecise answer, irrespective of whether it is accurate or not.

In an attempt to improve the accuracy of sample surveys, we therefore need to be more aware of the likely sources of sampling bias (the issue of increasing accuracy by improving survey instrument validity will be discussed in Chapter 5). Some common sources of sampling bias include:

(a) Deviations from the principles of random sampling including: the deliberate selection of a "representative" sample which results in too many observations at the extremes of the population distribution; the deliberate selection of an "average" sample which results in too many observations near the average value and not enough at the extremes; the initial selection of a random sample from which the investigator discards some values because they are not considered to be random.

(b) Use of a sampling frame whose characteristics are correlated with properties of the subject of the survey, e.g. using a telephone interview survey to obtain car ownership rates. Richardson (1985) shows the effect of sampling bias in telephone surveys, and suggests means of correcting for this bias.

(c) Substitution sampling, where the interviewer in the field changes the specified sample because of difficulties experienced in obtaining the originally selected sample (e.g. barking dog in front garden, lack of time to reach next specified household, non-response from selected household).

(d) Failure to cover the selected sample. This can result in bias if those sampling units left unsurveyed are atypical of the total sample. For example, in a household survey of transit usage, where the interviewer uses transit to reach the survey area, lack of time may result in those households closest to the transit line being surveyed whilst those farther away are left unsurveyed.

(e) Pressures placed on interviewers by the method of payment adopted. If payment is by interview completed, then there is an incentive to the interviewer to complete as many interviews as possible in the shortest possible time. This increases pressures for substitution sampling and also encourages the interviewer to complete each interview as quickly as possible. For travel surveys, this will result in fewer trips being reported by respondents during the interview. If payment is by the hour, then the reverse incentives are present. While more desirable than hurried interviews, the expenditure of large amounts of time on each completed interview inevitably means that interviewer productivity must fall.

(f) Falsification of data by interviewer when the interview has not even been conducted. Where such falsification is caused by difficulties in contacting the respondent, then sampling bias may be introduced.

(g) Non-response effects. The bias introduced by non-response varies with the type of survey method used. Thus for mail-back questionnaires, non- response is generally an indication of a low level of interest in the subject of the survey by the non-respondent. For self-completion travel surveys, this often means that non-respondents travel less than respondents, at least for the purpose and/or mode which is the subject of the survey. Empirical verification of this trend can be found in Brög and Meyburg (1981) and Richardson and Ampt (1994). On the other hand, for personal interview surveys, non-response and in particular non-contact is generally more of a problem for those respondents who are more mobile, and hence less often at home to be contacted by the interviewer (see Brög and Meyburg, 1982). Thus for self-completion mail-back questionnaire surveys, non-response will bias travel estimates upwards whereas for personal interview surveys, non-response will bias travel estimates downwards.

Whilst all the above sources of bias are potentially serious, there are a number of safeguards against the introduction of sampling bias including:

(a) Use a random sample selection process and adopt, in full, the sample generated by the process.

(b) Design the survey procedure and field administration such that there is no opportunity or need for interviewers to perform "in-field" sampling. (c) Perform random call-backs on some respondents who have been

interviewed to check on the accuracy of the data obtained and the adherence of the interviewers to the specified random sample.

(d) Perform cross-checks with other secondary sources of data to check on the representativeness of the respondents.

(e) Make every attempt to increase response rates by means of reminder letters for self-completion, mail-back surveys and repeated call-backs for personal interview surveys.

(f) Attempt to gain as much information about the characteristics of the entire sample (i.e. in terms of Figure 4.9, try to know the target at which you are shooting) by identifying the characteristics of non-respondents so that adjustments can be made to the survey results to account for the degree of non-response (see Brög and Meyburg, 1980; Meyburg and Brög, 1981).

One final point with respect to sampling bias is that it will vary with the type of survey method being used and with the parameters which the sample survey seeks to estimate. Only careful consideration of the individual circumstances will determine whether significant bias is likely to exist in the survey results. In all cases, however, it is only possible to correct for sampling bias if sufficient effort has been made to gather information about the entire sample and the entire population which the sample purports to represent.

4.6 SAMPLE SIZE CALCULATIONS

Of all the questions concerned with sample design, the one most frequently addressed is that of required sample size. As mentioned earlier, one way of reducing sampling error is to increase sample size; the question remains, however, as to how much one should increase sample size in order to obtain an acceptable degree of sampling error. This section will attempt to provide some guidance on this matter, particularly for the case of simple random sampling. Estimation of required sample sizes for other sampling methods rapidly becomes more complex (see, for example, Kish 1965) and will not be covered in detail in these notes.

In discussing required sample sizes for simple random samples, it is emphasised that guidance only can be given. Much to the chagrin of many investigators, no firm rules can be given for sample size calculations for use in all circumstances. Whilst the calculations are based on precise statistical formulae, several inputs to

the formulae are relatively uncertain and subjective and must be provided by the investigator after careful consideration of the problem at hand. Importantly, it is often difficult for the survey designer to convey the nature of sample size calculations to clients, who are most often ignorant of the statistical concepts involved. This chapter will attempt to provide some assistance in conveying these concepts to clients with little or no statistical background.

The essence of sample size calculations is one of trade-offs. Too large a sample means that the survey will be too costly for the stated objectives and the associated degree of precision required. Too small a sample will mean that results will be subject to a large degree of variability and this may mean that decisions cannot reliably be based on the survey results. In such a situation, the entire survey effort may have been wasted. Somewhere between these two extremes there exists a sample size which is most cost-effective for the stated survey objectives.

In the context of survey objectives, it is useful to distinguish between two broad purposes for which survey data may be collected:

(a) The main purpose is often to estimate certain population parameters, e.g. average person trip rates, car ownership, mode split, etc. In such cases, a sample statistic is used to estimate the required population parameter. However, because all sample statistics are subject to sampling error, it is also necessary to include an estimate of the precision which can be attached to the sample statistic. This level of precision will be affected,

inter alia, by sample size.

(b) A second purpose of a survey (or surveys) may be to test a statistical hypothesis concerning some of the population parameters, e.g. are there significant differences in trip rates in different areas, or has mode use risen following introduction of a new transport service? To test such hypotheses, it is necessary to compare two sample statistics (each being an estimate of a population parameter under different conditions), each of which has a degree of sampling error associated with it. The tests are performed using statistical significance tests where the power of the test is a function of the sample size of the survey(s).

Whilst the use of sample survey data to fulfil each of these objectives requires different statistical techniques, they are linked by a common usage of the concept of standard error. This concept will now be described, initially with reference to the former objective of sample surveys - that of obtaining population parameter estimates.