Noncontact as a Contributor to Net Nonresponse Bias 74

In order to clarify the contribution component sources make to the net nonresponse bias found in the prior section, an attempt was made to estimate error by response disposition type. First, differences on frame variables were examined. Then, for a number of variables for which no frame data existed, correlation analysis and wave extrapolations were employed in an attempt to generate indications of nonresponse bias direction and magnitude.

3.5.1.

Component Bias as Measured Against Frame Variables

The existence of frame data for a number of key demographic and household variables presented an opportunity to compare differences across the response groups. For each survey, the average value for sample units in each response group was calculated for the same variables in the frame-based analyses undertaken in the previous section. Table 25 presents a summary of the comparisons. It is important to note that the figures presented are averages across the six ISSP studies. That is, they are ‘averages of averages’. For example, the value ‘50’ in the ‘Valid’ column for ‘Average Age’ indicates that, across the six studies, the valid response groups had

an average age of 50. This average comprises the average ages for the valid groups in ISSP01 through ISSP06 of 49, 49, 52, 52, 48 and 49.

It is acknowledged that the presentation of ‘averages of averages’ can obscure substantial differences in individual results. However, the key trends apply in all but a few cases across each of the six individual studies.

Table 25: Average values for frame variables by response disposition

Response Disposition

Frame Variable All Valid Inactive GNA Refusal Ineligible

Proportion of Sample (%) 100 52 32 9 4 3 Average Age 47 50 42 43 59 60 % Male 48 45 52 51 41 45 % Maori Descent 14 11 19 17 9 7 % Employed 59 61 59 58 41 34 % Student 8 6 11 10 2 7 % Retired 11 13 6 8 25 36 Avg. Age (HH) 47 49 43 44 58 61 Avg. # Electors (HH) 3.2 2.8 3.0 3.8 3.0 10.0 Avg. # Surnames (HH) 2.1 1.7 2.0 2.9 2.1 8.4

Note, this includes samples with stratification in 2003 (Maori), 2005 (Age) and 2006 (Age/Gender). The patterns still apply despite this.

The different sources of nonresponse error do appear to exhibit different profiles. Specifically, the refusal and ineligible groups are similar to one another on many variables but differ on gender, occupation and household composition. Furthermore, the GNA and inactive groups are strikingly similar on all variables except household electors and surnames. More importantly, the refusal and ineligible groups tend to contribute opposing bias compared to the GNA and inactive groups. Indeed, for every variable except “% Employed”, the net nonresponse bias across the surveys, as indicated by the difference between the all and valid columns, is in the same direction as that for the refusal and ineligible groups. Said another way, incorporating more refusers or ineligibles in the valid group would typically make

estimates worse. Thus, the net nonresponse bias in the studies examined is attributable to nonresponse from the GNA and inactive groups.

These results lend support to hypotheses three and four; that the component sources would contribute a different profile of nonresponse bias to the studies and variables examined and that, where component sources affect the same variable, a net bias would still arise despite limited offsetting effects.

Moreover, they indicate an opportunity to reduce noncontact bias via methods targeted at noncontact. Given the findings of chapter 2, it is reasonable to assume that the contribution of noncontact to the net nonresponse bias presented in Table 25 is underestimated because not all noncontacts were reported. Indeed, in the Address Change study, the total noncontact rate was estimated to be double the amount reported in GNA figures for an unmessaged treatment after multiple contact waves (12% total vs. 6% reported GNAs, see Table 17, p. 45). Since unreported noncontacts reside in the inactive group, the GNA and inactive groups are so similar in nature, and net nonresponse bias is attributable to an underrepresentation of people from them, the proportion of net bias attributable to noncontact can be estimated at around 40%. Specifically, if reported GNAs represent half of total noncontact then, on average, noncontact accounts for 18% (9%*2) of response to the surveys covered by Table 25. This reduces the proportion attributable to inactives to 23% (32%-9%). Thus, noncontact represents 44% (18%/(18%+23%)) of the total noncontact bias they contribute.

In addition, there are two good reasons to expect that efforts targeted at noncontact nonresponse may be more effective than those aimed at further reducing inactives via common appeals to motivation or interest. First, some studies cited in section 3.2.2 found that general inducements and other design modifications can exacerbate respondent composition disparities. Second, the results in Table 25 reflect response after several follow-up contacts and for surveys following good design practice. Hence, in the absence of reliable and effective targeted inducements, it could be said that those people who remain in the inactive category are unlikely to be swayed by further appeals.

3.5.2.

Nonresponse Bias in Non-Frame Variables

One of the criticisms levelled against bias estimation via frame variable analysis is that there are often only a select number of such variables available, and those are not necessarily related to items of interest in the survey.

Table 26: Relationship between frame and survey variables

Frame variable Correlates with these survey variables:

Age Religiosity, Views on importance of religion, Time at residence, Current employment status, Activity re finding work, Attitude toward men’s/women’s roles in relationships, Attitudes toward women working, Likelihood of having disability, Type of disability, Marital status, Household size, Type of housing, Children in household, Highest level of education, Level of income, Attendance at political meetings, Interest in politics, Attitudes toward marriage/cohabitation, Attitudes toward maternity leave, Feelings of stress, Mother worked while respondent was young, Views on Treaty of Waitangi, Views on strike activity, Views on publicity for extremist ideas, Number of people in daily contact with, Views on alcohol use reduction measures, Views on overweight people.

Gender Household duties undertaken, Level of income, Likelihood of being a homemaker, Work history, Attitudes toward women working, Hours of work, Worked while children less than school age, Gender of closest friend, Work in dangerous conditions, Reason ended last job.

Maori Descent Chances of voting for Maori party, Views of importance of NZ ancestry on citizenship, Views on offshore ownership of land, Views on land claim limits, Views on Treaty of Waitangi, Views on

indigenous governance, Views on Maori language, Views on naming of NZ, Views on foreshore and seabed legislation.

Occupation: Employed Level of income, Marital status, Feelings of stress, Work history, Level of education, Main source of economic support, Seriousness of disability, Number of people in daily contact with.

Occupation: Student Marital status, Job prospects, Want for job, Ever had paid job, Age, Type of housing, Live with siblings/parents, Level of education, Main source of economic support.

Occupation: Retired Marital status, Level of income, Number of children under 18 in household, Age, Type of housing, Live with siblings/parents, Length of residence in current town, Attitudes toward women working, Feelings of stress, Reason last job ended, Main source of economic support, Number of people in daily contact with.

Household Age Most of those identified for the individual age variable, Attitude towards demands from family, Caring for a dependant.

In light of this, a correlation analysis was performed on survey variables from all six surveys to identify those related to items on the frame. Table 26 (above) presents the findings, reflecting variables that had correlation coefficients of magnitude +/-0.25 or higher with the various frame items. As might be expected, age was related to the widest range of survey variables. Yet, the other frame variables were also related to a number of survey items, many of which were not also highly correlated with age. This indicates that the net nonresponse bias found in the frame variables, attributable to the under-representation of noncontacts and inactives, is likely to have also occurred to varying degrees in the range of survey variables identified above.

It is also worth noting at this point that, although there are correlations between the frame variables examined, those relationships vary in strength and direction. Hence, the under-representation problem is a multivariate one. Indeed, evidence from Table 22 (p. 71) suggests that over-sampling on ancestry (ISSP03 – Maori Descent) affects estimates for ethnicity but does not make much difference to bias in age estimates. Similarly over-sampling on age (ISSP05, ISSP06) does not make much difference to estimates of ethnicity. It therefore appears that efforts aimed at mitigating nonresponse should take a multivariate approach rather than relying on improving representativeness of the achieved sample on only one dimension.

Another approach to investigating nonresponse bias in variables beyond those for which frame information exists is to analyse and extrapolate point estimates across waves of response. As noted earlier, there are many potential pitfalls with this technique due to the frailty of the ‘continuum of resistance’ assumptions it relies upon. Nevertheless, it does represent another lens through which to view nonresponse error in ISSP survey items.

Table 27 presents wave-extrapolated point estimates for the surveys and survey variables examined previously, and for which census figures were available. The “projected respondent” extrapolation procedure was used to generate item estimates for a 100% response rate. Filion (1976) describes the procedure as follows:

“Changes observed in an estimated parameter value as the response rate is increased using follow-ups of nonrespondents may be used to predict what the parameter value should be for a 100% response rate. Thus, a linear regression line may be fitted to data depicting an observed variable as a function of the cumulative response rate after each wave of replies. That is to say, fit

Y = a + bx

where Y = observed value of a variable per unit of response based on the respondents up to a given wave of replies

x = cumulative response rate up to a given wave

a,b = regression parameters (intercept and slope)” (p. 403)

Table 27: Wave extrapolated unweighted estimates compared to census

Extrapolated Survey Estimate

Census Result11 Variable 2001 2002 2003 2004 2005 2006 2001 2006 % Male 52 42 45 41 50 52 48 48 % 20-34 Years old 26 26 27 27 *31 *27 29 28 % 65+ Years old 14 10 15 16 13 13 17 17 % Maori Ethnicity 11 11 ^22 14 12 13 11 11 % Marital: Single 24 22 27 24 21 25 31 31 % Bach/PG Qual 14 19 19 24 19 25 10 14 % Income <$20k 38 27 38 38 31 31 49 39 % Income >$50k 14 21 17 17 25 27 13 20 % Not Religious 24 34 27 29 32 36 28 33 % Employed Fulltime 50 51 53 50 52 52 46 48 % 1 Person HH 13 10 10 9 9 10 23 23 % 5+ Person HH 18 18 22 17 18 16 12 12

Note: The 2001-2003 surveys had four waves of contact while the 2004-2006 surveys had three. * The 2005 ISSP was stratified by age group. The 2006 ISSP was stratified by age/gender group. ^ The 2003 ISSP oversampled from the Maori roll.

11

Readers are directed to Appendix section A2.1, p. 174, for a background to the census figures in the table above.

Although it is not immediately evident from the table of extrapolated estimates, a comparison with the pre-extrapolation estimates presented earlier in Table 22 (p. 71) reveals the following:

 Compared to census, the extrapolated estimates generally still underestimate the proportion of 20-34 year olds, ‘Singles’, and 1 person households in the sample. Furthermore they overestimate the proportion of people with a bachelor’s degree, high incomes, and 5+ person households.

 Although the extrapolation moved some estimates closer to the census parameters12, this occurred consistently for only two variables: ‘%20-34 year olds’ and ‘% singles’. Overall, only 27 (38%) of the 72 cells in Table 27 represent an improvement over pre-extrapolation estimates.

 In 17 cases, the extrapolated estimates led to deviations compared to census figures that were in a different direction to those from pre-extrapolated estimates. That is, the extrapolated estimates overshot the census figures. This occurred in every instance for the ‘%65+ year olds’ variable. In some cases, this still led to estimates closer to the census figures. However, in 9 cases it did not.

 Thirty six cells contained estimates that were in the same direction of deviation from census results, but greater than those for the pre-extrapolated estimates. The vast majority of these (30 out of 36) occurred for the survey variables in the bottom section of the table (i.e., from ‘% Singles’ down). The implication is that, for a number of survey-only variables (qualifications, low income, and large household size in particular), later waves attracted respondents that led to more

bias in estimates. This underscores the point made earlier, that while multiple contacts often do improve estimates, this does not occur in every case and, so, care must be taken when employing wave extrapolation to estimate survey bias.

Consistent with the findings of prior research (e.g., Ellis et al., 1970), a number of cells (18 out of 72) did not exhibit linear changes in cumulative estimates across waves, contrary to ‘continuum of resistance’ assumptions. This rose to 32 out of 72 when wave estimates were examined individually, rather than cumulatively.

12

Overall, then, it is not clear that wave extrapolation is a reliable mechanism for estimating the direction or magnitude of nonresponse bias in the studies and variables examined here, let alone a good tool for enabling the isolation of noncontact bias. The fact that the trends in underestimation and overestimation for certain variables identified earlier remained despite extrapolation for nonresponse is at least consistent with this technique’s systematic exclusion of noncontact. Nevertheless, it is not possible to draw any solid conclusions about whether this a key cause of the method’s performance, because there are too many potential sources of error involved.

For instance, measurement and coverage error are likely to account for some of the difference between survey estimates and census figures, which confounds any analysis of bias magnitude. Moreover, because wave extrapolation treats all forms of nonresponse together and is dependent on response rates, it is not clear what portion of the variation it its performance above can be attributed to small sample sizes for later waves, the absence of information relating to noncontacts, or residual differences between later responders and remaining passive refusers. Thus, hypothesis 5 is not supported.

4.

Approaches to Reducing Noncontact Bias