Question Order

The impact of question order on the survey responses was not as dramatic as in the ABC survey of 1996, but similar to the General Social Survey. Indeed, the control group generated 10.8 percent inconsistent respondents – only slightly more than the proportion of confused respondents produced by the 1996 GSS (9.7%) (Carlson 2000). The control group mimicked the GSS question order, mixing the questions somewhat randomly and then putting the “any reason” question at the end of the battery of abortion questions. The first experimental group, known as GSU1, ranked the abortion questions in a Guttman scale order of severity, starting with the

question about the women’s health endangered, followed by the questions about a strong chance of a defect and about rape or incest and then the three “hard” questions about the family not being able to afford more children, the woman not being married and the married woman not wanting more children, ending with the any reason question. This question order produced the least amount of inconsistency, with only 8.5 percent falling in the inconsistent category. This order did produce a depressed number of pro life positions, however, with only 9.0 percent saying no to every question, significantly fewer than the 15.3 percent for the control group (z=1.931, p<.10) and 15.0 percent for GSU2 (z=1.844, p<.10).

The second experimental group, called GSU2, followed the ABC survey question order, putting the any reason question first followed by the questions in the same deescalating order of urgency as GSU1. Unlike the ABC survey, which produced 25.1 percent inconsistent

respondents, GSU2 produced inconsistency among only 13.5 percent of its respondents (Carlson 2000). Interestingly, of the 27 respondents who produced inconsistent responses in GSU2, 23, or 85.2 percent, of them said yes to the any reason question and then changed their minds as they heard the individual scenarios described in the Guttman scale order. This was by far the largest proportion of pro choice inconsistent produced by any question order (Control 77.3%, GSU1 64.7%, GSU3 79.2%) and made GSU2 the only significant dummy variable in the logit analysis of the respondent demographics, where the reference variable was the control group order (see Table 4.8). GSU2 also produced significantly fewer moderate respondents compared with GSU1, which had the same question order except the any reason question was last instead of first

(42.5% vs. 52.8%, z=2.06, p<.05). The third and final experimental group, GSU3, used the same

mixed question order as the control group with one major exception – the any reason question came at the beginning of the survey instead of the end. This placement generated slightly more

inconsistent respondents than the control group – 12.6 percent. It also had significantly fewer moderate respondents than GSU1 (42.6% vs. 52.8%, z=2.013, p<.05), but its proportions were the closest in line with the total survey averages and otherwise with the other question orders.

While there was some distinction between the question orders, the overall difference between the percentages of inconsistent respondents produced by each question order was not statistically significant (see Table 4.1). Note that pro choice includes respondents who answered yes to all seven questions, pro life includes those who answered no to all seven questions and moderate includes those who answered no or don’t know to the any reason question and either yes, no or don’t know to the more specific questions, as well as a few (n=20) who said yes to the any reason question but refused to give an answer to one or more of the more specific questions.

*Moderates includes those who answered no or don’t know to the any reason question and either yes, no or don’t know to the

more specific questions, plus a few (n=20) who said yes to the any reason question but did not answer one or more of the more specific questions.

**Inconsistent includes some (n=70) who said yes to any reason but no to at least one of the other questions, plus those (n=20) who said no to “a woman’s life endangered” but yes to at least one of the other questions.

The question order actually had relatively little impact on the answers to all of the questions, including the general any reason question. Respondents in GSU3, the group with a mixed order and the any reason question first, gave significantly more positive responses to one “easy” questions compared with the control group – 80.5 percent for the rape question vs. 69.5 percent (z=2.51, p<.05). As for the any reason question, the two groups that had the question last

have fewer positive responses and the two groups that heard it first gave more positive responses,

Table 4.1: Impact of question order on abortion answers (in percentages)

Answer category Total (n) Control GSU1 GSU2 GSU3

Pro Choice (Yes to all) (217) 27.1 25.6 27.5 29.5

Pro Life (No to all) (92) 13.8 9.0 13.5 10.0

Moderate* (393) 48.3 56.8 45.5 47.9

but the differences were not statistically significant. There was some evidence, however, that placement of the any reason question at the end of the battery of questions increased negative responses, particularly when the questions were ranked by severity of the woman’s

circumstances. More respondents in GSU1, where the any reason question was asked last and the questions were asked in descending order of severity, offered negative responses to the any reason question than did respondents in GSU3, where the question order was mixed and the any reason question came first (61.8% “no” in GSU1 vs. 51.6% “no” in GSU3, z=2.03, p<.05). Otherwise, the responses were surprisingly stable despite the question order (see Table 4.2).