ARM Lecture 6

(1)

LOGO

Construction of

Questionnaire

Construct

Construction o

ion o

f

Questionnaire

Lecture # 6 Lecture # 6

(2)

(3)

1.

Framework for

Framework for questionsquestions

2.

Types of questionnaires Types of questionnaires

3.

Guidelines Guidelines

4.

(4)

LOGO

What is

questionnaire?

What is

(5)

What is

questionnaire?



_{A questionnaire is a research}

instrument consisting of a series of

questions and other prompts for the

purpose of gathering information from

respondents.



_{Although they are often designed for}

statistical analysis of the responses,

this is not always the case.



_{Remember, good questionnaires take}

time

to

develop.

They

are

not

just

questions on a page.

(6)



Questionnaires are used in sample

surveys or censuses to elicit reports

of facts, attitudes, and other

subjective states.



_{The developments are beginning to}

transform survey questionnaire

construction from an art to a

science.

(7)

What is not questionnaire?

A questionnaire is not just a list of

questions.

It is tempting to begin questionnaire design

by drafting possible qns on the survey

topics.

That is obviously important, but not the right

place to start.

We must first think about the following

topics.

(8)

Objectives of questionnaire

construction?



To maximize the proportion of

subjects answering our

questionnaire - that is,

the response

rate.



_{To obtain accurate relevant}

information for our survey

(9)

History



In 1086 William the Conqueror

surveyed the wealth and

landholdings of England using a

standard set of inquiries and

compiled the results in the

“Domesday Book.”



_{The questionnaire was invented by}

Sir Francis Galton: 1874

(10)

Advantages of Written

Questionnaires



 _{Questionnaires are very cost effective}_{Questionnaires are very cost effective}

when compared to face-to-face interviews.

This is especially true for studies involving

large sample sizes and large geographic

areas. Written questionnaires become

even more cost effective as the number of

research questions increases.



 _{Questionnaires are easy to analyze. Data}_{Questionnaires are easy to analyze. Data} entry and tabulation for nearly all surveys

entry and tabulation for nearly all surveys

can be easily done with many computer

software packages.

(11)



 _{Questionnaires are familiar to most}_{Questionnaires are familiar to most}

people. Nearly everyone has had some

experience completing questionnaires and

they generally do not make people

apprehensive.



 _{Questionnaires reduce bias. There is}_{Questionnaires reduce bias. There is} uniform question presentation and no

uniform question presentation and no

middle-man bias. The researcher's own

opinions will not influence the respondent

to answer questions in a certain manner.

There are no verbal or visual clues to

influence the respondent.

(12)



Questionnaires are less intrusive

than telephone or face-to-face

surveys. When a respondent

receives a questionnaire in the mail,

he is free to complete the

questionnaire on his own time-table.

Unlike other research methods, the

respondent is not interrupted by the

research instrument.

(13)

Disadvantages Of Written

Questionnaires



Possibility of low response rates

:

Low response is the curse of

statistical analysis. It can

dramatically lower our confidence in

the results. Response rates vary

widely from one questionnaire to

another (10% - 90%), however,

well-designed studies consistently

produce high response rates.

(14)



 _{Inability to probe responses.}_{Inability to probe responses.} Questionnaires are structured

Questionnaires are structured

instruments. They allow little flexibility to

the respondent with respect to response

format. In essence, they often lose the

"flavor of the response" (i.e., respondents

often want to qualify their answers).



 By allowing frequent space for comments, the rBy allowing frequent space for comments, the researcher canesearcher can

partially overcome this disadvantage. Comments are among

the most helpful of all the information on

the most helpful of all the information on the questionnaire,the questionnaire,

and they usually provide insightful information that would

have otherwise been lost.

(15)



 Nearly ninety percent of all communicationNearly ninety percent of all communication

is visual. Gestures and other visual cues

are not available with written

questionnaires. The

questionnaires. The lack of personallack of personal

contact

contact will have different effectswill have different effects

depending on the type of information being

requested. A questionnaire requesting

factual information will probably not be

affected by the lack of personal contact. A

questionnaire probing sensitive issues or

attitudes may be severely affected.

(16)



 When returned questionnaires arrive in theWhen returned questionnaires arrive in the

mail, it's natural to assume that the

respondent is the same person you sent

the questionnaire to. This may not

the questionnaire to. This may not actuallyactually

be the case.

be the case. Many times businessMany times business

questionnaires get handed to other

employees for completion.

employees for completion. HousewivesHousewives

sometimes respond for their

sometimes respond for their husbands.husbands.

Kids respond as a prank. For a

Kids respond as a prank. For a variety of variety of

reasons, the respondent may not be who

you think it is. It is a confo

you think it is. It is a confounding errorunding error

inherent in questionnaires.

(17)



Finally, questionnaires are simply

not suited for some people

. For

example, a written survey to a

group of poorly educated people

might not work because of reading

skill problems. More frequently,

people are turned off by written

questionnaires because of misuse.

(18)

LOGO

Framework for

construction of

questionnaire

Framework for

construction of

questionnaire

(19)

Clarify goals

(20)

Conversations with several of the

stakeholders

(21)

Focus your goals

(22)

Invite people to take survey

(23)

Analyze responses

(24)

Translating data into meaningful

information

(25)

Present Results

(26)

Celebrate your success

(27)

Steps for Developing

Questionnaires



Develop the PurposeDevelop the Purpose



_{Create the Conceptual}

_Framework



Write Questions



(28)

Proper construction of the

questionnaire is essential to its

success

Powell, Ronald R.

(29)

Framework: Questionnaire

Construction

Specify What Specify What Information Information Will be Sought Will be Sought Identify Identify Correct Correct Respondent Respondent Determine Determine Method of Method of Administration Administration Question Structure

Question Structure _{Question Wording}_{Question Wording} _{Question Sequencing}_{Question Sequencing}

PRETEST PRETEST and and Revise Revise

(30)

Steps to design a questionnaire:

1.

1. Write out the primary and secondary aimsWrite out the primary and secondary aims

of your study.

2.

2. Write out concepts/information to beWrite out concepts/information to be

collected that relates to these aims.

3.

3. Review the current literature to identifyReview the current literature to identify

already validated questionnaires that

measure your specific area of

measure your specific area of interest.interest.

4.

4. Compose a draft of your questionnaire.Compose a draft of your questionnaire.

5.

5. Revise the draft.Revise the draft.

6.

(31)

Questionnaire Design - General

Considerations



 _{Most problems with questionnaire analysis}_{Most problems with questionnaire analysis} can be traced back to the design phase of

can be traced back to the design phase of

the project.

the project. Well-defined goals are theWell-defined goals are the

best way to assure a good questionnaire

design.



 _{When the goals of a study can be}_{When the goals of a study can be} expressed in a few clear

expressed in a few clear and conciseand concise

sentences, the design of the questionnaire

becomes considerably easier.



 _{The questionnaire is developed to directly}_{The questionnaire is developed to directly} address the goals of the study.

(32)



One of the best ways to clarify

your

study goals is to

decide how you

intend to use the information.

Do

this before you begin designing the

study. This sounds obvious, but

many researchers neglect this task.

Why do research if the results will

not be used?

(33)



Be sure to commit the study goals to

writing. Whenever you are unsure of

a question, refer to the study goals

and a solution will

become clear.

Ask only questions that directly

address the study goals.

Avoid the

temptation to ask questions because

it would be "interesting to know".

(34)



 As a general rule, with only a fewAs a general rule, with only a few

exceptions,

exceptions, long questionnaires get lesslong questionnaires get less

response than short questionnaires.

response than short questionnaires. KeepKeep

your questionnaire short. In fact, the

shorter the better.

shorter the better. 

 Response rate is the single most importantResponse rate is the single most important

indicator of how much confidence you can

place in the results. A low response rate can

be devastating to a study.

be devastating to a study. 

 One of the most effective methods of One of the most effective methods of

maximizing response is to shorten the

questionnaire.

(35)



 _{If your survey is over a few pages,}_{If your survey is over a few pages,} _{try to}_{try to} eliminate questions.

eliminate questions.



 _{Many people have difficulty knowing which}_{Many people have difficulty knowing which} questions could be eliminated.

questions could be eliminated.



 _{For the elimination round, read each}_{For the elimination round, read each}

question and ask, "How am I going to use

this information?" If the information will

be used in a decision-making process, then

keep the question... it's important. If not,

throw it out.

(36)



 _{One important way to assure a successful}_{One important way to assure a successful} survey is to

survey is to include other experts andinclude other experts and

relevant decision-makers

relevant decision-makers in thein the

questionnaire design process.



 _{Their suggestions will improve the}_{Their suggestions will improve the}

questionnaire and they will subsequently

have more confidence in the results.

(37)



 _{Formulate a plan for doing the statistical}_{Formulate a plan for doing the statistical} analysis during the design stage

analysis during the design stage of theof the

project. Know how every question will be

analyzed and be prepared to handle

missing data. If you cannot specify how

you intend to analyze a question or use

the information, do not use it in the

survey.

(38)



 Give your questionnaire a title Give your questionnaire a title that is short andthat is short and

meaningful to the respondent.

meaningful to the respondent. A questionnaireA questionnaire

with a title is generally perceived to be more

credible than one without



 Include clear and concise instructions on howInclude clear and concise instructions on how

to complete the questionnaire

to complete the questionnaire. These must be. These must be

very easy to understand, so use short

sentences and basic vocabulary. Be sure to

print the return address on the questionnaire

itself (since questionnaires often get separated

from the reply envelopes).

(39)



Begin with a few

Begin with a few

non-threatening and

interesting items.

If the first items are

too threatening or "boring", there is

little chance that the person will

complete the

questionnair

e.



People generally look at the first few

People generally look at the first few

questions before deciding whether or

not to complete the

questionnai

re. Make

them want to

continue by putting

interesting questions first.

(40)



 Use simple and direct Use simple and direct language.language. The questionsThe questions

must be clearly understood by the respondent.



 The wording of a question should be simple andThe wording of a question should be simple and

to the point.



 Do not use uncommon words or long sentences.Do not use uncommon words or long sentences.

Make items as brief as possible. This will

Make items as brief as possible. This will reducereduce

misunderstan

misunderstandings and make dings and make the questionnairethe questionnaire

appear easier to complete.



 One way to eliminate One way to eliminate misundersmisunderstandings is totandings is to

emphasize crucial words in each item by using

bold, italics or

(41)



 _{Leave adequate space for respondents to}_{Leave adequate space for respondents to} make comments.

make comments.



 _{One criticism of questionnaires is their}_{One criticism of questionnaires is their} inability to retain the "flavor" of a

inability to retain the "flavor" of a

response.



 _{Leaving space for comments will provide}_{Leaving space for comments will provide} valuable information not captured by the

valuable information not captured by the

response categories.



 _{Leaving white space also makes the}_{Leaving white space also makes the} questionnaire look easier and this

questionnaire look easier and this

increases response.

(42)



 _{Place the most important items in the first}_{Place the most important items in the first} half of the questionnaire.

half of the questionnaire.



 _{Respondents often send back partially}_{Respondents often send back partially} completed questionnaires.

completed questionnaires.



 _{By putting the most important items near}_{By putting the most important items near} the beginning, the partially completed

the beginning, the partially completed

questionnaires will still contain important

information.

(43)



 _{Use professional production methods}_{Use professional production methods} _for_for the questionnaire--either desktop

the questionnaire--either desktop

publishing or typesetting and keylining.



 _{Be creative.}_{Be creative.} 

 _{Try different colored inks and paper. The}_{Try different colored inks and paper. The} object is to make your questionnaire stand

object is to make your questionnaire stand

out from all the others the respondent

receives.

(44)



The final test of a questionnaire is toThe final test of a questionnaire is to try it ontry it on

representatives of the target audience

representatives of the target audience..



If there are problems with the questionnaire, theyIf there are problems with the questionnaire, they

almost always show up here. If possible, be present

while a respondent is

while a respondent is completing the questionnairecompleting the questionnaire

and tell her that it

and tell her that it is okay to ask you foris okay to ask you for

clarification of any item.



The questions she asks are indicative of problems inThe questions she asks are indicative of problems in

the questionnaire (i.e., the questions on the

questionnaire must be without any

questionnaire must be without any ambiguityambiguity

because there will be no chance to clarify a

question when the survey is

(45)

LOGO

Types of questions

(46)

Classification # 1

1.

1. Factual questions:Factual questions: age, gender…age, gender…

2.

2. Opinion and attitude questions:Opinion and attitude questions: intended tointended to

determine person’s ideas, inclinations, prejudices,

… with attitude scales, and indexes

3.

3. Information questions:Information questions: designed to measuredesigned to measure

respondent’s knowledge

4.

4. Self-perception questionsSelf-perception questions: Similar to attitude: Similar to attitude

questions but are restricted to one’s opinions

about himself or herself

5.

5. Standards of action questionsStandards of action questions: used to determine: used to determine

how respondents would act in certain

circumstances

6.

6. Questions about actual past or present behaviourQuestions about actual past or present behaviour::

7.

(47)

Classification # 2



Four types of response scales for

closed-ended questions are

distinguished:



 Dichotomous,Dichotomous, where the respondent has two optionswhere the respondent has two options



 Nominal-polytomousNominal-polytomous, where the respondent has , where the respondent has moremore

than two unordered options



 Ordinal-polytomous,Ordinal-polytomous, where the respondent has morewhere the respondent has more

than two ordered options



 (bounded)Continuous(bounded)Continuous, where the respondent is, where the respondent is

presented with a continuous scale

(48)

1. Contingency questions

1. Contingency questions - A question that is- A question that is

answered only if the respondent gives a

particular response to a previous

particular response to a previous question.question.

This avoids asking questions of people that

do not apply to them

2. Matrix questions

2. Matrix questions - Identical response- Identical response

categories are assigned to multiple

questions. The questions are placed one

under the other, forming a matrix with

response categories along the top and a

list of questions down the side. This

list of questions down the side. This is anis an

efficient use of page space and

respondents’ time.

(49)

3. Closed ended

3. Closed ended questionsquestions - Respondents’ answers are- Respondents’ answers are

limited to a fixed set of responses.



 Yes/no questions Yes/no questions - The respondent answers with a “yes” or - The respondent answers with a “yes” or aa

“no”.



 Multiple choiceMultiple choice - The respondent has several option from which- The respondent has several option from which

to choose.



 Scaled questionsScaled questions - Responses are graded on a continuum- Responses are graded on a continuum

(example : rate the appearance of the product on

(example : rate the appearance of the product on a scale from 1a scale from 1

to 10, with 10 being

to 10, with 10 being the most preferred appearance). Examplesthe most preferred appearance). Examples

of types of scales include the

of types of scales include the Likert scale, semantic differentialLikert scale, semantic differential

scale, and rank-order scale (See scale for a complete list

scale, and rank-order scale (See scale for a complete list of of

scaling techniques.).

(50)



 Open ended questionsOpen ended questions -- No options or predefinedNo options or predefined

categories are suggested. Examples of types of open

ended questions include:

ended questions include: 

 Completely unstructuredCompletely unstructured - For example, “What is your opinion of - For example, “What is your opinion of

questionnaires?”



 Word associationWord association - Words are presented and the respondent mentions- Words are presented and the respondent mentions

the first word that comes to mind.



 Sentence completionSentence completion - Respondents complete an incomplete- Respondents complete an incomplete

sentence. For example, “The most important consideration in my

decision to buy a new house is . . .”



 Story completionStory completion - Respondents complete an incomplete story.- Respondents complete an incomplete story.



 Picture completionPicture completion - Respondents fill in an empty conversation balloon.- Respondents fill in an empty conversation balloon.



 Thematic apperception testThematic apperception test - Respondents explain a picture or make- Respondents explain a picture or make

up a story about what they think is happening in the picture

(51)

Question Structure

Open-end questions

If you were to purchase a new automobile during 2001, what brand of automobile would

y

yoou u mmoosst t lliikkeelly y ppuurrcchhaassee? ? _____________________ _ bbrraanndd

Closed-ended with ordered categories Closed-ended with ordered categories

If you were to purchase a new automobile during 2001, how likely is it that you would

purchase the following brands of

purchase the following brands of automobiles?automobiles?

V Veerry y LLiikkeellyy SSoommeewwhhaat t LLiikkeellyy SSoommeewwhhaat t UUnnlliikkeellyy VVeerry y UUnnlliikkeellyy T Tooyyootta a CCaammrry y 1 1 2 2 3 3 44 H Hoonndda a AAccccoorrdd 1 1 2 2 3 3 44 N Niissssaan n MMaaxxiimma a 1 1 2 2 3 3 44

Closed-ended with unordered categories Closed-ended with unordered categories

If you were to purchase a new automobile during 2001, which one of the following

automobiles would you most likely purchase?

Toyota Camry

Toyota Camry tt _{Honda Accord}_{Honda Accord} tt

Nissan Maxima

Nissan Maxima tt

Partially closed-ended Partially closed-ended

If you were to purchase a new automobile during 2001, which one of the following would

you most likely purchase?

Toyota Camry

Toyota Camry tt _{Honda Accord}_{Honda Accord} tt

Nissan Maxima

Nissan Maxima tt _{Other (please}_{Other (please}

specify)______________

(52)

Closed Vs. Open-ended Questions

A

Addvvaannttaaggeess DDiissaaddvvaannttaaggeess

Closed-ended

Closed-ended ••Easy and quick to answerEasy and quick to answer

•

•Answers across resps easy toAnswers across resps easy to

compare

•

•Answers easier to analyze onAnswers easier to analyze on

computer

•

•Response choices makeResponse choices make

question clearer

•

•Easy to replicate studyEasy to replicate study

•

•Can put ideas in resp's headCan put ideas in resp's head

•

•Resps w/ no opinion answer anywayResps w/ no opinion answer anyway

•

•Resps can Resps can feel constrained/frustratedfeel constrained/frustrated

•

•Many choices can be confusingMany choices can be confusing

•

•Can't tell if resp. misinterpreted theCan't tell if resp. misinterpreted the

question

•

•Fine distinctions may be lostFine distinctions may be lost

•

•Clerical mistakes easy to makeClerical mistakes easy to make

•

•Force respondents into simpleForce respondents into simple

responses

Open-ended

Open-ended ••Permit unlimited number of Permit unlimited number of

answers

•

•Resps can qualify and clarifyResps can qualify and clarify

responses

•

•Can find the unanticipatedCan find the unanticipated

•

•Reveal resps thinking processesReveal resps thinking processes

•

•Resps give answers w/ diff. level of Resps give answers w/ diff. level of

detail

•

•Answers can be irrelevantAnswers can be irrelevant

•

•Inarticulate or forgetful resps are atInarticulate or forgetful resps are at

disadvantage

•

•Coding responses is subjective andCoding responses is subjective and

tedious

•

•Requires more resp. time and effortRequires more resp. time and effort

•

•Intimidates respondentsIntimidates respondents

•

•When resp omits a response, can't tellWhen resp omits a response, can't tell

if its because

of belief or just forgetfulness

(53)

Closed Vs. Open-ended Questions

Evaluation Evaluation Criteria Criteria A Addvvaannttaaggeess DDiissaaddvvaannttaaggeess T Tiimmee LLeesss s ttiimme e ttoo

respond & record

Longer time to Longer time to develop develop Quick transfer to Quick transfer to computer form computer form

(54)

Evaluation Evaluation Criteria Criteria A Addvvaannttaaggeess DDiissaaddvvaannttaaggeess C Coosstt CChheeaappeer r dduue e tto o lloowweer r recording & recording & interpreting time interpreting time

(55)

Evaluation Evaluation Criteria Criteria A Addvvaannttaaggeess DDiissaaddvvaannttaaggeess A Accccuurraaccyy LLeesss s iinntteerrvviieewweer r recording error recording error

Forcing answers into

categories categories Respondent Respondent convenience convenience More convenient to More convenient to answer answer

(56)

Structured Questions:

Effect on Respondent Answers



_{Response order effects}



_Visual

_{Visual presentation}

_{presentation of}

_{of response}

_{response alternatives}

_alternatives



Primacy effects



 initial items establish a cognitive framework to guideinitial items establish a cognitive framework to guide processing

processing



 _{initial items get deeper cognitive processing & less}_{initial items get deeper cognitive processing & less}

interference than later items interference than later items



 _{satisficing — minimize psychological costs by}_{satisficing — minimize psychological costs by}

choosing first acceptable response

choosing first acceptable response alternativealternative

SOURCE: Krosnick and Alwin, POQ SOURCE: Krosnick and Alwin, POQ

(57)

Response order effects

Oral

Oral presentation

presentation of

of response

response alternatives

alternatives

Recency

effects

externally based processing

each read alternative terminates processing of previous alternative each read alternative terminates processing of previous alternative increased processing of later response alternatives

increased processing of later response alternatives

Structured Questions:

Effect on Respondent Answers

Source

Source:: An evalAn evaluatiouation of a cognn of a cognitive thitive theory of reeory of responssponse-ordee-order effects in sur effects in surveyrvey measurement,” J. Krosnick and D. Alwin,

measurement,” J. Krosnick and D. Alwin, Public Opinion Quarterly Public Opinion Quarterly , 1987,, 1987, 51, 1201-1219.

(58)

Question Sequence

and Order Effects

Question

Do y

Do you think

ou think a comm

a communist country

unist country lik

lik

reporters come in and send back to t

Do y

Do you think

ou think the United States shoul

the United States shoul

other countries come in and send ba

Q

Qu

ue

es

sttiio

on

n s

se

eq

qu

ue

en

nc

ce

e 2

2 Y

Ye

es

s

Do you think the United States should let communist newspaper reporters Do you think the United States should let communist newspaper reporters from other countries come in and send back to their papers the news as they from other countries come in and send back to their papers the news as they see it?

see it? Do you th

Do you think a commink a communist countunist country like Russia shoury like Russia should let Amld let American erican newnewspaper spaper reporters come in and send back to their pape

reporters come in and send back to their pape rs the news as they see it?rs the news as they see it?

36% 36% 66% 66% Once Committed Once Committed Hard to be Hard to be Unfair Unfair 37% 37% Difference ! Difference ! Source

(59)

Questionna

ire

Format



Matrix questions are often useful.



 They use space efficientlyThey use space efficiently



 _{Respondents can answer quickly}_{Respondents can answer quickly}



 _{Respondents can compare across questions for}_{Respondents can compare across questions for}_consistency._consistency.



_{However, matrix questions have some}

problems.



 You might be tempted to put questions into a matrix whenYou might be tempted to put questions into a matrix when

they belong in some other format.



 _{Respondents may pattern their}_{Respondents may pattern their}_{answers inappropriately.}_{answers inappropriately.}



 _{Order effects may}_{Order effects may}_{be quite pronounced, with strong potential}_{be quite pronounced, with strong potential}

for abuse of the survey.

(60)



Some special comments on order

effects.



 Early answers tend to Early answers tend to influence later answers.influence later answers.



 _{Some respondents (such as less}_{Some respondents (such as less}_{educated respondents) are}_{educated respondents) are}

more subject to order effects than other

more subject to order effects than other respondents.respondents.



 _Randomizi_Randomizi_{ng questions tends not to help - it}_{ng questions tends not to help - it}_{just makes the}_{just makes the}

survey more confusing.



 Fun or interesting questions in the early part of aFun or interesting questions in the early part of a

questionnai

questionnaire can re can maintain interest, but general,maintain interest, but general,

demographic sorts of questions can establish rapport.

•

• You might want to adjust the order of questions according tYou might want to adjust the order of questions according too

the type of survey –

(61)

Other Types



Leading Questions:

A leading

question is one that forces or implies

a certain type of answer. E.g.

  SuperbSuperb   ExcellentExcellent   GreatGreat   GoodGood   Fair Fair 

(62)



Embarrassing Questions:

Embarrassing Questions:

Embarrassing

questions dealing with personal or

private matters should be avoided. Your

data is only as good as the

data is only as good as the trust and

trust and

care that your respondents give you. If

you make them feel uncomfortable, you

will lose their trust. Do not ask

embarrassing questions.

(63)



Hypothetical Questions:

Hypothetical are based, at best, on

conjecture and, at worst, on fantasy.

I simple question such as:



 If you were governor, what would you do to combatIf you were governor, what would you do to combat terrorism?

(64)

LOGO

Most frequent errors

in questionnaires

Most frequent errors

in questionnaires

(65)

Issues to Consider

To Neutral or Not to Neutral.

Think about how a respondent enters a

response

scale and how many decisions the person

must

make.

S Sttrroonngglly y DDiissaaggrreeee DDiissaaggrreeee NNeeiitthheer r AAggrreee e nnoor r Disagree Disagree A Aggrreeee SSttrroonngglly y AAggrreeee 1 2 3 4 5 1 2 3 4 5

(66)

Pre-coding can save time with data entry.

Has this presentation increased your

Has this presentation increased your understanding of questionnaireunderstanding of questionnaire

development?

1

1 YesYes 22 NoNo

There is more than one way to

ask a question. Use a

variety of question formats.

Pre-code

(67)

What do you think?

Were you satisfied with the quality of

our

services and programs?

A. Yes

B. No

Double

Barrel:

Ask

only

one

question

at

a

time

(68)

most frequent errors in

questionnaires



 _{questions which don't quite mean what}_{questions which don't quite mean what} the researcher intended

the researcher intended



 _{questions which don't probe to find out}_{questions which don't probe to find out} what the respondent really meant

what the respondent really meant

long and complicated questions



 _{questions which repeat what has already}_{questions which repeat what has already} been asked

been asked



 _{questions which don't allow the}_{questions which don't allow the}

respondent to answer in a way which is

relevant



 _{questions which are inappropriate to the}_{questions which are inappropriate to the} research method

(69)



 _{bad routing which leaves the interviewer}_{bad routing which leaves the interviewer} wondering which question to ask next, or

wondering which question to ask next, or

worse, routing to the wrong one

too many questions



 _{poorly laid out questions which are badly}_{poorly laid out questions which are badly} grouped in the questionnaire

grouped in the questionnaire



 _{questions which have been missed out}_{questions which have been missed out} completely

completely



 _{pages of the questionnaire which are}_{pages of the questionnaire which are} missing or out of order.

(70)



Research showing that small

changes in question wording or

order can substantially affect

responses has reinforced the

assumption that questions must be

asked exactly as worded, and in the

same order, to

produce comparable

data.

(71)

LOGO

Guidelines

(72)

Quality aims in survey research

Goal is to collect information that is:



 Valid:Valid: measures the quantity or concept thatmeasures the quantity or concept that

is supposed to be measured



 Reliable:Reliable: measures the quantity or concept inmeasures the quantity or concept in

a consistent or reproducible manner



 Unbiased:Unbiased: measures the quantity or conceptmeasures the quantity or concept

in a way that does

in a way that does not systematically under-not systematically

under-or overestimate the true value

or overestimate the true value



 Discriminating:Discriminating: can distinguish adequatelycan distinguish adequately

between respondents for whom the underlying

level of the quantity or concept is

(73)

Qualities of a Good Question



 Evokes the truth.Evokes the truth. Questions must be non-Questions must be

non-threatening. When a respondent is

threatening. When a respondent is

concerned about the consequences of

answering a question in a particular

manner, there is a good

manner, there is a good possibility that thepossibility that the

answer will not be truthful. Anonymous

questionnaires that contain no identifying

information are more likely to produce

honest responses than those identifying

the respondent. If your questionnaire does

contain sensitive items, be sure to clearly

state your policy on confidentiality.

(74)



 Asks for an answer on only one dimensionAsks for an answer on only one dimension..

The purpose of a survey is to find out

information. A question that asks for a

response on more than one dimension will

not provide the information you are

seeking. For example, a

seeking. For example, a researcherresearcher

investigating a new food snack asks "Do

you like the texture and flavor of the

snack?" If a

snack?" If a respondent answers "no",respondent answers "no",

then the researcher will not know if the

respondent dislikes the texture or the

flavor, or both

(75)



 _{Can accommodate all possible answers.}_{Can accommodate all possible answers.}

Multiple choice items are the most popular

type of survey questions because they are

generally the easiest for a respondent to

answer and the easiest to analyze. Asking

a question that does not accommodate all

possible responses can confuse and

frustrate the respondent. For example,

consider the question:



 What brand of computer do you own? What brand of computer do you own? __ __



 A. IBM PC A. IBM PC



(76)



 Has mutually exclusive options.Has mutually exclusive options. A goodA good

question leaves no ambiguity in the mind of

the respondent. There should be only one

correct or appropriate choice for the

respondent to make. An obvious example is:



 Where did you grow up? Where did you grow up? __ __



 A. country A. country



 B. farmB. farm



 C. city C. city



 A person who grew up on a farm in theA person who grew up on a farm in the

country would not know whether to select

choice A or B.

(77)



 Produces variability of responses.Produces variability of responses. When aWhen a

question produces no variability in

responses, we are left with considerable

uncertainty about why we asked the

question and what we learned from

question and what we learned from thethe

information. If a question does not

produce variability in responses, it will not

be possible to perform any statistical

analyses on the item. For example:



 What do you think about this report? What do you think about this report? __ __



 A. It's the worst report I've read A. It's the worst report I've read



 B. It's somewhere between the worst and best B. It's somewhere between the worst and best



 C. It's the best report I've read C. It's the best report I've read



(78)



Order of questions:



 Go from general to Go from general to particular.particular.



 Go from easy to difficult.Go from easy to difficult.



 _{Go from factual to abstract.}_{Go from factual to abstract.}



 _{Start with closed format}_{Start with closed format}_questions._questions.



 Start with questions relevant to the Start with questions relevant to the main subject.main subject.



 _{Do not start}_{Do not start}_{with demographic and personal}_{with demographic and personal}

questions.

(79)



 _{Follows comfortably from the previous}_{Follows comfortably from the previous} question.

question. Writing a questionnaire is similarWriting a questionnaire is similar

to writing anything else. Transitions

between questions should be smooth.

Grouping questions that are similar will

make the questionnaire easier to

complete, and the respondent will feel

more comfortable. Questionnaires that

jump from one unrelated topic to another

feel disjointed and are not likely to

produce high response rates.

(80)



 Does not presuppose a certain state of Does not presuppose a certain state of

affairs.

affairs. Among the most subtle mistakes inAmong the most subtle mistakes in

questionnaire design are questions that

make an unwarranted assumption. An

example of this type of mistake is:



 Are you Are you satisfied with your satisfied with your current auto current auto insurance? insurance? (Yes or (Yes or No)No)



 This question will present a problem forThis question will present a problem for

someone who does not currently have auto

insurance. Better:



 Are you satisfied with your Are you satisfied with your current auto insurance? current auto insurance?

___

___ YesYes __ __ NoNo

___

Don't have auto

insurance

(81)



 Does not imply a desired answer.Does not imply a desired answer. TheThe

wording of a question is extremely

important. We are striving for objectivity in

our surveys and, therefore, must be careful

not to lead the respondent into giving the

answer we would like to receive. Leading

questions are usually easily spotted

because they use negative phraseology. As

examples:



 Wouldn't you like to receive our free brochure? Wouldn't you like to receive our free brochure?



(82)



 _{Does not use emotionally loaded or}_{Does not use emotionally loaded or} vaguely defined words.

vaguely defined words. This is one of theThis is one of the

areas overlooked by both beginners and

experienced researchers. Quantifying

adjectives (e.g., most, least, majority) are

frequently used in questions. It is

important to understand that these

adjectives mean different things to

different people.

(83)



 Does not use unfamiliar words orDoes not use unfamiliar words or

abbreviations.

abbreviations. Remember who your audienceRemember who your audience

is and write your questionnaire for them. Do

not use uncommon words or compound

sentences. Write short sentences.

Abbreviations are okay if you are absolutely

certain that every single respondent will

understand their meanings. If there is any

doubt at all, do not use the abbreviation. The

following question might be okay if all the

respondents are accountants:



(84)



 Is not dependent on responses to previousIs not dependent on responses to previous

questions.

questions. Branching in written questionnairesBranching in written questionnaires

should be avoided. While branching can be used as

an effective probing technique in telephone and

face-to-face interviews, it should not be used

face-to-face interviews, it should not be used inin

written questionnai

written questionnaires because res because it sometimesit sometimes

confuses respondents. An example of

confuses respondents. An example of branchinbranching is:g is: 

 1. Do you currently have a life insurance policy ? (Yes or No) If 1. Do you currently have a life insurance policy ? (Yes or No) If no, gono, go to question 3

to question 3 

 2. How much is your annual life insurance premium ? ______2. How much is your annual life insurance premium ? ______



 Alternate Option: Alternate Option: 

 How much did you spend last year for life insurance ? ______How much did you spend last year for life insurance ? ______

. How much did you spend last year for life insurance ? ______

(85)



 Does not ask the respondent to order or ranDoes not ask the respondent to order or rank ak a

series of more than five items.

series of more than five items. This becomesThis becomes

increasingly difficult as the number of items

increases, and the answers become less reliable.

This becomes especially problematic when asking

respondents to assign a percentage to a series of

items. In order to successfully complete this task,

the respondent must mentally continue to r

the respondent must mentally continue to re-

e-adjust his answers until they total one hundred

adjust his answers until they total one hundred

percent. Limiting the number of items to

percent. Limiting the number of items to five willfive will

make it easier for the respondent to answer.

(86)

Questionnaire Construction



Keep it Keep it short, simple, and specific (KISSS)short, simple, and specific (KISSS)



Logical question progressionLogical question progression



Flow more important than grammaticalFlow more important than grammatical

correctness



Key questions first, sensitive Key questions first, sensitive lastlast



Keep allied questions togetherKeep allied questions together



Be Be reasonable reasonable when when requesting requesting forecastsforecasts



Mix Mix fully-sfully-structured tructured and and open-ended open-ended questionsquestions



ARM Lecture 6

LOGO

LOGO

Construction of

Construction of

Questionnaire

Questionnaire

Construct

Construction o

ion o

f

f

Questionnaire

Questionnaire

Contents

Contents

1.

1.

2.

2.

3.

3.

4.

4.

LOGO

LOGO

What is

What is

questionnaire?

questionnaire?

What is

What is

What is

questionnaire?

questionnaire?

A questionnaire is a research

A questionnaire is a research

instrument consisting of a series of

instrument consisting of a series of

questions and other prompts for the

questions and other prompts for the

purpose of gathering information from

purpose of gathering information from

respondents.

respondents.

Although they are often designed for

Although they are often designed for

statistical analysis of the responses,

statistical analysis of the responses,

this is not always the case.

this is not always the case.

Remember, good questionnaires take

Remember, good questionnaires take

time

time

to

to

develop.

develop.

They

They

are

are

not

not

just

just

questions on a page.

Questionnaires are used in sample

Questionnaires are used in sample

surveys or censuses to elicit reports

surveys or censuses to elicit reports

of facts, attitudes, and other

of facts, attitudes, and other

subjective states.

subjective states.

The developments are beginning to

The developments are beginning to

transform survey questionnaire

transform survey questionnaire

_{A questionnaire is a research}

_{A questionnaire is a research}

_{Although they are often designed for}

_{Although they are often designed for}

_{Remember, good questionnaires take}

_{Remember, good questionnaires take}

_{The developments are beginning to}

_{The developments are beginning to}

_{To obtain accurate relevant}

_{To obtain accurate relevant}

_{The questionnaire was invented by}

_{The questionnaire was invented by}