Introduction to

Questionnaire Design
Spring 2005 Seminar Series

Survey Research Laboratory

University of Illinois at Chicago
www.srl.uic.edu
 Questionnaire design
encompasses:
1. How to write questions
2. How to draft and organize a
questionnaire
 The Art of Asking Questions
You must ask the right question
Respondents must understand your

question
Respondents must know the answer
Respondents must be willing and
able to tell you the answer
Three Simple Rules for Writing
a Good Questionnaire
1) Think through your research questions and
objectives before you write questions
2) Prepare an analysis plan before you write
questions
3) Ask yourself, in relation to points #1 and #2
above, if each question on your list is
necessary? Even if the data would be
‘interesting’ it has to ultimately be used in
analysis to make the cut!
 Types of Survey Questions
1) Those that ask about behaviors or facts
 Non-threatening behavior questions
 Threatening behavior questions
 Demographics
2) Those that ask about psychological states
or attitudes
3) Those that ask about knowledge
 What Is A Good Question?
 One that yields a truthful, accurate answer
 One that asks for one answer on one dimension
 One that accommodates all possible contingencies
of response
 One that uses specific, simple language
 One that has mutually exclusive response options
 One that produces variability in response
 One that minimizes social desirability
 One that is pretested
 Ask questions one at a time
Bad question: Better question:
In the past 6 months, what
major appliances has your
household purchased new
from the store?
Now I’m going to read a list of
household appliances. As I
read each one, please tell me
whether or not your
household has purchased this
type of appliance new from
the store during the past 6
months. How about…
a refrigerator?
a kitchen range or oven?
a microwave?
 Ask questions one at a time
Bad Question:
Compared to one year ago, are
you paying more, less, or about
the same for your auto and life
insurance?
Better Question:
Compared to one year ago, are
you now paying more, less, or
about the same for …
a. auto insurance?
b. life insurance?
 Specify
Specify who, what, when, where and how.

For example, whose income? What’s included?

Over what period of time? Example:

“In 2002, what was your total household income,

before taxes? Please count income from all
members of your household, including wages
from employment, disability, social security, and
public aid.”
 Specify through cues
Example
People drink beer in many places – for
example, at home, at restaurants, at bars,
sporting events, at friends’ homes, etc.
During the past 30 days, did you drink
any beer?
Use words with singular meanings
Ambiguous: More Clear:
How would you Compared to your last
compare how close you neighborhood, do you
are to family in your
now live closer to your
current neighborhood
to how close you were family, are you further
in your old from your family, or are
neighborhood? Would you about the same
you say your family is distance?
closer here, further
here, or the same?
 Social Desirability
 Respondents will try to represent
themselves to the interviewer in a way that
reflects positively on them

As questions become more threatening,

respondents are more likely to overstate or
understate behavior, even when the best
question wording is used
 Minimizing Social Desirability
• For socially desirable behavior, it is better to ask whether
respondents have ever engaged in the behavior before asking
whether they currently engage in the behavior
• For socially undesirable behavior, it is better to ask about
current behavior first, rather than ask about their usual or
typical behavior
• Train interviewers to maintain a professional attitude
• Self-administered computer-assisted procedures can reduce
question threat and improve reporting on sensitive questions
• Longer questions reduce sensitivity when obtaining
information on frequencies of socially undesirable behavior
 Open vs. Closed Questions
General rule: closed questions are usually
better
– Easier for the respondent
– Less coding later
– Better to have respondent do categorizing
– Categories help define the question
 Disadvantages of
Closed Questions
• Categories may be leading to respondents
• May make it too easy to answer without
thinking
• May limit spontaneity
• Not best when
– asking for frequency of sensitive behaviors
– there are numerous possible responses
 General Principles for
Response Options
• Response categories should be consistent with
the question
• Categories must be exhaustive, including every
possible answer
• Categories must be mutually exclusive (no
overlap)
• If appropriate, include a “don’t know” category
 Constructing Response Scales
• Respondents can generally only remember a maximum
of 5 responses unless visual cues are used
• Using graphic images such as thermometers and ladders
and using card sorting for compex ratings is effective
• Number of points in scale should be determined by how
you intend to use the data
• With scales with few points, every scale can be labeled; in
longer scales, only the endpoints are labeled
 Ordering Response Categories

• Usually better to list responses from the lower

level to the higher level
• Associate greater response levels w/ greater
numbers
• Start with end of a scale that is least socially
desirable
Should you use a middle category?
– Common practice is to omit it to push respondents (Rs)
toward one end or the other, on the theory that few
individuals are truly in the middle on a particular issue
– Evidence from empirical studies shows that use of an
explicit middle alternative will often be taken by Rs in a
forced choice situation if offered; at the same time, it
does not affect the ratio of ‘pro’ to ‘con’ responses or
the size of the don’t know category
– Our usual recommendation is to include it unless there
are persuasive reasons to exclude
 Part 2:

How to draft and organize your

questionnaire
 Ordering the Questions
• Segment by topic
• Ask about related topics together
• Salient questions take precedence over less salient ones
• Ask recall backwards in time
• Use transitions when changing topics – give a sense of
progress through the questionnaire
• Leave objectionable questions (e.g., income) for the end
• Put demographic questions at the end
How to Start the Questionnaire
• Start with easy questions that all respondents
can answer with little effort
• First questions should also be non-threatening
• Don’t start with knowledge or awareness
questions
• First questions should be directly related to the
topic as described in the introduction or
advance/cover letter
 Survey Intro/Cover Letter
Introduction should indicate:
– who is conducting the survey
– the topics to be covered in the survey
– an assurance of confidentiality
– any IRB stipulations
– whether you offer how long it will take
depends on mode, topic, population
 Physical Format of the Self-
Administered Questionnaire
• Careful formatting is necessary to
decrease errors and increase motivation
• Respondent’s needs must always take
priority, followed by interviewer and
data processors
 Physical Format Checklist
• Number all questions sequentially
• Use large, clear type; don’t crowd
• ‘White space:’ Place more blank space between
questions than between subcomponents of questions
• List answer categories vertically instead of horizontally
• Avoid double/triple ‘banking’ of response choices
• Be consistent with direction of response categories
• Be consistent with placement of response categories
 Physical Format, Continued
• Don’t split questions across pages. If necessary
(e.g., question requires 1.5 pages), restate
question and response categories on next page
• Put special instructions on questionnaire as
needed, next to question
• Distinguish directions from questions
• Precode the questionnaire (vs. check boxes)
 Mail questionnaires
• Include a cover letter and contact information
if the respondent needs help
• Use a booklet format
– Easier to turn pages
– Prevents lost pages
– Permits double-page formats
– Looks more professional
– Include a title, graphic, name/address of sponsor on
cover
 Testing the Questionnaire
• Preferable to test the questionnaires with
people like those in your main study
study population
• Test in same mode to be used for main
study
• Consider cognitive pretesting
 Questionnaire Design Steps
1. Decide what information is needed
2. Search for existing questions
3. Focus groups
4. Draft new questions/revise existing ones
5. Sequence the questions
6. Get peer evaluation
7. Revise and test on self/co-workers
Questionnaire Design Steps
8. Think-aloud interviews
9. Revise/eliminate questions
10. Prepare interviewer instructions for
pilot test
11. Pilot test (10-20 cases)
12. Revise eliminate questions based on
respondent & interviewer comments
Questionnaire Design Steps
13. Pilot test again, if necessary
14. Prepare final interviewer instructions
15. Be prepared to modify questionnaires if
interviewer training raises problems
16. After interviewing is complete, debrief
interviewers for potential problems
17. Use experience from one study for
future planning
Key References To Get Started
Dillman, Don. Mail and Internet Surveys: The
Tailored Design Method. New York: John, Wiley
& Sons, Inc, 2000.
Bradburn, N, Sudman, S. and Wansink, B.. Asking
Questions: The Definitive Guide to Questionnaire
design – for Market Research, Political Polls, and
Social and Health Questionnaires. San Francisco:
Jossey Bass, 2004.

See also past issues of Public Opinion Quarterly