Creating Effective Surveys:

Design and Question Formation

All College Day
January 26, 2018
Inge Bond, Brian Goo, Erik Hou
Office of Research, Planning, and Institutional Effectiveness
Session Objectives:
• Steps to survey design
• Getting your questions right
• Response options
• Survey resources at Mission
• Survey resources, external
Steps to Survey Design: Rationale
• What problem are you trying to solve?
• Ask yourself: Can I take action based on what I learn?
• What question(s) are you trying to answer?
• Is there another way to get this information?
Steps to Survey Design: Respondents
• Who has the information needed to answer your research
question(s)?
• When thinking about your audience:
• Consider equity issues – is your group representative of the population?
• Are there questions you can ask that will help you to identify
pertinent subgroups?
• Demographic information
• Habits (e.g. drivers versus cyclists)
Steps to Survey Design: Deployment
• Online (electronic) surveys:
• Benefits: respondents can take their time, feel less recognizable in their
responses
• Drawbacks: harder to gather responses
• In person (especially in-class):
• Benefits: captive audience – participation is likely to be high
• Drawbacks: respondents may not be as candid as they would be in an
electronic survey
Steps to Survey Design: Survey Format
• Explanation and Consent:
• Respondents must be competent and willing to answer the questions
• Make the purpose of your survey clear to the respondents
• Describe your anonymity/confidentiality policies and procedures
• Keep it short and simple
• Shorter surveys = higher response rates
• Arrange questions in a logical order
• Incentives
• Increase response rates
• Pre-test your survey
• Have 3-5 colleagues test your survey out to look for grammatical errors, branching
issues, and overall clarity
Steps to Survey Design: Questions
• Make sure that every question is necessary
• Identify the data you need to collect, then write your questions
• Use language that is simple and familiar to your respondents
• Avoid acronyms or other “edu-speak” (know your audience)
• Ask one question at a time
• Expecting a single answer to a question that has multiple parts – muddies the
responses and can confuse/frustrate respondents
• and is a red flag of a double-barreled question
• Example question: “Rate the effectiveness of Basic Skills STEM courses in
student success”
Steps to Survey Design: Questions
• Avoid leading or biased questions
• Look at your adjectives and adverbs – if not needed, take them out
• “How much do you like…”
• Avoid prestige bias
• Some will exaggerate where they think a response is desirable (such as
working long hours).
• Avoid negative questions
• Easily misinterpreted
• not, prohibit, impossible are red flags of negative questions
Steps to Survey Design: Questions, cont’d
• Open-ended questions:
• Enable respondents to provide information beyond what a fixed set of
response options would
• Entail more complex data analysis
• Close-ended questions:
• Greater uniformity of responses
• Enable scaled responses
• Simplify data analysis
Steps to Survey Design: Likert Scale Questions
• Use response scales whenever possible:
• Allows for nuance and a more accurate data than binary
response
• Ideally, between five and seven points is best
• In general, providing a middle category provides better
data
• Only caveat is if you want to make respondents give a non-
neutral answer
Steps to Survey Design: Likert Scale Questions
• If possible, avoid agree/disagree questions, as respondents can be biased
toward the “agree” option(s) (alternatives include measures of likelihood or
satisfaction)
• Example scaling options
• https://www.uc.edu/content/dam/uc/sas/docs/Assessment/likert-
type%20response%20anchors.pdf
• Types of scales
• Satisfaction
• Awareness
• Importance
• Support/Opposition
Steps to Survey Design: Creating the
Survey/Survey Platforms
• Survey Monkey – The college has established accounts
• Advanced data analysis and survey export
• Office365/One Drive Forms – New-ish. Comes with the college’s
subscription to Office365
• For surveys of people with WVM id’s this can be a good way to limit
participation
• Google Forms – Most established open survey platform
Survey Resources at Mission College
• ORPIE is here to help!
• Survey schedule:
http://missioncollege.org/research/documents/surveys/Surveys_Scheduled_
2017_2018.pdf
• Avoiding survey fatigue
• Instrument review
• Survey Monkey accounts:
• ORPIE (no public access – used for surveys with high need of confidentiality)
• Office of Instruction (access through Patty E.)
• Office of Student Services (access through Zita M.)
Survey Resources, External:
• Harvard University Program on Survey Research Tip Sheet on
Question Wording
• Survey Monkey survey guidelines
• Qualtrics Research Success Kit (requires registration)
• FERPA compliance
Questions? Comments?