Collecting primary data: questionnaires

Questionnaire
Strict definition

Surveys where the person answering the question actually records their own answers.

Broad view

Include all techniques of data collection in which each person is asked to respond to the same set of questions in a predetermined order.

Important design issues

It

is far harder to produce a good questionnaire than you might think Issues to maximize response rates, validity and reliability

Careful design of individual questions (item) Clear layout of the questionnaire form logical explanation of the purpose of the questionnaire Pilot testing
3

When to use questionnaires

Not particularly good for exploratory study or other research that requires large number of open-ended question

What would be the most suitable data collection method for exploratory study?

Work best with standardized questions that you can be confident will be interpreted the same way by all respondents.

When to use questionnaire

Descriptive research

Opinion, attitude and experience of something Eg. End-user satisfaction of a new system

Experimental research

Cause-and-effect relationship between variables Eg. Would color scheme of an e-commerce website affect customers purchasing behavior

Multi-method

Eg. A questionnaire to discover customers attitudes followed by in-depth interviews to explore and understand these attitudes.

When to use questionnaire

To collect self-reported beliefs or behaviors.

Behavior Attitudes/beliefs/opinions Expectations Self-classification Knowledge

Require less skill and sensitivity to administer than interviews. Data are usually quantified and analysed by computer program
6

Types of questionnaire

Self-administered questionnaires

Completed by the respondents On-line questionnaires Postal or mail questionnaires Delivery and collection questionnaires Responses are recorded by the interviewer Telephone questionnaires Interviewers physically meet respondents and ask the questions face to face.
7

Interviewer-administered questionnaires

Structured interviews

The choice of questionnaire

Characteristics of the respondents from whom you wish to collect data Importance of reaching a particular person as respondent Importance of respondents answers not being contaminated or distorted Size of sample you require for your analysis Types of questions you need to ask to collect your data Number of questions you need to ask to collect your data. Available resources

The choice of questionnaire

It

is difficult to know the actual respondents from online or postal questionnaire Possible contamination of respondents answers

In telephone or structured interview, interviewer might lead to some certain response In self-administered questionnaire, respondents might discuss answers with others

The choice of questionnaire

Telephone

or structure interview usually get high response rate (50%-70%) Postal or delivery and collection questionnaire can get moderate response rate (30%-50%) Online questionnaire usually get lowest response

30% within organization; 10% or lower for public Internet survey

10

Deciding what data need to be collected

Example

Problem: towards the end of your information system project, you need to measure the satisfaction of your client Solution: Email your client a short questionnaire contains two questions

1. Are you satisfied with the system we developed? YES/NO 2. Are you going to use the system in the future? YES/NO

11

Deciding what data need to be collected

Theory

Suggest constructs of questionnaire How do we measure End User satisfaction of computer system?

Content, Accuracy, Format, Ease of Use, Timeliness Voluntariness, Relative Advantage, Compatibility, Image, Ease of use, Result Demonstrability, Visibility, Trialability

How do we measure perception of adoption?

www.isworld.org is an excellent source for Information system survey instruments! http://www.isworld.org/surveyinstruments/tutor.htm

12

Type of data we collect through questionnaire

Attribute

Respondents generally background or some special characteristics Eg. Do you think the workload of this course is too high? What do respondents do? Eg. How often do you visit your dentist?
13

Opinion

Behavior

Ensuring that essential data are collected

What question (s) you want to answer from the questionnaire? What might influence a consumer to purchase online? What constructs you might need to answer those questions Online payment, Internet product choice, Internet customer relation, What variables you need to collect data for those constructs Customer relation Want to know customers opinion on:

It is important to provide an easy return process It is important to provide quality after-sale service.

How you might measure those variables Rating or scaling

14

Designing the questionnaire

Designing

individual questions (items)

Adopt questions used in other questionnaires

See survey instruments in ISworld Lots of instruments (questions) reported in academic papers

Develop your own

15

Individual question style

Open-ended question

Respondents answer in their own words Respondents choose a response from those provided Why do you blog? (OPEN) Which one of the reasons make you start your first blog? () share daily activities as well as pictures with family members () share information on a particular hobby () it makes me look cool () other reasons
16

Closed-ended question

Examples

Open vs. Closed

Open-ended question

Great freedom for respondent to answer Responses may be ambiguous Coding is time-consuming and costly which usually results in some degree of error Entail more work from respondents

Closed-ended question

Require less effort and less facility with words Difficult to develop good closed questions Recommendation for designing closed-end question: use open questions in preliminary interviews or pretests
17

Closed question style

List

The respondent is given a list of choices, any of which may be selected Only one response can be selected Respondent is asked to place something in order A rating device is used to record response Response is a number giving the amount
18

Category

Rank

Rating or Scale (Likert style)

Quantity

List questions
Eg.

Please choose all communication tools you ever used in campus.

Internal Message Board WebCT E-mail Other ___________

19

Category question
Each

respondents answer can only fit one category Eg.

How often do you visit this shopping center

Once a week Twice a week or more Less than once a week

Whats wrong with this responses?

20

Ranking question
Rank-order the three most important things you want in the job you make your lifes work? (1 indicates the most important one 4 the least important one) ______ ______ ______ ______ Making a lot of money Being creative Being free from supervision Having opportunities for advancement

Should not ask respondent to rank more than four levels In telephone questionnaire, should not ask respondent to rank more than four responses
21

Rating or Scale question

Likert-style question

Ask respondents how strongly they disagree or agree with a statement or a series of statements

For the following statement please circle the number that matches your view most closely I feel that employees views have influenced the decisions taken by management.

1 Strongly Disagree

2 Disagree

3 Neutral

4 Agree

5 Strongly Agree

22

Quantity question
Eg.

When did you start writing blog? ________

23

Opening and closing of questionnaire

A cover letter explaining the purpose of the questionnaire Include date, recipients name, address and salutation if available First set of messages What research is about why it is useful Second set of messages Why recipients response is important, how long it will take to complete Third set of messages Promise of confidentiality or anonymity Fourth set of messages How results will be used; Final set of messages Whom to contact if have any queries, who to return filled questionnaires to and date by which should be returned Closing remarks Thank recipient for their help Signature, name and title.
24

Pilot testing

Purpose is to refine the questionnaire

respondents will have no problems in answering them You will have no problems in recording the data Obtains some assessment of the questions validity and reliability of the data Ask expert or group of experts to comment on the representativeness and suitability of the questions Administer the questionnaire to a group as similar as possible to the final population in your sample.
25

Steps

Pilot testing

Use pilot testing to find out

How long the questionnaire took to complete The clarity of instructions Which, if any, questions were unclear or ambiguous; Which, if any, questions the respondent felt uneasy about answering Whether in their opinion there were any major topic omissions Whether the layout was clear and attractive Any other comments.
26

Administer the questionnaire

Online

questionnaire / postal questionnaires

Pre-survey contact Contact the recipients by email/post Send out the first follow-up one week after the first contact Send out the second follow-up to people who have not responded after three weeks. Send out a third follow-up if time allows or the response rate is low
27

Administer the questionnaire

Delivery

and collection questionnaires

Pre-survey contact Same day vs. different days

Telephone

questionnaires

Try to make pre-survey contact Record date and time of call and whether or not the questionnaire was completed down Try a few more times for unsuccessful call
28