Survey Research Methods

Presentation to the class of CIS 9280 Quantitative Methods in IS Research at Georgia State University By Punit Ahluwalia Date: Oct 23rd2002

History
Systematic enumerations used to conduct census for centuries Social surveys between late 1800s and 1920s on conditions of urban poor in many cities in USA, Britain, Germany and other countries

Resembled modern surveys but did not involve sampling

Late 1940s called early adulthood in life of survey methods [Ornstein]: basic elements of questionnaire design, sampling, interviewing and data analysis came together as a method of social research

History Cont..
Incremental evolution of questionnaire design paralleled by introduction of computers accelerating generation of cross tabulations and facilitating factor analysis and regression analysis. Early 1970s: Criticism that the method was empiricist and untheoretical led to project by US national Academy of Science known as CASM (Cognitive aspects of survey methodology) Lazarsfeld, Gallup and Centril are identified as inventors of modern survey research method. Led by applications in social and political science research, survey as research method is now established across many disciplines. An introduction about surveys and a repository of survey instruments on IS World website http://www.ucalgary.ca/~newsted/surveys.html

Some statistics of survey studies in MIS

Between 1980 and 1990

[Pinsoneault and Kraemer]

Surveys in exploratory or descriptive studies

Before 1985 :70% After 1985 :54%

Surveys in explanatory studies

Before 1985 :30% After 1985 :46%

Longitudinal Studies
3 out of 122 (2%)

Multiple research methods

6 out of 122 (5%)

Pre-test
93 out of 122 (76%) either did not pre-test the questionnaire or did not report

about it.

Response rate
90 out of 122 studies had response rate less than 51%

Exploratory and descriptive studies are of moderate to poor quality overall. Explanatory studies are of good quality.

Surveys with Class A instruments

This slide was used with permission from the author)

Technically acceptable instruments

A Technically Acceptable Survey Instrument (Class A): Zmud and Boynton (1991) considered that an instrument met this criterion if it had at least the four following characteristics:

Its construct(s) were each measured by multiple items. An attempt was made to assess the instrument's psychometric properties (i.e. reliability and validity). Its use was described in a referred journal. It could be reconstructed.

Definition
Epistemologically surveys provide one way of obtaining and validating knowledge Websters Dictionary defines a survey as a general study or inspection A survey is a means of "gathering information about the characteristics, actions, or opinions of a large group of people, referred to as a population" [Pinsonneault and Kramaemer]

Survey Research
Survey for research is distinctly different Focus on surveys that are conducted to advance scientific knowledge, which we refer to as survey research.

Distinctiveness of survey research

The purpose of the survey is to produce quantitative descriptions of some aspects of the studied population.. requires standardized information from and/or about the subjects being studied relationships between variables projecting findings descriptively to a predefined population The main way of collecting information is by asking people structured and predefined questions. Their answers constitute the data to be analyzed. Information is generally collected about a fraction of the study population--a sample--but it is collected in such a way as to be able to generalize the findings to the population-such as service or manufacturing organizations, line or staff work groups, MIS departments, or various users of information systems.

Techniques
Survey research combines three techniques

Collection of answers to standardized questionnaires Random sampling from a known population Statistical analysis of a quantified representation of the survey answers

Research methods in IS
Case Study Examination of phenomena in natural setting No control over phenomena Research variables may or may not be defined

Laboratory experiments

Controlled setting Researcher manipulates independent variables to observe effects on dependent variables Examination of phenomena in natural setting Clearly defined independent and dependent variables Phenomena of interest occur in current time or in recent past Control of Research variables is not desirable

Survey

Purposes
Exploratory

Little is known about a population Further information is desired about research variables Prelude to a costlier, larger research endeavors Focus on who, what, when, how Existence of opinions and attitudes Cause and effect, focus on why Reasons for existence of facts and opinion is of interest

Descriptive

Explanatory

Key problems
Each focus makes different demands of the data which are to be collected. Three key problems :

Measurement Control Representation

Survey stages
Clarify the research problem Identify the practical constraints Decide what type of survey design is appropriate in light of 1 and 2 Decide what type of sample design is most appropriate Decide on the size of sample Plan sampling procedures What form of data collection Design the questions and construct or select any tests to be used Undertake the fieldwork and data collection Process the data Analyze the data and interpret your results Disseminate your findings

Clarify the research problem

Move from general idea to focused topics Degree to which study meets its aims is determined will be determined by relevance and completeness of research questions addressing the research problem

Empirical to theoretical mapping

This slide was used with permission from the author

Identify practical constraints

Time Finance Personnel Research sponsor

Specification of research topic Constraints within which design must operate Dissemination of findings Effect on the field credibility of the survey

Decide type of survey

Cross-sectional or one-off Longitudinal or time series ( collection of data at two or more points in time)

Before and after studies Panel studies

Combination of Cross-sectional and longitudinal

Sample design
Decide which type of sample design is most appropriate

Probability samples
Simple random samples Cluster samples

Non-Probability samples

Quota samples Volunteer samples Convenience samples Snowball samples Purposive sampling

Errors
Sampling errors

Difference between the results you will get from a full census and the results you will get from a sample Systemic Random

Non-Sampling error

Sample Size
Inevitably a compromise between methodological imperative and practical constraints Determined by level of sampling error How about Non-response? Formula [Calder Judith, 1998]
n = (desired confidence level)2 * (s.d) 2 /desired level of precision) 2

Decide form of data collection

Questionnaires Interviews Observation

Design the questions

Presentations by:

Stacy and Adriane Ricardo and JJ

Data Collection
Some steps

Organize in advance Agree access to your respondents Collect the data Monitor progress Chasing up Quality of the process of fieldwork and data collection
Report honest and reflect on fieldwork in research

reports[Calder Judith]

Process and transform data

Some issues

Loss of detail due to coding Deal with non-responses

0 1 4 3 2 5 4 1 4 3 X 2 4 3 2 3 1 4 5 3 2

And Finally
Analyze the data and interpret your results

Does the data make sense? Beware of spurious patterns Compare like with like Always allow role of chance in your findings

Present and disseminate your findings

Benefits
Theoretical propositions can be tested in an objective fashion Enable generalizations Easy to use (although may be an illusion) Established and accepted research method across multiple fields Aided by statistical tools Immensely helped by advancement computing Strong in external validity if sampling errors are minimized.

Weaknesses
Difficulty in obtaining truly random sampling Low response rates. Business and IT staff are inundated with surveys Weak linkages between units of analysis and respondents, and Over-reliance on cross-sectional surveys where longitudinal surveys are really needed single-method designs where multiple methods are needed Inappropriateness

Different results with behavioral observation and self report

Usefulness
For Positivist standpoint Good for providing actual, observed values against which to compare the predicted values Not good for telling us what the theory or even the variables are in the first place From Interpretive standpoint Good for complimenting other sources of data documents, participant observations, conversations Not Good for single objective truth
Ref: Wynne Chin (1996)

Attributes
Report the approach used to randomize or select samples Report a profile of sample's frame Report characteristics of the respondents Use a combination of personal, telephone and mail data collection Append the whole or part of the questionnaire Adopt a validated instrument or perform a validity or reliability analysis Perform an instrument pretest Report on response rate Perform a statistical test to justify the loss of data from non-respondents Ref. : Grover,Lee and Durand, Information and Management (1993)

Pragmatism
Interests of academic and governmental researchers are different.[Ornstein] Governmental Researchers Concerned with employment, industry, health etc. Bias, systemic over or under estimation of population characteristics does not diminish with sample size, and is more of a concern than sampling error. Push in direction of careful measurement and doing fewer things more exactly Academic Researchers

Concerned with relationships between variables than estimates of population characteristics. Smaller samples and differences in sub-groups academics are more concerned with sampling error than bias. Work in shorter time frames Less concerned with making small mistakes if their research goals can be achieved.

References
Survey research : Ornstein, Michael, Current Sociology, V46 October 1998 Survey Research methods: Gail Hackett, Personnel and Guidance Journal, May 1981 Survey Research methodology in management information Systems: An assessment: Alain Pinsonneault and Kenneth L. Kraemer: Journal of Management Information System Fall 1993 Research methods for the study of conflict and cooperation: The American Sociologist, 1976, Vol. 11, November Survey Research methods Judith Calder, Medical education 1998, 32 636-652 Research methods in Organizational behavior: Stone Eugene F., Scott Foresman and company

References
Newsted, Chin, Ngwenyama, and Lee, 1996 International Conference on Information Systems Zmud, R. W., and A. C. Boynton, "Survey Measures and Instruments in MIS: Inventory and Appraisal." In K. Kraemer (ed.), The Information Systems Research Challenge: Survey Research Methods, Boston: Harvard Business School, 1991, pp 149-180. [has a list of organization-related instruments, too]