Think Tank Methods:

Quantitative Survey Analysis

On Think Tanks #TTMethods Session 3

Courtney Tolmie
October 21, 2015

Session Plan
Introduction what is Quantitative Analysis?
Quantitative Surveys and Data
Methods of Analysis
Deeper dive Linking Context study

Introduction what is Quantitative Analysis

Quantitative analysis use of statistical/econometric
techniques to measure relationships between independent
and dependent factors
Types of Quantitative Analysis:
Summar
y Stats

Cross
tabulatio
ns

OLS
regressio
n

Structura
l
equation
models

Complexity, logistical needs, and

sophistication

Introduction Advantages and Disadvantages

Allows assessment of trends across larger
samples
Advantages

Allows for simpler comparisons of unlike

units
Allows for corrections of
endogeneity/unobserved heterogeneity
Allows researcher to assess causality, not just
correlation
Methods can gloss over nuanced differences

Disadvantage Methods harder to have revealed

s
conclusions/findings
Logistics the challenge of data

Quantitative Surveys and Data - Options

Use Existing
Data

Collect New
Data

Do Both

Quantitative Surveys and Data Considerations for

Collecting Data
Developing the model to inform variables dependent and
independent
Unit of analysis organization, other?
Sampling size and bias
Survey cross-section, repeated cross-section, panel
Method of data collection in person surveys, phone, electronic
Is there existing data that you can merge?

Analysis - Overview
Summar
y Stats

Cross
tabulatio
ns

OLS
regressio
n

Structura
l
equation
models

What do you WANT to test?

Change over time

Changes compared to the counterfactual
Correlations
Causation

What CAN you test with the data you have?

Analysis Summary Statistics and Cross Tabs

Summar
y Stats

Cross
tabulatio
ns

OLS
regressio
n

Structura
l
equation
models

Summary Statistics
Averages for variables across the dataset
Comparing averages across subgroups of dataset
Cross tabulations
Basic correlations between variables in the dataset
Significance tests possible for cross tabulations or pairwise
tabulations

Analysis Regression analysis

Summar
y Stats

Cross
tabulatio
ns

OLS
regressio
n

Structura
l
equation
models

More sophisticated analysis of relationships between

variables
What regressions can (but will not necessarily) do:
Assess impact of changes in think tank characteristics or other
factors over time (panel data)
Correct for unobserved heterogeneity
Assess causation

Analysis an example
Think tank
funding
Qualitative research
Summary stats

Placement of
research in
top-tier
journals

Averages

Cross tabulations
Significance of relationship between these

Regression analysis
Direction of relationship
Other important factor (number of PhD researchers on staff, for
example)

10

Case Linking Think Tank Performance, Decisions,

and Context
Research by Results for Development Institute and University
of Washington, with funding from the Think Tank Initiative
Objective explore the relationship between political,
economic, and social contexts and think tanks strategic
behavior and performance
Mixed methods literature mapping, comparative case
studies, interviews/FGDs, and quantitative analysis
Study completed in 2014

11

Linking Context the model

Unit of analysis individual organization

Context variables country level
Third term how exogenous context interacts with
endogenous characteristics
Challenges:
How do we define outcomes?
How do we assess think tank characteristics?

12

Linking Context the hypotheses

13

Linking Context the data

Think tank data
Electronic survey
Convenience sample
Three languages English,
Spanish, French
94 respondents in 48 countries
Combined with existing datasets

Challenges:
Response rate 94 of over 300
sent
Completion 61%
Development of survey itself

14

Linking Context other data sources

15

Linking Context analysis summary statistics

16

Linking Context analysis cross tabulations

17

Lessons
Significant challenges to fielding quantitative survey in the
field should be weighed out
With proper data, quantitative analysis has potential to
assess causation and correct for endogeneity
Smart mixed methods is best

18

Thank you!

19 | R4D.org