Econometrics Review for ECON215

Seonghoon Kim
(courtesy of Bryant Kim at Cornell)
School of Economics
Singapore Management University

August, 2015

1 / 35

Introduction
I

What is econometrics?
I

I
I

A set of statistical techniques that allow us to examine

empirical relationships between variables.

Empirical: Based on data

Relationship:
I

Causal: Assess consequence that changing the value of one

variable (X) has on the value of another variable (Y)
Association: Assess the extent to which two variables move
together in the data. This affects the predictability that one
variable has on another and vice versa.

Key idea: Throughout the course, we will constantly be

asking whether the relationship being examined is causal or an
association.

QUESTION: Why is this important for public policy?

2 / 35

Introduction
Example - A study published in Nature (Rauscher, Shaw and Ky,
1993) suggested that listening to Mozart for 10-15 minutes could
temporarily raise your IQ by 8 or 9 points. In fact, shortly after the
study was published and reported in the popular press, the U.S.
state of Georgia began handing out classical-music CDs to the
parents of all infants, and there were similar but less official
programs in Colorado, Florida and elsewhere.

QUESTION: Would this be a good or a bad policy to increase

the intelligence of children?

3 / 35

Correlation vs. Causality

4 / 35

Introduction
There are two types of causal relationships commonly assessed in
the policy world:
I

Effect of one variable on another

I
I

How much does maternal nutrition affect infant birth weight?

How much do political campaign expenditures affect voting
outcomes?
Does increasing the minimum wage increase unemployment?
I

1% increase in state-level minimum wages reduces

employment of young blacks and Hispanics by 0.5% to 0.6%.
(Neumark and Wascher, 2007)

Other?

5 / 35

Introduction
I

Effect of a social program

I

Does giving deworming drugs to children in Kenya improve

their health status? Their school attendance?
I

Does an extended school day program affect student

outcomes? Test scores? Drop-out rates? Other measures of
achievement? By how much? Does this vary across different
student populations?
What is the effect of a housing subsidy program on
employment?
I

school absenteeism by 25% but no evidence on test score.

(Miguel and Kremer, 2004)

Among working-age, able-bodied adults, housing voucher use

reduces quarterly labor force participation rates by
4percentage points (Jacob and Ludwig, 2008)

Other?

6 / 35

Causality
Causality: A specific action leads to a specific, measurable
consequence
I We are often interested in assessing the magnitude of the
causal effect that a certain factor (X) has on an outcome
(Y)
I
I

Y may be a function of many factors other than X

We want to establish the causal link between X and Y. We
want to measure any changes in Y that are directly
attributable to X, not to other factors.
Changes in X may be linked with (correlated with) changes in
Y, but this alone is not sufficient to establish a causal link.

Examples:
I
I

Do car seatbelts (X) save lives (Y)?

Do industrial emissions (X) cause the temperature of the
planet (Y) to rise?
Do smaller classes (X) increase student learning (Y)?
7 / 35

Causality
I

A key concept needed to answer these causal questions is the

counterfactual
I
I

What would have happened otherwise

Compare outcome Y when X occurs and when X does not
occur

QUESTION: What is the counterfactual for each of the 3

examples above?
I

Ideally, we want to compare the state of the world at a

given point in time with the alternate state of the world
or counterfactual at that same point in time.

QUESTION: Conceptually, how can we use the concept of the

counterfactual to assess the causal effect of being in a small
class on the test score of a given individual?
8 / 35

Causality
I

The difficulty of inferring causality is that we do not observe

the counterfactual
I

We do not observe what would have happened to seatbelt

wearers, if they had not been wearing seatbelts.
We do not observe what the temperature of the planet would
have been at the current time, had there been a different level
of industrial emissions.
We do not observe how students in large classes would have
fared at the same point in time had they been in small classes.

9 / 35

Causality
I

While the counterfactual cannot be directly observed, the goal

of empirical analysis aimed at uncovering causal relationships
is to mimic the counterfactual using data and statistical
techniques.
I
I

Ideal situation: randomized controlled experiment

More common situation: observational data analyzed with
econometric techniques

10 / 35

Validity
Validity represents a set of criteria by which the credibility of
research may be judged.
I

A key goal of any research study should be to achieve high

validity

Our focus: internal validity and external validity

11 / 35

Validity
Internal Validity
A study has strong internal validity if it estimates the causal
effect of interest for the population represented by our sample.
I

To what extent does the evidence presented support a causal

link between X and Y, for this population?

Key question: Is there any factor other than X that could be

partly responsible for the observed association between X and
Y?
I

I
I

Generally not a problem in a properly conducted randomized

experiment
Questions more likely to arise when using observational data
Linked to assumptions behind regression models (more on this
later in the course)

12 / 35

Validity
External Validity
A study has strong external validity if its findings can be
generalized to other settings (i.e. people, time periods, locations,
age groups, etc.)
I

Do the conclusions hold for other geographic locations,

socioeconomic conditions, time periods?

Do other empirical studies investigating the same (or similar)

research questions yield the same(or similar) results?

Tools to improve external validity:

I
I

Selection of the sample

Replications of a given evaluation in other times/locations

Internal and external validity in the context of empirical study

I
I

Random Assignment
Random Sampling
13 / 35

Randomized Experiments
I

Conceptually, this is the ideal method to estimate the causal

effect of a treatment(program, intervention, etc.) - Gold
standard

Also known as random assignment studies, social experiments,

randomized controlled trials, randomized trials, etc.
I

Randomized experiment compares two groups that are alike

except for treatment
I

Participants volunteer for the experiment. This constitutes the

sample.
The sample is randomly divided into treatment and
control groups XCoin flip, random number generator
(assigned by a computer), etc.
XTreatment group is offered the treatment
XControl group does not get the treatment, and may get
a placebo.
XControl group is meant to mimic the counterfactual

14 / 35

Randomized Experiments
I

Isolate the causal effect of one factor on outcomes

F
F

Only difference between two groups is the treatment.

Any difference in outcomes must be due to the treatment.

Randomized Experiments in the Social Sciences

F
F
F

About 11,000 known experiments in social sciences

Contrast with over 250,000 in medicine
Wide variety of areas: Poverty, Labor, Health, Education,
Crime, etc.
Examples of randomized experiments:
XEffect of a deworming medication on health outcomes Kenya
XEffect of cost of health insurance (premiums, co-pays) on
health outcomes - RAND
XEffect of class size on student outcomes - Tennessee STAR

15 / 35

Randomized Experiments
I

Randomized Experiment
I

Methodologically, they are the ideal way to mimic the

counterfactual and draw causal inference
This does not mean that all policy-relevant questions can be
addressed with experiments or that experiments always yield
valid answers to causal questions

Today we will examine a policy-relevant causal question using

both experimental and non-experimental (i.e.observational)
methods.

QUESTION: Is random assignment (to treatment vs.

control group) the same as random sampling?

16 / 35

Experiment Vs. Observation Studies

Tennessee STAR Experiment:
I

Project STAR (Student-Teacher Achievement Ratio)

I
I
I

1980s
Four-year study
Examined the effect of class size in grades K-3

Students entering the school system were randomly assigned

to one of three groups:
I
I
I

Small class (13-17 students)

Regular class (22-25 students)
Regular class with teachers aide

Compare outcomes of children in small classes with outcomes

of children in regular classes.

Our focus will be kindergarten

Question: What are the treatment and control groups in this

case?
17 / 35

Experimental Data
I

The great benefit of an experiment to address this question is

the internal validity we gain by randomization.
I

Internal validity in this context means that we can be confident

that any differences in the outcomes between the treatment
and control groups indicate a causal effect of the treatment
Key: The treatment and control groups are alike except for
their treatment status

Did randomization work?

I
I

We can review the protocol for the experiment

We can look for evidence to assess whether the protocol was
followed
We can look at the treatment and control groups to see if they
are comparable
This evidence is very important since the control group can
mimic the counterfactual only if the treatment and control
groups are alike before the treatment.

18 / 35

Experimental Data
Mean Background Characteristics
Free Lunch(%)
Male(%)
Black(%)
+

Treatment
47.2
51.5
31.1

Control
48.5
51.3
32.5

Difference
-1.3
0.2
-1.4

P-Value+
0.325
0.883
0.302

: Corresponds to a Z test where the null hypothesis is that the proportions in the treatment and control

are the same

Question: Based on the table above, did randomization work?

19 / 35

Experimental Data
I

Return to our policy question: What is the effect of small

class sizes on test scores?
How do we address this question explicitly?
I

Estimation: Compare average test scores in small classes with

average test scores in regular classes.
Hypothesis Testing: Test the null hypothesis that average test
scores are the same between small and regular size classes
Confidence Interval: Estimate a confidence interval for the
difference of mean test scores in small vs. regular size classes.

20 / 35

Experimental Data
Estimation
. tab small, sum(tscorek)
small class |
Summary of tscorek
in K |
Mean
Std. Dev.
Freq.
------------+-----------------------------------0 |
918.20133
72.214225
4048
1 |
931.94189
76.358633
1738
------------+-----------------------------------Total |
922.32872
73.746597
5786
Question: What is the difference in average test scores between
small and regular classes?

21 / 35

Experimental Data
Hypothesis Testing
H0 : T = C
HA : T 6= C
YT YC
t=
SE (YT YC )
Where SE (YT YC )is the standard error of YT YC
s
(YT )2 (YC )2
SE (YT YC ) =
+
nT
nC
t=

YT YC
931.94 918.20
13.74
q
=
=
= 6.38
2
2
2.15
SE (YT YC )
(76.36)
(72.21)
1738 + 4048

22 / 35

Experimental Data
QUESTION: Is this difference in mean test scores statistically
significant at the 5% level?

95% Confidence Interval

(Y T YC ) 1.96SE (Y T YC )

23 / 35

Experimental Data
Conclusions from analysis of STAR Experiment (Kindergarten Result)

Being in a small class rather than a regular-sized class

I
I
I

Increases average test score by 13.74 points.

Difference is highly statistically significant
What about significance from a policy perspective?
I
I
I
I

Effect size = (effect on Y)/(standard deviation of Y)

Effect size= (13.74/73.74)
Effect size= 0.19 standard deviations of test score
Useful for cost-benefit analysis

24 / 35

Observational Data
Observational Data for the Class Size and Student Outcome
Question: California Test Score Data Set
I

All K-6 and K-8 California school districts (n=420)

Class size measured by student-teacher ratio (STR) = number

of students in the district dividing by number of full-time
equivalent teachers.

This measure of class size says nothing about what is going on

in individual classrooms.

25 / 35

Observational Data
Look at the data in a scatterplot:

QUESTION: What does this figure show?

26 / 35

Observational Data
How do we address the question of whether students do
better in small classes, using these data?
I

Divide the school districts into those with small classes (STR
less than 20) and those with larger classes (STR more than
20).

As before, use estimation, hypothesis testing, and confidence

interval to ask the same question we did with the experimental
data

27 / 35

Observational Data
Estimation:
. tab small, sum(testscr)
|
Summary of testscr
small |
Mean
Std. Dev.
Freq.
------------+-----------------------------------0 |
649.97885
17.853364
182
1 |
657.35126
19.358012
238
------------+-----------------------------------Total |
654.15655
19.053348
420
QUESTION: What is the difference in average test scores
between small and regular classes?
28 / 35

Observational Data
I

In test score points:

Hypothesis Testing:
H0 : small = large
HA : small 6= large
t=

Ysmall Ylarge
7.4
=
= 4.05

1.8
SE (Ysmall Ylarge )

95% Confidence Interval:

(Y small Ylarge ) 1.96SE (Y small Ylarge )
[3.8, 11.0]
QUESTION: Have we answered the key question: What is
the effect of small classes on test scores in California?
29 / 35

Observational Data
Compare characteristics of treatment and control groups:

Free Lunch(%)
English as a second language(%)
Average Income(thousand $)
+

Small
41.6
12.5
16.3

Regular
48.7
20.0
14.0

Difference
-7.1
-7.5
2.3

P-Value+
0.001
0.001
0.001

: Corresponds to a Z test (t test) where the null hypothesis is that the proportions (means) in the

treatment and control groups are the same

Question: Is the control group here a good estimate of the

counterfactual for the treatment group? Why or why
not?

30 / 35

Observational Data
Conclusions from analysis of non-experimental data:
I

In the context of an ideal experiment, the difference in mean

outcomes between treatment and control groups can give us a
good estimate of the causal effect of the intervention.

In the context of an observational study, the difference in

mean outcomes across groups does not usually give us a good
estimate of the causal effect because the groups are rarely
alike

We need another tool since experiments are not always

available

31 / 35

Observational Data
I

Regression analysis can be a powerful tool for assessing causal

effects.
I

The simplest form of regression (bivariate regression), which

we will start with, is not much better for internal validity than
this difference of means.

But regression more generally is very powerful to get around

some of the problems with internal validity using observational
data.

32 / 35

Why We Need Econometrics

I

Key goal of this course is to develop your ability to assess

critically the quality/credibility of empirical studies on health
economics.

Randomized experiments are a benchmark

I

Not always available or possible (expensive, difficult to

administer, ethical issues)
Not always flawless in providing an answer to the important
policy question
BUT provide good estimate of causal effect when designed and
conducted properly

33 / 35

Why We Need Econometrics

I

In the absence of a randomized experiment, what can we do?

I

Analyze observational data

I
I

Surveys (CPS), administrative records

Key distinction: Variable or factor of interest not randomly
assigned

Use econometric techniques to conduct regression analysis

and look for causal effects

34 / 35

Why We Need Econometrics

I

Additional Topics in Evaluating Causal Relationships

I
I

Issues with randomized experiments

Regression-based techniques to assess causal effects
I
I
I
I

Differences-in-differences
Fixed effects
Instrumental variables
Regression discontinuity

35 / 35