1 1

Experimental and Quasi-

Experimental Research
Ex Post Facto
2
Types of research design experiments
Chapter 8 in Babbie & Mouton (2001)
Introduction to all research designs
All research designs have specific
objectives they strive for
Have different strengths and limitations
Have validity considerations
3
Experimental Research Tries
to Establish Cause and Effect
Selection of a good theoretical framework
Application of appropriate experimental
design
Use of correct statistical model and
analysis
Proper selection and control of
independent variables
Appropriate selection and measurement of
dependent variables
Correct interpretation of results
3
4
Three Criteria for Cause and
Effect
1. The cause must precede
the effect in time
2. The cause and effect must
be correlated with each
other
3. The correlation between
cause and effect cannot be
explained by another
variable
4
5
The logic of causal social research in the
controlled experiment
Explanatory rather than descriptive
Different from correlational research - one variable is
manipulated (IV) and the effect of that manipulation
observed on a second variable (DV)
If then .
E.g.
"Animals respond aggressively to crowding" (causal)
"People with long engagement experience have more stable
marriages" (noncausal)
6
Reviewing Important Terms
Independent variable
Dependent variable
Categorical variable-(age, sex=not controlled)
Control variable-exclude
Extraneous variable

6
7
Types of Validity
Internal validity
External validity
Trade-off between internal and external validity
Series of experiments (studies)
7
INTERNAL VALIDITY
Internal validity is threatened
whenever there exists the possibility
of un-controlled extraneous variables
that might otherwise account for the
results of a study
8
9
Threats to Internal Validity
History-during
Maturation
Testing
Instrumentation
Statistical regression
Selection bias
9
10
Threats to Internal Validity, contd
Experimental mortality
Selection-maturation interaction
Expectancy
10
11
Threats to External Validity
Reactive or interactive effects of testing
Interaction of selection biases and treatment
Reactive effects of experimental arrangements
Multiple-treatment interference

11
12
Controlling Threats to
Internal Validity
Randomization
Real randomization
Matched pairs (but not matched groups)
Randomizing treatments or counterbalancing
Placebos
Blind setups
Double-blind setups
12
13
Uncontrolled Threats to
Internal Validity
Reactive effects of testing
Instrumentation
Experimental mortality
13
EXTERNAL VALIDITY
Concerned with whether the
results of a study can be
generalized beyond the study
itself
14
15
Controlling Threats to External
Validity
Selecting from larger population
Participants
Treatments
Situations
Ecological validity-emulates the real world?
15
Types of design -
experiments 16
The generic experimental design:
R O
1
X O
2

R O
3
O
4

The IV is an active variable; it is manipulated
The participants who receive one level of the IV
are equivalent in all ways to those who receive
other levels of the IV
17
Types of Designs:
Pre-experimental Designs (invalid)
One-shot studies
T O

One-group pretest-posttest
O
1
T O
2
Statistical analysis? Dt

Static group comparison (one shot w/2 groups)
T O
1
--------------- Statistical analysis? It
O
2
18
Types of Designs:
True Experimental Designs
Randomized-groups design
R T O
1
Statistical analysis? It
R O
2


Extending the levelsrandomized-groups
design
R T
1
O
1
Statistical analysis? Ab
R T
2
O
2

R O
3

19
Types of Designs:
True Experimental Designs, contd
-1 interaction
-2 main effects
20
Types of Designs:
True Experimental Designs, contd
A factorial design with a categorical factor (B)
R A
1
O
1

B
1
R A
2
O
2

R A
3
O
3
Statistical analysis? M
----------------------------
R A
1
O
4

B
2
R A
2
O
5

R A
3
O
6


Types of Designs:
True Experimental Designs
Pretest-posttest randomized-groups
R O
1
T O
2

R O
3
O
4
Statistical analysis? MFA

Extending the design on the RM factor
R O
1
T O
2
T O
3
Statistical analysis? MFA-RM
R O
4
O
5
O
6




21
22
Types of Designs:
True Experimental Designs, contd
Extending the pretest-posttest randomized
groups design on both factors
R O
1
T
1
O
2
T
1
O
3
R O
4
T
2
O
5
T
2
O
6
R O
7
O
8
O
9
Statistical analysis? AN (covariate)

23
Types of Designs:
True Experimental Designs, contd
Solomon four-group designpurpose=pretest effects
R O
1
T O
2


R O
3
O
4


R T O
5


R O
6


Statistical analysis (factorial ANOVA)
No treatment Treatment
Pretested O
4
O
2

Unpretested O
6
O
5

24
Quasi-Experimental Designs:
Time Series (slope)
D.T. Campbell and J.C. Stanley, Experimental and quasi-experimental designs for research.
Copyright 1963 by Houghton Mifflin Company. Reprinted with permission.
25
Quasi-Experimental Designs:
Reversal (return to baseline)
26
Quasi-Experimental Designs:
Ex Post Facto (discriminate)
This is one of the pre-experimental designs,
but with the treatment not under the control
of the experimenter.
T O
1

------------------ Statistical analysis? A
O
2


27
Quasi-Experimental Designs:
Switched Replication
Trials
Levels 1 2 3 4 5
1 O
1
T O
2
O
3
O
4
O
5

2 O
6
O
7
T O
8
O
9
O
10

3 O
11
O
12
O
13
T O
14
O
15

4 O
16
O
17
O
18
O
19
T O
20


28
Quasi-Experimental Designs:
Single Participant
Identify participant and follow over time.
Does the treatment produce the same
effect each time?
Are treatment effects cumulative, or does
participant return to baseline?
Does participants response become less
variable over treatment times?
29
Quasi-Experimental Designs:
Single Participant, contd
Is participants magnitude of response
sensitive to multiple treatment
applications?
Do varying intensities, frequencies, and
lengths of treatment produce varying
responses?
Ex Post Facto
How it differ from experimental
research?
30
Purpose of causal comparative
research
Attempts to determine cause for
Existing conditions
Preexisting differences in groups
Alleged cause and effect have already
occurred
Orientations
Retrospective (basic): starts with an effect and
seeks possible causes
Prospective (variation): starts with a cause and
investigates its effects on some variable
Causal-comparative (ex post facto)
research
The independent variable (IV) is not
manipulated; it has already occurred
Independent variables sometimes called
attribute variables
Less costly and time-consuming to
conduct
Establishing cause-effect relationships is
more difficult than in experiments
Procedures in causal-
comparative research
Identify an existing condition or event
(e.g., differences in socialization among
1st grade students)
Look backwards to see what may have
caused this difference/condition to occur
(i.e., some attended preschool, some did
not)
Rule out other causal factors
Sometimes confused with
correlational research:
Both lack manipulation of variables
Both require caution in interpreting
results
Both can support subsequent
experimental research
Causal comparative vs.
correlational research
Causal comparative
Attempts to identify
cause-effect
relationships
At least one
independent variable
Two or more groups
Involves a comparison
Correlational
No attempt to
understand cause and
effect
Two or more variables
Only one group
Sometimes confused with
experimental research:
Both try to establish cause-effect
relationships
Both can test hypotheses concerning the
relationship between an independent (X)
and a dependent variable (Y)
Both involve group comparisons

Comparison to experiments
Causal comparative
Individuals already in
groups before study
begins
Independent variable
has already occurred
Independent variable
is not manipulated
Cannot be
Should not be
Could be, but is not
Experiment
Individuals randomly
assigned to groups
(e.g., treatment or
control)
Independent variable
manipulated by the
researcher
Examples of non-manipulated
independent variables
Age
Sex
Ethnicity
Learning style
Socioeconomic status (SES)
Parent educational level
Family environment
Type of school attended
Design of causal-comparative
research
Select 2 groups that differ on some IV
One group possesses a characteristic that the other
does not
Each group possesses the characteristic, but in
differing amounts
Randomly sample Ss from each group
Collect info on Ss to determine equality of the
groups
Compare groups on the DV
Difficulty in interpreting findings
Establishing cause and effect requires
caution!
Alternative explanations:
Different causal variable
Order of causation
Reverse causality
Order of occurrence
Evidence necessary to
demonstrate that X causes Y:
Establish statistical relationship between X
and Y (i.e., correlational research);
determine that X precedes Y in time
(collect data over time, i.e., longitudinal
research);
demonstrate that other, unknown factors
did not determine the dependent variable
(i.e., experimental research).

Becker & Gersten (1982): Effects of
Project Follow-Through

Quasi-experimental study

Ex post facto study

Problem: Are the two groups in this study
comparable to one another?
In order to make sure that the two groups
are comparable, and to ensure that the only
post-test differences between the groups
are due to the independent variable (the
Follow-Through intervention), data were
obtained on students:
family income
gender
language spoken in home
mothers education
ethnicity
number of siblings.

Research Design
FOLLOW-THROUGH
Year 1 (1975) Gr 5
Site 1
Site 2
Site 3
Site 4
Site 5
Year 2 (1976) Gr 6
Site 1
Site 2
Site 3
Site 4
Site 5
NO FOLLOW-THROUGH
Year 1 (1975) Gr 5
Site 1
Site 2
Site 3
Site 4
Site 5
Year 2 (1976) Gr 6
Site 1
Site 2
Site 3
Site 4
Site 5

Dependent variables
Wide-Range Achievement Test (WRAT)
reading
mathematics
Metropolitan Achievement Test (MAT)
reading
mathematics
RESULTS
A total of 180 comparisons of FT to No-FT
students. Of these, only 56 (31%) favored
FT students!
Largest differences between FT and No-
FT students were in basic skills areas.
FT students achievement declined by
grades 5 and 6 (2-3 years after end of FT).
Critique of this research
What are the strengths of the study?
Groups are comparable to one another.
Contrasted statistical with practical
significance.
Large sample size.
Multiple replications of treatment effect.
What are the weaknesses of the study?
Lack of random assignment.
Focus on standardized test performance.
You have to creatively design
your experiment!!!
48
49
What about Sampling???
1. Selecting subjects to participate in the
research
Careful sampling to ensure that results can be
generalized from sample to population
The relationship found might only exist in the
sample; need to ensure that it exists in the
population
Probability sampling techniques
50
Sampling
2. How the sample is divided into two or
more groups is important
to make the groups similar when they start off
randomization - equal chance
matching - similar to quota sampling
procedures
match the groups in terms of the most
relevant variables; e.g. age, sex, and race
Make sure that your measurement
is VALID & RELIABLE
51
NEXT WEEK:
VALIDITY & RELIABILITY
52