Experimental, Quasi-

Experimental, and Ex Post Facto

(Causal-Comparative) Research
 Characteristics of
Experimental Research
• There is a control or comparison group
• Subjects are randomly assigned to
groups
• The treatment is randomly assigned to
groups.
 Characteristics of Quasi-
Experimental Research
• There is a control or comparison group
• Intact groups are used
• The treatment is randomly assigned to
groups.
 Characteristics of Ex Post
Facto Research
• There is a control or comparison group
• Intact groups are used
• The treatment is not manipulated, it
has already occurred.
 Diagramming Research
• To illustrate research designs, a number
of symbols are used
– X1 = Treatment
– X2 = Control Group
– O = Observation (pretest or posttest)
– R = Random Assignment
 A Sample Research Design
• Single-Group Pretest-Treatment-
Posttest Design
R O X1 O This
Thismeans
meanssubjects
subjectsareare
randomly
randomlyassigned
assignedto toaa
group,
group,which
whichisisthen
thengiven
givenaa
pretest,
pretest,then
thenthere
thereisisaa
treatment,
treatment,then
thenthere
thereisisaa
posttest.
posttest.
 R O X1 O

• This is not really an experimental design because

there is no control group
– It is often referred to as a preexperimental design
• Novice researchers often use this research design
• There are some major problems with this design
– did the treatment really make the difference or
was something else happening.
 R O X1 O

• What are the threats to the Internal

Validity of this type of research (Did
the treatment really cause a
difference?)
 R O X1 O
Internal Validity Threats
• History
– Another event occurs during the time of the
experiment that might cause the difference
• An experiment to heighten racial awareness was
conducted by a researcher during February. This is
Black History month; so the results might be affected
by events that occur during Black History month and
not the treatment.
 R O X1 O
Internal Validity Threats
• Maturation
– People naturally change and evolve over time. This
may cause the difference.
• A college develops a new housing plan to promote more
open-mindness and acceptance of others. The students are
tested when they enter college and when they graduate. The
results show they are now more open-minded and tolerant of
others. Did the housing plan work or do students just mature
and grow as a result of the college experience.
 R O X1 O
Internal Validity Threats
• Mortality
– Some people drop out during an experiment. This
may affect the outcome.
• I am teaching a new experimental seminar on study skills.
About half of the class stopped coming to the seminar
before the semester was over. The students who remained
improved their study skills. So my course was effective!
– Probably not. The half that stopped coming might not have
gained anything; that is why they stopped attending.
 R O X1 O
Internal Validity Threats
• Testing
– Whenever you give a pretest, the students may remember
the test questions, and get them correct on the posttest.
• I gave a test to my study skills group on Monday, presented
some unique concepts on Tuesday, then gave them the posttest
on Wednesday. The grades were significantly higher on the
posttest.
– It is possible the grades were higher because the students still
remembered the questions from the pretest.
 R O X1 O
Internal Validity Threats
• Instrumentation
– To overcome the testing threat to internal validity, a
researcher develops a different form of the test
instrument, but it is not really equivalent.
• I gave a test to my study skills group on Monday, presented
some unique concepts on Tuesday, then gave them an
alternative form of the pretest on Wednesday. The grades were
significantly higher on the posttest.
– It is possible the grades were higher because the second test was
easier than the first.
 O X1 O
Internal Validity Threats
• Regression
– When subjects are selected because of extreme scores on
some type of instrument, there is tendency for their scores
to move more toward the average on subsequent tests.
• An experimenter selected students for a reading program based
on their low test scores. At the end of the treatment, the test
scores had improved.
– Extreme scores naturally move toward the mean on subsequent
tests.
 How to Handle Internal
Validity Threats
• Have a control group and use
randomization.This design is the Two-
Group Pretest-Treatment-Posttest
Design. The Control Group would experience the same
history and maturation. Mortality should be the

R
ROOX
X11 O
O
same because of random assignment. Random
assignment eliminates the selection threat.
However testing and instrumentation could still
R
ROOX
X22 O
O be a threat.
 Other Research Designs
• Two-Group Treatment-Posttest-Only
Design
There is no pretest so this
eliminates the testing and
R
R X
X11 O
O instrumentation threat to
internal validly but you
R
R X
X22 O
O don’t know about their
knowledge or attitude
coming into the study.
 Other Research Designs
• Solomon 4-Group Design
Note: A blank
indicates the
R
R O
OXX11 O
O control group,
same as X2
R
R X
X11 O
O
R
R O
O O
O
R
R O
O
 Quasi-Experimental Designs
• Posttest Only Nonequivalent Group
Design The absence of R indicates
there is no random
X
X11 O
O assignment. Sometimes you
will see a dotted line
X
X22 O
O between the two groups.
This indicates the two
groups may not be
equivalent.
 Quasi-Experimental Designs
• Pretest-Posttest Nonequivalent Group
Design

O
O X
X11 O
O
O
O X
X22 O
O
 Time Series Designs

O
OOOO
OXX11 O
OOOO
O

In the next course, AEE 579 Research Design,

many more research designs are examined.
 External Validity
• Can the research be generalized to
other settings?
– Population Validity
– Personological Variables
– Ecological Validity
 Population Validity
• Is the sample population similar to the
population the researchers wishes to
generalize to
 Personological Variables
• Different people have different
personalities, learning styles, etc., so
the results may not be generalizable to
people who are substantially different
on these personological variables.
 Ecological Validity
• The setting or situation in which the
experiment occurred may be different
than other settings.
 Social Interaction Validity
Threats
• Diffusion or Imitation of Treatment
– This occurs when a comparison group learns
about the program either directly or indirectly
from program group participants.
• This group may try to imitate or emulate what the
treatment group is getting.
 Social Interaction Validity
Threats
• Compensatory Rivalry
– The comparison group knows what
the program group is getting and
develops a competitive attitude with
them.
 Social Interaction Validity
Threats
• Resentful Demoralization
– This is almost the opposite of compensatory
rivalry. Here, students in the comparison
group know what the program group is
getting. But here, instead of developing a
rivalry, they get discouraged or angry and
they give up.
 Social Interaction Validity
Threats
• Compensatory Equalization of
Treatment
– The researcher is under pressure to
“enrich” the experiences of the control
group. This pressure may come from
parents, school administrators, etc.
 Ex Post Facto (Causal-
Comparative) Research
• Explores possible causes and effects
• The independent variable is not manipulated,
it has already been applied
• Focuses first on the effect, then attempts to
determine what caused the observed effect.
 Statistical Analysis
• If we are comparing the scores of two
groups – a t-test is normally used. The
value of t means nothing by itself
(unlike the value of R). We have to
determine if t is statistically significant

Tea for two

 Statistical Analysis
• If we are comparing the scores of three
(or more) groups – Analysis of
Variance (ANVOA) is used. This test
gives us a f value which means nothing
by itself. We have to determine if it is
statistically significant.
 Statistical Analysis
• If we want to statistically equate two or more
groups (because one group had a high pretest
score) we use Analysis of Covariance
(ANCOVA). This test gives us a f value
which means nothing by itself. We have to
determine if it is statistically significant.