See

discussions, stats, and author profiles for this publication at:

https://www.researchgate.net/publication/232569186

A Quasi-Experimental Approach to
Assessing Treatment Effects in
Nonequivalent Control Group
Designs

Article in Psychological Bulletin · May 1975

DOI: 10.1037/0033-2909.82.3.345

CITATIONS READS

123 2,913

1 author:

David A. Kenny
University of Connecticut
205 PUBLICATIONS 93,152 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Essentials of Interpersonal Perception (a revision of my 1994 book) View

project

Interpersonal Perception View project

All content following this page was uploaded by David A. Kenny on 20 May 2014.

The user has requested enhancement of the downloaded file.

Psychological Bulletin
1975, Vol. 82, No. 3, 345-362

A Quasi-Experimental Approach to Assessing Treatment Effects

in the Nonequivalent Control Group Design
David A. Kenny
Harvard University
Four statistical tests of treatment effect are evaluated for the nonequivalent con-
trol group design. This design consists of pre- and posttreatment measures of a
dependent variable with biased assignment to treatment groups. The biased assign-
ment creates a treatment-pretest confounding for which different statistical tech-
niques adjust. The different statistical tests discussed are the analysis of covariance,
analysis of covariance with reliability correction, raw change score analysis, and
standardized change score analysis. If assignment to treatment groups is based on
-j^ ,the pretest score (a very infrequent event), analysis of covariance is the appropriate
mode of analysis. Selection based on the pretest true scores necessitates a reliability
correction procedure. Selection based on stable group differences and selection that
-occurs midway between the pre- and posttest necessitates change score analysis.
Compensatory education programs can be made to look mistakenly harmful if they
are analyzed by an inappropriate statistical technique when they should have been
analyzed by standardized change score analysis. Given a model of the process of
^selection into treatment groups, the nonequivalent control group design can yield
Snterpretable results.

All quasi-experimental designs meet the of quasi-experimentation. It is the concern

following three requirements: (a) There must
be a treated and untreated group.1 (b) There
must be pretreatment and posttreatment
measures, (c) There must be an explicit model
that projects over time the difference between
the treated and untreated groups, given no
treatment effect. The third requirement is a
synthesis of the other two and is at the heart
This research was supported in part by the Spencer
and Milton Funds of Harvard University and National
Science Foundation Grant GS-30273X, Donald T.
Campbell, principal investigator. The paper grew out
of a joint effort with Donald T. Campbell to revise his
1971 paper on the same topic. His assistance was in-
valuable. Thanks are also due to Thomas Cook of
Northwestern University and Pierce Barker of the
Rand Corporation, who commented on an earlier draft
of the manuscript. The author accepts full responsi-
bility for any errors. Special thanks are due to both
Lenor Kirkeby and Ian Shrank, who were helpful in
the preparation of the manuscript.
Inquiries concerning this article should be sent to
David A. Kenny, Department of Psychology and
Social Relations, Harvard University, Cambridge,
Massachusetts 02138.
1
Designs 7, 8, and 9 in Campbell and Stanley (1963)
are times series designs and do not involve groups as
such. However, these designs do have a group of
pretreatment observations (control group) and post-
treatment observations (treated group). Since the
observations are not independent, standard t tests are
not valid.
345
with plausible, rival explanations of any hy-
pothesized treatment effect, or as Campbell
(1957) calls it, internal validity. In true
experiments, where by definition assignment
into treatment conditions is random, the rival
explanation of the treatment causing a mean
difference between the treatment and control
group is sampling error, and a significance test
is used to rule out that explanation. For most
quasi-experiments both sampling error and
biased selection into groups loom as plausible
rival hypotheses. Given nonrandom assign-
ment, the treated and untreated groups may
differ even in the absence of any treatment
effect. Time is brought into the design to
project the amount of this difference between
the treated and untreated groups. Certain
nonexperimental designs that do not include
time, like the cross-sectional survey, are not
amenable to a quasi-experimental analysis, but
the nonequivalent control group design is.
Weiss (1972) has suggested that the most
common design in evaluation research is a
pretest-posttest design where subjects are not
randomly assigned to groups, or as Campbell
and Stanley (1963) refer to it, the nonequiv-
alent control group design. The failure to
randomize may be due to the fact that:
(a) a treatment is administered to classroom,
346 DAVID A. KENNY
school, school system, and an untreated class-
room, school, or school system is taken as a
control group; (b) a true experiment is planned
but because of mortality, contamination of the
control persons by the experimentals, or varia-
tion in experimental treatment, the true
experiment has become a quasi-experiment;
(c) because of scarce resources the treatment
is only given to a select group; (d) subjects
self-select their own treatment level. Anyone
familiar with the problems encountered in
natural settings must realize that although
randomization and true experimentation are
ideal goals in research, they are not always
possible. At least a pretreatment measure
allows the researcher to assess how confounded
the experimental treatment is with the subjects'
predisposition to respond to the dependent
variable.
The magnitude of these differences between
treated and untreated groups can be assessed
in many different ways: mean difference, mean
difference relative to variance within groups,
or t ratio. Campbell (Note 1) has suggested an
alternative measure: the treatment-effect
correlation. The treatment variable is dummy
coded (1 for treatment and 0 for control)
and correlated with the pretest. A correlation
of a dichotomous variable with another vari-
able is sometimes called a point biserial corre-
lation and is itself a product moment correla-
tion. A correlation may seem to be an unusual
measure of difference between treated groups,
but it can be shown that both mean difference
and the t ratio can be easily transformed into r.
The formula for t to r is
=vs N - 2'
where N is the total number of subjects. The
formula_for the difference between treated
mean (Xi) and control mean (Xc) to r is
where s% is the variability of the observations,
P-s is the proportion of the total sample in the
treated group, and Pc is for the control group
(PT + PC = 1). The treatment-effect corre-
lation is, then, closely related to t and the
difference between means, but like all measures
of correlation it is a standardized measure of
the strength of a relationship. The use of
dummy codes for categorical independent
variables has been advocated by many authors
(e.g., Cohen, 1968).
Given these pretreatment differences on the
dependent variable, interpreting a difference
between the experimental and the control
group on the posttreatment measure becomes
problematic. To interpret a posttreatment
difference one must examine the magnitude
of the pretreatment difference, and that
difference or perhaps some adjusted difference
can be used as a baseline to judge the post-
treatment difference. At issue is whether
pretreatment differences should increase, de-
crease, or remain stable if there is no treatment
effect. One suggested mode of analysis has
been to match control and experimental subjects
on the pretest to make the pretest differences
vanish. Although this is a very popular
strategy, matching must be rejected out of
hand because of possible regression artifacts
(e.g., Campbell & Stanley, 1963, pp. 10-12;
Rulon, 1941) and because of increased Type II
errors (Chen, 1971).
The literature suggests four modes of sta-
tistical analysis to measure treatment effects
in the nonequivalent control group design:
(a) analysis of covariance, (b) analysis of
covariance with reliability correction, (c) raw
change score analysis, and (d) standardized
change score analysis. The differences between
these four techniques can be illustrated by
considering what each technique takes as the
dependent variable (see Table 1). In covariance
analysis (or, equivalently, multiple regression)
the posttest is regressed on the pretest and the
residual from the regression is taken to be the
dependent variable. Some authors (e.g., Lord,
1960; Porter, 1967) have suggested that the
estimated regression coefficient is attenuated
by measurement error in the pretest, so the
regression coefficient must be corrected for
attenuation. The appropriate correction is to
divide the regression coefficient of posttest
on pretest by the reliability of the pretest. A
third approach is raw change score analysis.
Although change score analysis.;is often con-
demned (e.g., Cronbach & Furby, 1970; Werts
& Linn, 1970a), it is perhaps the most common
way of analyzing this design. As the name
 NONEQVIVALENT
TABLE 1
NULL HYPOTHESES FOR THE FOUR MODES OF ANALYSIS OF THE NONEQUIVALENT CONTROL GROUP DESIGN
Statistical technique
Analysis of covariance
Covariance with reliability correction
Raw change score analysis
Standardized change score analysis
Note: 621 • T is the unstandardized partial regression of X 2 on Xi, controlling for T; * indicates standardization of variable; Xi is
the pretest, Xz the posttest, and T the treatment variable.
suggests, the dependent variable is the posttest
minus the pretest. The popularity of this mode
of analysis is no doubt due to its seeming ease
of interpretation and the fact that it can be
viewed as part of a repeated measures analysis
of variance. Raw change score analysis is
equivalent to the Time X Treatment inter-
action of a repeated measures analysis. It
should be noted here that using the change score
as the dependent variable and the pretest as a
covariate yields significant test results that are
identical to the analysis of covariance with a
metric of units of change (Werts & Linn,
1970a). A fourth mode of analysis is standard-
ized change score analysis. The pretest and
posttest are separately standardized (given
unit variance and zero mean) and then
differenced. (Note that this technique does not
standardize within treatment groups.) This
method is identical to raw change score
analysis with the exception that the variance
of the dependent variable is made stationary
over time.
There is for each mode of analysis a sta-
tistical expression that equals zero if there
were no treatment effect, that is, the null
hypothesis of each method. As stated earlier,
the orientation of this article is quasi-experi-
mental ; it seeks to determine the cases where a
statistical technique correctly infers no effect.
The differences among the four modes of
analysis can be brought into focus by express-
ing their null hypotheses as treatment-effect
correlations as in Table 1. Designations are
as follows: pretest = Xi, posttest = Xz, and
treatment variable = T.
All four null hypotheses can be viewed as
setting the post-test-treatment correlation
equal to the pretest-treatment correlation
times a correction factor. (See Appendix for
derivations.) For the analysis of covariance
CONTROL GROUP DESIGN 347
Dependent variable Null hypothesis
' A 2 — 021 * T-A- 1 PX2T = pXiTpXtX,
PXsT ~ PXiT&Xi/O'Xz
Y*2 — A
A V*i
PXST = PXjT
the correction factor is the pretest-posttest
correlation of X. Since the pretest-posttest
correlation is ordinarily less than one, analysis
of covariance implies that in the absence of
treatment effects, theposttest-treatment corre-
lation will be less in absolute value than the
pretest-treatment correlation. The correction
factor for the analysis of covariance with
reliability correction is the pretest-posttest
correlation of X divided by the reliability of
the pretest. This correction factor is ordinarily
less than one,2 making posttest-treatment less
in absolute value than the pretest-treatment
correlation. The correction factor for raw
change score analysis is the standard deviation
of the pretest divided by the standard devia-
tion of the posttest. Standardized change
score analysis has no correction factor, or more
precisely, has a correction factor of one: The
two treatment-effect correlations should be
equal.
Campbell and Erlebacher (1971) have
suggested a fifth method if the pretest and
posttest are "factorially similar": covariance
analysis with common factor coefficient correc-
tion. This method resembles analysis of co-
variance with reliability correction, the differ-
ence being that the regression coefficient is
divided by the pretest-posttest correlation of
X instead of the reliability of Xi. This correc-
tion yields the same null hypothesis as the
null hypothesis for standardized change score
analysis, since the pretest-posttest correlations
cancel each other out.
1
Given the standard formula for correction for
attenuation
it directly follows that
x^Xi < 1 if
 348 DAVID A. KENNY

It should be clear from Table 1 that the
null hypotheses of the four modes of analysis
will rarely all be equal to zero except in highly
trivial cases. Each of the four modes of analysis
has been advocated as the method of analysis
for the nonequivalent control group design by
various authors. Other authors (e.g., Lord,
1967) have pointed out that it is paradoxical
that different methods yield different con-
clusions. Cronbach and Furby (1970) state
that treatments cannot be compared for this
design. The literature on this design is, there-
fore, very confusing and not at all instructive
to the practitioner.
The validity of any mode of analysis
depends on its match with the process of
selection into groups. In the remaining part of
this article a general model of selection is
elaborated and various special cases of that
model are considered. Each mode of analysis
is appropriate for a given model of selection.
Though not advocated as the mode of analysis,
standardized change score analysis is empha-
sized. It has been previously discussed by
Woodrow (1939) and by Bereiter (1963) as a
test of treatment effects, but both authors
recommended some form of regression analysis.
Campbell (Campbell & Clayton, 1961; Camp-
bell, Note 1) revived the method but gave
only an intuitive justification. Standardized
change score analysis should be seriously con-
sidered as an alternative in the analysis of the
nonequivalent control group design.
THE GENERAL MODEL
There are three measured variables: X\ — a
pretest measure of the variable the treatment
is to change, X2 = a posttest measure, and
T = the treatment variable.
The causal function of X is divided into
three unmeasured, latent components: G
= group difference such as sex, race, classroom,
and others; Z = individual differences within
groups; and E = totally unstable causes of X
(errors of measurement). G must be included where U is a residual term that is uncorrelated
in the specification of causes of X because in with all unmeasured variables. U is simply a
field settings it can rarely be assumed that
sampling is done from a single homogeneous
population. Sampling is usually done from without altering many of the conclusions. I have
multiple populations, each of which has a chosen to make the variables uncorrelated to simplify
different mean level of X, presentation.
All variables (both measured and un-
measured) are standardized because standard-
ization decreases algebraic complexity and
because in this case it means little loss of
generality. For instance, the null hypothesis for
analysis of covariance can be stated in terms of
correlations. Raw change score analysis, how-
ever, necessitates the original metric.
The equations for Xi and X2 can be ex-
pressed in terms of their causes G, Z, and E:
Xu = aid + hZu + dEn (1)
Xu = a£!i + 62Z2i + e2E2i (2)
where the subscripts 1 and 2 refer to time and
subscript i to subject. Since no treatment
effects are assumed, T is not written into the
causal function of Xi or X2. The group differ-
ences variable (G) is assumed to be perfectly
stable, making its autocorrelation unity. This
explains why G needs no time subscript.
Relative position within groups (Z) may not
be perfectly stable, making its autocorrelation
less than one; the errors of measurement vari-
able (E) is perfectly unstable, making its
autocorrelation zero. All unmeasured variables
are assumed to be uncorrelated with each
other3 with the previously stated exception
that Z\ and Z2 may be correlated (pz^z^ =•])•
If the treatment is correlated with the
pretest, it must be confounded with the causes
of the pretest. Thus, the treatment must be
correlated with either group differences (G),
relative position within groups (Z), errors of
measurement (E), or any combination of the
three. So in writing the causal function for
the treatment variable, G, Z, and E must be
included. At the beginning the assumption is
made that the occasion of selection of the
persons into treatment occurs at the pretest,
thus making T confounded with Z and E at
Time 1. (This assumption will later change.)
The causal function of the treatment variable
is thus
Ti = qd + mZu + sEu + fU<, (3)
3
The latent variables may be allowed to be correlated
 NONEQUIVALENT CONTROL GROUP DESIGN
variable that represents all other causes of
selection besides G, Z, and E.4
For readers who are aided by path diagrams
Equations 1, 2, and 3 are expressed in the path
diagram in Figure 1. A path diagram not only
pictorially represents a set of equations but
also is a heuristic to aid in the derivation of
correlations (Werts & Linn, 1970b). Using
either the path diagram or the algebra of
expectations the correlations between the
measured variables are
= qa>i + mbi + (4)
(5)
— a-ia-2 + (6)
The remaining part of this article will
consider various special cases of this general
model of selection that yield overidentifying
restrictions, that is, an algebraic constraint
on the correlations generated by the model.
Then it will determine if the overidentifying
restrictions derived from the model of selection
match the null hypothesis of any of the four
modes of analysis given in Table 1.
Three different types of selection processes
will be considered. The first type is selection
based on the measured pretest. This is the
only type of selection considered that is con-
trolled by the researcher. In this case the
pretest is correlated either by design or by acci-
dent with the treatment and therefore corre-
lated with all the causes of the pretest: G,
Z, and E. For this type of selection the analysis
of covariance is appropriate. The second type
of selection is selection based on the true
pretest. For this case subjects select treatments,
and the selection process is related to G and Z
but not -E; analysis of covariance with relia-
bility correction is appropriate. The third type
is selection based on group differences. Subjects
are assigned to the treatment because of
demographic and social variables, which are
confounded with the pretest. If the effect of
these demographic variables is stationary over
time, standardized change score analysis is
the appropriate mode of analysis. Last dis-
cussed is the case in which the occasion of
4
Since T is measured dichotomously it may seem
as if U must be correlated with G, Zi, and E\. But like
any residual of a linear model, U can be constructed
to be uncorrelated with G, Z\, and £1.
349
FIGURE 1. Path model of selection at the pretest.
(Xi is pretest, X% is posttest, T is treatment, G is group
differences, E\ and Ei are errors of measurement, Z\
and Z% are individual differences, and a\, <z2, 61; b%, elt e2,
s, q, m, f, and j are path coefficients.)
selection into treatment groups is not at the
pretest.
Selection Based on the Pretest
At times selection into treatment groups can
be controlled, but the experimenter decides not
to assign subjects randomly to treatment condi-
tions. The reason for not assigning randomly is
that it may not be fair for all types of subjects
to have the same probability of receiving the
treatment; that is, certain persons (e.g., the
injured or disadvantaged) are considered more
deserving of the treatment. One strategy,
called the regression discontinuity design by
Thistlethwaite and Campbell (1969), is to
assign subjects to the treatment on the basis
of the pretest. Persons scoring above a certain
point would be given the treatment and those
scoring below or equal to that point would be
the controls. (Since the pretest is measured
on an interval scale and treatment is dicho-
tomous, pXlT will not equal 1 but it will be
high.) Actually a pretest itself need not be
used; any measure of "deservingness" will do.
The treatment, then, is a function of the pretest
(Xi) and random error (U):
Ti = kXu + fUi. (7)
In this case the treatment is deliberately
correlated with the pretest and therefore with
the causes of the pretest, whereas in the true
350 DAVID A. KENNY
0--
-2-
PRETEST FOSTTEST
• EXPERIMENTALS
X CONTROLS
TREND FOR CHANGE SCORE ANALYSIS
TREND FOR ANALYSIS OF COVARIANCE
FIGURE 2. Pretest and posttest means for a random-
ized experiment with zero mean and within-group
regression of .5.
experimental case the treatment is deliberately
uncorrelated with the pretest through random-
ization. But, .as in the true experimental case,
an assumption or specification has been gained
by controlling the assignment to groups. In
the population the treatment should correlate
with the unmeasured causes of pretest (G, Z\,
and Ei) to the degree to which they cause the
pretest. If Equation 1 is substituted for ; ^i
within Equation 7 the result is Ti = k(a\Gi
+ biZu + eiEu) + fUi. But since T also
equals Equation 3, the following must be true:
ajk = q, bik = m, and eik = s, or alternatively
q/ai = m/bi = s/e-i = k. Using either of these
two equations and Equation 4, and remember-
ing that ai2 + &i2 + ei2 = 1 (since Xi is
standardized), it can be shown that pxlT = k,
and with Equations 5 and 6 it can be shown
that pXiT = kpXlXf Solving both of the
above equations for k, it follows that px^r
= pXjTpx^y- which is the null hypothesis for
analysis of covariance in Table 1. Thus, when
subjects are assigned to the treatment on the
basis of the pretest, analysis of covariance is
the appropriate mode of analysis.6
Special attention should be paid to the
validity of covariance's assumptions of homoge-
neity and linearity of regression. Lack of
5 individual differences, whereas the controls
1 claim no originality for this proof. For a related
proof see Goldberger (Note 7), and for simulations see
Sween (1971).
linearity can usually be remedied by including
higher order terms such as the pretest squared.
If the treatment interacts with the pretest,
this would violate the assumption of homoge-
neity of regression.
Unhappy randomization. It seems to be an
all too common occurrence that randomization
of subjects into treatment groups produces
treatment differences even before the treatment
is^'administered. Although it is clear that the
probability of such pretest differences is 1 out
of 20, given the conventional .05 level of
significance, the experimenter still has the
feeling of being persecuted by fate. It seems
that the experiment has been doomed and
there is no way to achieve valid inference.
I should be clear about what I do not mean
by unhappy randomization. I do not mean
cases of failure to randomize or cases where
there is randomization but a selected group of
control and experimental subjects fail to
provide posttest data. I am speaking of a
randomized experiment with a pretest differ-
ence between the experimentals and controls.
Randomization has not failed, as is sometimes
mistakenly thought; it is only that an un-
likely type of event has occurred.
Valid inference is possible in a way similar
to that in the discussion of the previous section.
If there is a pretest difference, the treatment
is confounded with the causes of the pretest.
The expected degree of this confounding with
each cause is proportional to its causal effect
on the pretest.
For a randomized experiment with the pretest
correlated with the treatment, the analysis of
covariance is not only appropriate but,neces-
sary. Consider the illustration in Figure 2.
It has been assumed that the experimentals
scored four units higher than the controls on
the pretest even though subjects were ran-
domly assigned to groups. Assuming that the
pooled within-regression coefficient is .5 (the
grand mean is zero at Times 1 and 2), the
expected difference at the posttest is two units.
This regression toward the mean is expected
because at the pretest the experimentals
should have positive errors and positive
should have negative errors and individual
differences. Over time this gap should narrow
 NONEQVIVALENT CONTROL GROUP DESIGN
and both groups regress toward the grand
mean.
The dashed line in Figure 2 indicates the
expectation of group differences at the posttest
given no regression toward the mean. This
would be the baseline expected by change score
analysis or equivalently by the lack of a
Treatment X Time interaction in a repeated
measures analysis of variance. Change score
analysis fails to take into account the regression
toward the mean that should be expected.
Covariance analysis is also generally more
powerful than change score analysis.
The control of assignment to groups either
by randomization or assignment based on
some other measured criterion necessitates
analysis of covariance. Once again, special care
should be taken to meet the assumptions of
analysis of covariance, especially those of
linearity and homogeneity of regression.
Selection Based on the Pretest True Score
The case of interest here is not the one in
which the researcher controls selection into
treatment groups but the one in which this is
not the case. Two types of subject selection
will be discussed: first, selection based on the
pretest true score (G and Z) and second, selec-
tion based only on group differences (6).
The model previously discussed assumes that
the treatment is correlated with the errors of
measurement in the pretest. This is generally
implausible when selection is uncontrolled. If
persons select themselves into programs,
selection is more likely to reflect their true
ability and not their chance performance on the
pretest. To repeat, the subjects, not the
experimenter, control selection, making the
treatment correlated with only the true causes
of the pretest. (The true scores are not actually
measured.) If errors of measurement do not
cause selection, the result is the same model as
in Figure 1 except that s = 0. As earlier, it is
assumed that the ratio of the effect of group
differences on the treatment to their effect on
the pretest equals the ratio of the effect of
individual differences on the treatment to their
effect on the pretest, that is:
(8)
This assumption presupposes that selection
351
into treatment groups is made on the basis of
the true pretest, aiG + biZi, and not on errors
of measurement or any other function of G
and Z. In the case of selection based on the
measured pretest a similar hypothesis was
made, but that hypothesis was justified by the
design of the research. In this case Equation 8
is an assumption that must be justified by
evidence from the selection process itself.
If s = 0 and Equation 8 holds, Correlations
4 and 5 become
(9)
The reliability of the pretest px^ is defined
as oi2 + bf, making PX:T = kpxlXl, and given
Equation 6, px^r — kpx^x^- The analysis of
covariance null hypothesis will not equal zero
since
Except in trivial cases, the above will equal
zero only if the reliability of the pretest is
perfect.
The reason for this bias is that the within-
groups regression coefficient is attenuated
because the pretest is measured with error.
The posttest should be regressed not on the
measured pretest as in covariance analysis but
on the true pretest. It is because of this bias
that both Lord (1960) and Porter (1967) have
suggested a correction for the analysis of
covariance. This correction can be thought of
as correcting the regression coefficient for
unreliability in the pretest. To make the
reliability correction there must be an estimate
of the reliability of the pretest : a? + hi2.
Assuming that there is an estimate of relia-
bility and 5 = 0 and Equation 8 holds, the
formula in Table 1 for the analysis of covariance
with reliability correction equals zero if there
are no treatment effects.
The difficulty with the reliability correction
procedure is the necessity of having a relia-
bility estimate. The inclusion of any ad hoc
estimate, for example, internal consistency,
into the correction formula almost certainly
increases the standard error of the estimate.
Ideally the reliability estimation procedure
should be part of a general model. Following
352 DAY ID,A. KENNY

Lord (1960), consider, for example, a parallel 3. To receive a selective treatment a person
measure of Xi, say F. Let or that person's sponsor must be highly
motivated or have political connections and
Yi = a3G; + btZii + e3Esi, organizational "savvy." These volunteers differ
where £3 is uncorrelated with all other un- systematically from nonvolunteers on a num-
measured variables. If we assume that Equa- ber of characteristics (Rosenthal & Rosnow,
tion 8 holds and that s = Q, then it follows that 1969).
4. The treatment either is a sociological or
PXlY — WJ.W3 , viva demographic variable or hopelessly confounded
with one. Examples of this are a study on the
effects of dropping out of high school or testing
PTY = k(aias+ 6163). (11) for differences in socialization between males
Given Equations 9, 10, and 11, it follows that and females. The reader can probably also
conceive other patterns of sociological selec-
tion. Suffice it to say that it is a rather common
form of selection into social programs.
Substituting the above formula for reliability In terms of the general model, selection
into the reliability correction formula in Table based on group differences implies that
1 yields the following null hypothesis : j = m = 0. To gain an overidentifying restric-
PXfl =
tion one must assume some form of station-
arity, that is, that the effect of group differences
or equivalently in vanishing tetrad 6 form : is the same at the pre- and posttest. Campbell
(Note 2) has argued for just such a model
= 0.
with what he called the fan spread hypothesis.
The vanishing tetrad can be tested by the null The fan spread hypothesis can be illustrated
hypothesis of a zero second canonical correla- pictorially. In Figure 3 two groups start out
tion between variables Xi and T and variables at Time 1 with divergent means. Campbell
X, and F (Kenny, 1974). (1969) suggests that associated with this mean
difference is a difference in maturation; those
Selection Based on Croup Differences with the higher mean mature at a greater
rate than those with the lower mean. Campbell
For most social programs assignment to the calls this the "interaction of selection and
treatment is not based on some psychological maturation" and has used this interaction as
individual difference, that is, true score, but an argument against raw change score analysis.
on some sociological, demographic, or social Since the mean difference between groups is
psychological characteristic. This sociological widening over time, change score analysis only
selection is brought about in a variety of ways : indicates the more rapid rate of maturation of
1. It may be a matter of policy or legislation the initially higher group. The fan spread
that treatment is available to a particular hypothesis is that increasing variability within
social group, for example, persons living in groups accompanies increasing mean differ-
particular census tracts. It may be virtually ences. In its strictest form the fan spread
impossible to find members of that social group hypothesis is that the difference between group
who did not receive the treatment. means relative to the pooled standard deviation
2. Treatments are administered to members within groups is constant over time. In Figure 3
of an entire organization (e.g., school system), the ratio of mean difference to standard
and members of the treated organization must deviation is always 4:1.
be compared with another organization. The rationale for the fan spread hypothesis
is that the different groups are members ©f
6
A vanishing tetrad is a general null hypothesis in different populations living in different en-
factor analysis of the form vironments. The different environments create
P12P34 — PlSPU — 0, and maintain different levels of performance
where 1, 2, 3, and 4 are variables. and different rates of growth. Given that
 NONEQVIVALENT CONTROL GROUP DESIGN
growth' is a cumulative process, variability
increases over time. The groups would eventu-
ally approach asymptote at different levels.
The fan spread hypothesis is only a hy-
pothesis; it must be empirically validated. I
have found it to be a very workable hypothesis.
Generally racial and social class differences on
cognitive tests show fan spread, that is, the
correlations of these variables are stationary
over time. Raw change score analysis predicts
that co-variances are stationary. A powerful
illustration of fan spread is in Equality of
Educational Opportunity (Coleman et al., 1966),
in which the gap between blacks and whites as
measured by the correlation of race (dummy
coded) with verbal achievement is relatively
stationary over time, while the difference in
grade equivalents increases. A thorough review
of various literatures is needed to test the
adequacy of the fan spread model. Naively,
one would expect fan spread to be more likely
as (a) the time interval between pre- and
posttest becomes shorter, (b) the tests used
are more similar, and (c) the amount of growth
becomes smaller. Formally, the model for
selection on the basis of group differences is
TI = qd + fU(. Fan spread implies that
ffi = a2, making px-^r = PX%T (equality of the
treatment-effect correlations).
If the variance of the pretest and posttest is
stationary over time, then the results of
standardized change score analysis and raw
change score analysis converge. Thus, for cases
where variance is stationary, raw change score
analysis is valid. Standardization is needed
only to stabilize variance over time. Ideally a
priori power transformations can be found to
stabilize variance in a way akin to meeting
the homogeneity of variance assumptions of
analysis of variance.
The discussion has narrowly focused on the
univariate problem of attributing treatment
effects to change. The emphasis on correlations
is not meant to discourage control of back-
ground variables by multiple regression. These
background measures can be used both to
increase power and to reduce pretest-treat-
ment confounding. The choice of background
variables should be guided by theory and
experience, not solely by the desire to explain
the pretest-treatment correlation. Merely to
reduce the pretest confounding is "capitalizing
353
GRADE
FIGURE 3. Fan spread model with linear growth and
(J!"i - Xz)/s equal to 4.
on chance" in a way analogous to the shrinkage
in multiple correlations (cf. Lord & Novick,
1968, pp. 285-288). Rather the researcher
should a priori choose background variables
and partial them separately out of the pretest,
posttest, and treatment, adjusting degrees of
freedom accordingly (0'Conner, Note 3).
Rarely does this control procedure reduce the
pretest-treatment correlation to zero.
With these background variables, I suggest
comparing partial correlations of the treatment
with the pretest and the posttest, with the
background variables partialed out. The partial
correlations are compared instead of partial
regression coefficients because of the assump-
tion of stability of variance over time; that is,
the variability of the dependent variable that
is not explained by the background variables is
stable over time.
As an illustration, consider the correlations
in Table 2. The data are taken from Educa-
tional Testing Service's growth study (Hilton
& Myers, 1967). Of interest here is the effect
of social class on knowledge of industrial arts.
For a sample of 2,164 the correlation of social
class with industrial arts is positive at both
Grades 5 and 11. There is, however, a signifi-
cant decrease in correlation over time. Stan-
dardized partial regression weights reveal the
same effect. Accompanying the decrease in the
correlation of social class with industrial arts
is a dramatic increase in sex differences favoring
354 DAVID A. KENNY
TABLE 2
RELATIONSHIPS BETWEEN SEX AND SOCIAL CLASS WITH
KNOWLEDGE OE INDUSTRIAL ARTS AT STH
AND HTH GRADE
Variable Grade 5 Grade 11
Raw correlation coefficients
Sex .239 .508
Social class .240 .191
Partial regression coefficients
Sex .235 .508
Social class .236 .182
Partial correlation coefficients
Sex .242 .514
Social class .242 .211
Note. Males = 1; females = 0. N = 2,164.
males. A fairer test of the stability of the effect
social class over time would be to examine its
correlation with industrial arts, with sex
partialed out. As can be seen in Table 2, the
partial correlation of social class with in-
dustrial arts decreases less over time; the
decrease is statistically nonsignificant. The
decrease in correlation of social class with
industrial arts over time is thus an artifact
of increasing correlation of sex with industrial
arts. This artifact is not due to the correlation
between sex and social class (it is only .018)
but to a reduction in variability of industrial
arts at Grade 11 that is correctable with social
class given the increased effect of sex.
To summarize, if group differences are the
sources of confounding of the treatment with
the dependent variable, and if the effects of
these group differences are stationary over
time, then some form of change score analysis
should be used. If the variance of the dependent
variable is stationary, then original metric
should be used. If not, then some variance
equalizing transformation should be made.
One possible transformation is standardization.
Finally, background variables should be con-
trolled for by using partial correlations for the
standardized case and multiple regression
analysis for the raw metric. The advocacy of
regression analysis in this case is to control for
the background variables but not the pretest.
Selection Not at the Pretest
I have assumed that the occasion for selec-
tion into treatment is at the pretest. For most
programs the occasion for selection is not so
well denned. Treated and control subjects may
drop out of the program or move out of the
area, some controls may enter the program,
unsuccessful or unhappy treated subjects may
not show up at the posttest, and so on. (For
a review of some of the difficulties of longi-
tudinal studies see Crider, Willits, & Bealer,
1973.) Sometimes the "pretest" takes place
well before the treatment begins; for example,
test scores of the previous year are used as a
pretest for a remedial program for the current
year. For many real-world programs the
occasion of selection into the program is not
identical with the pretest.
This implies that if assignment is based
on the true score, the correlation of the treat-
ment with the effect variable is not the highest
at the pretest. This would substantially bias
analysis of covariance with reliability correc-
tion. Imagine not measuring X\ but using X$
as a "pretest." For such a case the analysis
of covariance with reliability correction would
be biased if parameters a, b, and j remained
stationary over time. The treatment-effect
correlations, however, are equal to each other,
given the assumption of stationarity. Thus, if
selection occurs midway between the pretest
and the posttest, or "averages out" midway,
and if stationarity can be assumed, then
standardized change score analysis is the ap-
propriate form of analysis. If selection is based
on group differences, the occasion of selection
is irrelevant since group differences are
perfectly stable.
Measurement of the Treatment and. the Effect
Special care must be taken in the measure-
ment of the treatment so that the above
assumptions are not violated. A troublesome
error is retrospective measurement of the
treatment at the posttest (Campbell &
Clayton, 1961). This practice usually insures
that errors of measurement are correlated with
the treatment and that individual differences
are more highly correlated with the treatment
at the posttest than at the pretest. To insure
that errors of measurement and other ir-
 NONEQUIVALENT CONTROL GROUP DESIGN
relevant factors are uncorrelated with the
treatment, the method of the measurement of
the treatment should not be self-report but
should be independent of the subject, nor
should measurement occur at the posttest.
Special effort should also be taken to insure
that the measurement of subjects is similar
both at the pretest and the posttest and for
both the treated and untreated subjects. For
example, the motivational states and testing
situations of both groups should be identical.
Ideally the testing of both groups should be
in the same setting, one not connected with
the program.
Plausible Rival Hypotheses of Changes in the
Treatment-Effect Correlations
The comparison of the simple correlations
or partial correlations over time has been
suggested as a test of treatment effects. There
are many plausible rival explanations of a
difference between correlations that do not
involve treatment effect. Changes in distri-
butions of a variable affect the size of corre-
lations. Floor effects at the pretest or ceiling
effects at the posttest loom as plausible rival
hypotheses. The distributions of the pre- and
posttest should be examined and compared.
To compare the pretest and posttest, the
tests must be structurally similar. They need
not be parallel in the strictest sense since it is
ordinarily expected that the mean and variance
of the test will change over time, but the tests
should have the same structural or causal
equation at both points in time. (Identical
structural equations is what was meant by
stationarity in a previous section of this article.)
Conceivably, the structural equations of the
true scores could be invariant over time but
error variance could vary over time. This
would create a strong plausible rival explana-
tion of a change in treatment-effect corre-
lation: a shift in the reliability of a measure
over time. For young children the reliability
of a test usually increases over time; failure
to take this fact into account leads to a
spurious increase in the treatment-effect
correlation, the increase being due to an
increase in reliability and not a treatment
effect. Special care must be taken to insure
that the reliability of the test is stationary
over time. If the reliabilities are not equal,
355
the treatment-effect correlations should be
corrected for attenuation.
The reliabilities themselves need not be
directly known, only the ratio of the relia-
bilities. If the null hypothesis is that the
treatment-effect correlations corrected for
attenuation are equal, that is,
PXjT _
and if we know k, which is defined as the
reliability ratio px^xjpx^^ it then directly
follows that
Kenny (1973) has shown that the reliability
ratio can be estimated given the assumption
of structural similarity of true scores over
time and given three or more repeatedly
measured dependent variables. Estimates of
reliability ratios are obtained by factoring a
matrix of ratios of synchronous correlations.
The resulting "factor loadings" are the relia-
bility ratios, or strictly speaking, communality
ratios. A difficulty remains because it is not
immediately clear whether (a) the raw corre-
lations or (b) the correlations with the treat-
ment variable partialed out should be used
to estimate reliability ratios. If the raw
correlations are used, the assumption of
stationarity is violated when the treatment
has an effect, but if the partials are used, the
reliability ratio may be biased since the
variance shared with the treatment is ignored.
This difficulty can be avoided by obtaining
the reliability ratios from the partials and
then adjusting them for the increased relia-
bility generated by the treatment variable
(Kenny, Note 4).
Significance Testing
The equality of the treatment-effect corre-
lations can be tested by a test credited to
Hotelling and reproduced by McNemar (1969,
p. 158). Dunn and Clark (1969, 1971) have
pointed out that Hotelling's test is a large-
sample test and have evaluated a number of
additional tests. Possibly a more powerful test
can be developed that takes into account
wi thin-group regression but allows for no
between-group regression.
356 DAVID A. KENNY
Between- and Within-Group Regression
The logic of covariance analysis is that it
uses the within-group regression of individuals
toward the group mean to estimate the be-
tween-group regression of the group mean
toward the grand mean. Implicit in this logic
is that both groups of subjects belong to the
same population and that the forces that effect
changes in individuals also effect changes in
groups (Bock, Note 5). However, if selection
into treatment groups is only based on group
membership and the groups differ equally on
the pretest and posttest, then there will be
no between-group regression.
Empirically, it is not an easy question to
decide whether there will be any between-
group regression. As a rule, however, the treat-
ment and control groups live in different
social, cognitive, and genetic environments,
which tends to maintain rather than diminish
any pretest differences. Ordinarily the re-
searcher should examine whether variables
known to cause .the dependent variable
partially explain the pretest difference. For
educational data, for instance, if the lower
scoring group is of lower socioeconomic status
than the higher scoring group, then there is
evidence consistent with the different-popula-
tion-variable hypothesis. Ideally these popu-
lation differences could be measured and
controlled. Unfortunately, it is usually im-
possible to specify and perfectly measure these
variables. Even if they were measured, the
treatment variable is often so hopelessly
confounded with them that it would be
conceptually impossible to distinguish the treat-
ment variable and the background variable.
In the case of selection occurring midway
between the pretest and posttest there may
be between-group regression, but the regression
occurs both backward and forward in time.
The treated and control groups are maximally
different at the point of selection. Over time
these differences narrow, and if the rate of
regression is stationary, the differences between
groups should be equal at the pretest and
posttest. So even given between-group regres-
sion, pretest and posttest differences could
still be equal.
The Evaluation of Compensatory Education
Programs
Compensatory education programs are those
in which the treated subjects start out behind
the controls in some cognitive ability. I make
the simplifying assumption that the relevant
comparison group is the control group. Using
the treatment-effect correlation as a measure
of pretest differences, the pretest-treatment
correlation starts out negative for compensa-
tory evaluation. A totally successful compen-
satory program should change the negative
pretest-treatment correlation to a zero treat-
ment-posttest correlation. A negative posttest-
treatment correlation indicates a "semi-
compensatory" program, and a positive
correlation indicates an "over-compensatory"
program. A harmful program would give the
treatment-posttest correlation a higher nega-
tive value than its baseline.
Campbell and Erlebacher (1970) suggest and
to some degree illustrate that compensatory
education has been unfairly evaluated. They
claim that there is a bias in the evaluation of
compensatory education that makes it look
harmful. This methodological bias can wash
out the beneficial treatment effects of a
successful program.
I agree with "Campbell and Erlebacher and
believe that many compensatory programs
should have been evaluated by the standard-
ized change score analysis. Since the control
group is often drawn from a radically different
environment than the treated group, selection
is often based on intact group differences.
Let us examine in a hypothetical example the
direction of bias obtained by using the wrong
statistical technique when standardized change
score analysis should have been used.
Selection is based on group differences :
Xn = .SGi + .TZu + Vi26£lf
T{ = - .4G<
where pZlzt = .6. Given the above equations,
the intercorrelations are
PX^T = — .2
:
PX^T = — .2
l>xx = -544.
 NONEQUIVALENT CONTROL GROUP DESIGN
Note that the treatment-effect correlations
are negative, indicating a compensatory
program. For this example covariance analysis
yields a negative estimate of treatment effect
since
< PXPXX
-.2 < -.1088.
It is negative bias since covariance analysis
predicts a smaller difference between the
experimental and control group at the post-
test. Covariance analysis underadjusts in this
case. The reliability of the pretest (.52 + .72)
is .74, making the baseline for covariance
analysis with reliability correction
PXST <
-.2< (-.2)(.544/.74)
-.2 < -.147.
So covariance analysis either with or without
reliability correction tends to falsely indicate
a harmful effect of the compensatory treat-
ment. Using the reliability correction only
partially adjusts for the bias of covariance
analysis since —.147 is less than —.1088.
Raw change score analysis also tends to be
biased given the earlier mentioned fan spread
hypothesis: It assumes that for growth data
there should be increasing variability over
time; for educational data, the common
finding is that the standard deviation increases
over time. Assuming the standard deviation
of Xi is 1 and the standard deviation of X2 is
2, the treatment effect estimate is also nega-
tive since
PX-tT < PXiTVxJvXz
-.2 < -.2/2
-.2< -.1.
The above illustration shows that analysis
of covariance, analysis of covariance with
reliability correction, and raw change score
analysis all tend to be biased toward finding
negative effects of a compensatory treatment.
Covariance analysis and raw change score
analysis, though usually viewed as competitors
for this type of data, both yield the same
biased conclusion. The dice are loaded against
finding beneficial effects of compensatory
357
education programs. Given this bias, it is
premature to conclude that "compensatory
education has been tried and it apparently has
failed" (Jensen, 1969).
For an empirical illustration of this bias,
consider data taken from Crano, Kenny, and
Campbell (1972). A total of 5,495 Milwaukee
children were measured on 11 achievement
tests (Lindquist & Hieronymus, 1964) at the
fourth and sixth grades. The children can be
divided into core or inner-city children and
suburban children.
Let us imagine that the core-suburban
variable is perfectly confounded with some
hypothetical treatment: All core children
received the treatment and none of the
surburban children received it. What would
analysis of covariance, raw change score
analysis, and standardized change score an-
alysis reveal about this pseudotreatment?
Table 3 gives the actual and predicted posttest-
treatment correlations of three different
methods.
Working through an example, one sees that
rx%T for vocabulary is —.3485. The negative
sign indicates the core children scored lower
than the suburban children at the posttest.
The pretest-treatment correlation is —.3412,
indicating that it is a good baseline for the
posttest-treatment correlation. Remember that
a nonexistent treatment is being evaluated, so
the predicted YX^T should equal the actual
rxfl. Analysis of covariance's baseline does
not fare so well; the autocorrelation of
vocabulary is .752, making the predicted rx^r
(.752)(—.3412) = —.2566. Covariance thus
predicts that the difference between the core
and suburban children should attenuate over
time. It presumes that pretest differences
diminish since persons belong to the same
population and group means should regress
to a common mean. In this case, however, there
is regression toward the mean within groups,
but the group means do not regress to a
common mean. There is within-group regres-
sion but no between-group regression.
The standard deviation of the vocabulary
test increases by a factor of 1.53, making the
predicted posttest-treatment correlation of
raw change score analysis —.2225. Change
score analysis, like analysis of covariance,
forecasts a smaller posttest difference between
 358 DAVID A. KENNY

TABLE 3
COMPARISON OP THREE METHODS IN PREDICTING THE TREATMENT-POSTTEST CORRELATION FOR MILWAUKEE DATA

Predicted rx&

Test Actual rx2T

Standardized change Analysis of Raw change
score analysis covariance score analysis
rxiT rx^xjZj rxiTSXi/sxi

Vocabulary -.3485 -.3412 -.2566 -.2225

Reading comprehension -.3293 -.3367 -.2434 -.2793
Spelling -.2085 -.2065 -.1514 -.1761
Capitalization -.3040 -.2882 -.1859 -.1594
Punctuation -.2771 -.2514 -.1412 -.1344
Verbal usage -.3004 -.3174 -.2244 -.1952
Map reading -.3512 -.2827 -.1444 -.2013
Graphs -.2718 -.3170 -.1680 -.2415
References -.3590 -.2478 -.1509 -.1900
Arithmetic concept -.3395 -.3447 -.2081 -.3664
Arithmetic problem solving -.3011 -.2637 -.1408 -.1982
M -.3082 -.2906 -.1832 -.2149

Note. N = 5, 495.

the core and suburban children than actually
exists. It too leads to the erroneous conclusion
that the nonexistent treatment is harmful.
For all 11 variables covariance analysis
yields large negative effects of a nonexistent
treatment. The average difference of the actual
posttest-treatment correlation from the pre-
dicted correlation for analysis of covariance
is —.1250. Raw change score analysis also
shows large negative effects but with not
quite the consistency of covariance. Its
average difference is —.0933. Only the stan-
dardized change score analysis generally shows
no difference. The slight difference between
treatment-effect correlations of —.0176 can
be explained by the probable increased relia-
bility of the Time 2 tests. Since there are no
reliability estimates for this data, covariance
analysis with reliability correction was not
performed.
It might be argued that both covariance and
raw change score analysis yield the correct
conclusion because they reveal an increased
cognitive deficit for the' inner-city children. A
counterargument is that given the non-
equivalent control group design, the experi-
mental and control groups normally live in
different worlds and therefore receive different
treatments. It is thus more sensible to expect
that the different environments of the two
groups preserve the pretest differences than
to be surprised that pretest differences do not
diminish.
For noncompensatory treatments, bias works
in the opposite direction. Treatments given to
those who need them least are made to look
mistakenly beneficial (Campbell, Note 1). This
can be seen in the above two examples. If we
change the signs of the treatment-effect corre-
lations, the inequalities (and therefore direc-
tion of bias) reverse.
CONCLUSION
The example presented in Table 3 clearly
illustrates the utility of standardized change
score analysis. The criticisms of change score
analysis by Cronbach and Furby (1970) and
Werts and Linn (1970a) are widely cited, but
these articles presume a model that implies
either covariance analysis or more generally
multiple regression analysis or covariance
analysis with reliability correction. I have
argued that these types of models are not
valid if selection is based on group differences
or if selection occurs midway between the
pre- and posttest. I hope that my analysis
persuades researchers not to automatically
use some form of regression analysis for the
nonequivalent control group design. I also
hope that my reiteration of Campbell's point
about the selection-maturation interaction
 NONEQV1VALENT CONTROL GROUP DESIGN
implied by the fan spread model persuades
researchers not to automatically use raw
change score analysis,
The main purpose of this article is not to
advocate standardized change score analysis
but to advocate that the researcher carefully
study the process of selection into treatments.
The critical questions are (a) With what and
by how much does the treatment correlate
with the causes of the dependent variable?
(b) When does selection take place? The
answer to these questions determines the mode
of analysis. The standard practice in which
the mode of analysis determines the assump-
tions about the data must be avoided. The
above questions, though they have statistical
and mathematical implications, are not sta-
tistical questions that can be answered by an
examination of data. Their answers require
that the researcher be in close contact with
the treatment program. The researcher should
not take for granted statements by admini-
strators about the process of selection but
rather should use the same observational skills
as those an anthropologist or an ethnomethod-
ologist uses to study the process of selection.
It may often be unclear which mode of
analysis is appropriate, or it may be clear that
no mode of analysis is appropriate. In such
cases one might perform multiple modes of
analysis, each with a different bias (Cook,
1974) and check to see if the results converge.
If they do, one can be confident of a result;
if they do not, the correct inference is in doubt.
I do believe that the nonequivalent control
group design can be analyzed. I do not agree
with Cronbach and Furby (1970), who state
for this design: "What cannot be done is to
compare 'treatment effects'" (p. 78). My
position is that the design can be analyzed
conditional on a given structural model.7 The
structural model must be justified by the
setting, population, measurement, selection,
and design of the research. There will usually
be some equivocalities in the analysis, but
all inference, be it true experimental or quasi-
experimental, is risky business. We cannot
expect certainty from data, only fallible
information.
7
To some extent this approach resembles that of
Bock (Note 5).
359
With regard to some topics, I have been too
brief. I have not considered treatment-pretest
interactions (Goldberger, Note 6) or more
generally the whole issue of linearity. I have
deliberately (and also perhaps inadvertently)
oversimplified many issues to ease and clarify
their presentation. I have ignored any dis-
cussion of power (Goldberger, Note 7; Porter,
Note 8) and have not discussed the issues of
estimation and significance testing enough.
I have not examined the relevance of econo-
metric literature. For instance, the method
of instrumental variables (Goldberger, 1964)
could be applied in certain cases. Also, I have
not considered the case of testing the persist-
ence of treatment effects. Finally, I have not
discussed either multilevel treatments or the
analysis of both manipulated and unmanipu-
lated treatment in pre-post true experimental
designs.
I wish to clear up one unintended emphasis
of the article. For heuristic purposes the
comparison of the treated and untreated
groups has been emphasized. Although such
overall evaluations are important, they have
been overemphasized. Since a program may
only work in certain highly specific conditions,
such global comparisons often mask the
interactive nature of treatment effects. Re-
searchers should examine not only the molar
effects of the treatment but also the molecular-
process effects within the treatment, so that
the adaptive aspects of programs can be
selected and not "thrown out with the bath-
water." An excellent illustration of within-
program evaluation is in Coulson (Note 9),
which examines the effect of variations within
Head Start programs such as program orien-
tation, teacher-student ratio, teacher creden-
tials, budget, and others.
Researchers have a special responsibility in
the design and evaluation of real-world
treatments. We not only have the usual
responsibility of scientists to find and report
the truth but also the responsibility of citizens
to contribute to the public good. We should
not routinely apply statistical methods that
were useful in the laboratory but should scrupu-
lously consider the biases of each method.
Some might argue that only true experi-
ments bring valid causal inference, and it is
always logically possible to contrive alter-
 360 DAVID A. KENNY
native explanations to any quasi-experimental
design. For instance, for the nonequivalent
control group design any set of results can be
explained by the plausible rival hypotheses
of the interactions of selection with history,
selection with maturation, and selection with
testing (see Cook & Campbell, 1975, for some
of these explanations). But because the
internal validity of quasi-experiments is lower
than true experiments, it does not argue
against using the judgments of quasi-experi-
ments. We would all prefer to have the testi-
mony about an event from a sighted man over
a blind man. But when we have only the blind
man, we would not dismiss his testimony,
especially if he were aware of his biases and had
developed faculties of touch and hearing that
the sighted man could have developed but has
neglected. The difference between the true
experiment and the quasi-experiment is of the
magnitude of the difference between sight and
blindness. We must often grope in the darkness
with quasi-experimental designs, but this
blindness both forces us to compensate for
biases and helps us develop a newfound
sensitivity to the structure of data. Finally,
it makes us appreciate the clarity of true
experimental inference.
REFERENCE NOTES
1. Campbell, D. T. Temporal changes in treatment-
effect correlations: A quasi-experimental model for
institutional records and longitudinal studies. In
G. V. Glass (Ed.), Proceedings of the 1970 Invita-
tional Conference on Testing Problems. Princeton,
N.J.: Educational Testing Service, 1971.
2. Campbell, D. T. The effects of college on students:
Proposing a quasi-experimental approach. Research
report, Northwestern University, 1967.
3. O'Conner, E. F. Unraveling Lord's paradox: The
appropriate use of multiple regression analysis in
quasi-experimental research. Educational Testing
Service Research Bulletin RB-73-53, 1973.
4. Kenny, D. A. PANAL: Panel data analysis of the
first year of "Sesame Street." Research report,
Harvard University, 1975.
5. Bock, R. D. Conditional and unconditional inference
in the analysis of repeated measurements. Paper
presented at the Symposium on the Application of
Statistical Techniques to Psychological Research,
Canadian Psychological Association, York Uni-
versity, 1969.
6. Goldberger, A. S. Selection bias in evaluating treat-
ment effects: Some formal illustrations. Madison:
Institute for Research on Poverty, University of
Wisconsin, 1972.
7. Goldberger, A. S. Selection bias in evaluating treat-
ment effects: The case of interaction. Madison: Insti-
tute for Research on Poverty, University of Wis-
consin, 1972.
8. Porter, A. C. Analysis strategies for some common
evaluation paradigms. Office of Research Consul-
tation, Michigan State University, 1973.
9. Coulson, J. E. Effects of different Head Start program
approaches on children of different characteristics:
Report on analysis of data from 1968-1969 national
evaluation. Office of Child Development, Department
of Health, Education, and Welfare Contract HEW-
OS-70-168, 1972.
REFERENCES
Bereiter, C. Some persisting dilemmas in the measure-
ment of change. In C. W. Harris (Ed.), Problems in
measuring change. Madison: University of Wisconsin
Press, 1963.
Campbell, D. T. Factors relevant to the validity of
experiments in social settings. Psychological Bulletin,
1957 54,297-312.
Campbell, D. T. Reforms as experiments. American
Psychologist, 1969, 24, 409-429.
Campbell, D. T., & Clayton, K. N. Avoiding regression
effects in panel studies of communication impact.
Studies in Public Communication, Department of
Sociology, University of Chicago, No. 3, 1961
(Bobbs-Merrill Reprint S-353).
Campbell, D. T., & Erlebacher, A. How regression
artifacts in quasi-experimental evaluations can
mistakenly make compensatory eduction harmful. In
J. Hellmuth (Ed.), The disadvantaged child (Vol. 3).
New York: Brunner/Mazel, 1971.
Campbell, D. T., & Stanley, J. C., Experimental and
quasi-experimental designs for research on teaching.
In N. L. Gage (Ed.), Handbook of research on teaching.
Chicago: Rand McNally, 1963.
Chen, M. K. A critical look at the matching technique
in experimentation. The Journal of Experimental
Education, 1967, 35, 95-98.
Cohen, J. Multiple regression as a general data-analytic
system. Psychological Bulletin, 1968, 70, 426-443.
Coleman, J. S. et al. Equality of educational opportunity.
Washington, D. C.: Government Printing Office,
1966.
Cook, T. D. The potential and limitations of secondary
evaluations. In M. Apple, H. J. Lufler, & B. M.
Subkoviak (Eds.), Educational evaluation: Analysis
and responsibility. Berkeley, Calif.: McCutchan
Press, 1974.
Cook, T. D., & Campbell, D. T. The design and conduct
of quasi-experiments and true experiments in field
settings. In M. D. Dunnette (Ed.), Handbook of
industrial and organizational research. Chicago: Rand
McNally, 1975.
Cranb, W. B., Kenny, D. A., & Campbell, D. T. Does
intelligence cause achievement? A cross-lagged panel
analysis. Journal of Educational Psychology, 1972, 63,
258-275.
Crider, D. M., Willits, F. K., & Bealer, R. C. Panel
studies: Some practical problems. Sociological
Methods fir Research, 1973, 2, 3-19.
 NONEQU1VALENT CONTROL GROUP DESIGN
Cronbach, L. E., & Furby, L. How we should measure
"change"—or should we? Psychological Bulletin,
1970, 74, 68-80.
Dunn, O. J., & Clark, V. Correlation coefficients
measured on the same individuals. Journal of the
American Statistical Association, 1969, 64, 366-377.
Dunn, O. J., & Clark, V. Tests of equality of dependent
correlation coefficients. Journal of the American
Statistical Association, 1971, 66, 904-908.
Goldberger, A. S. Econometric theory. New York:
Wiley, 1964.
Hilton, T. L., & Myers, A. E. Personal background,
experience, and school achievement: An investigation
of the contribution of questionnaire data to academic
prediction. Journal of Educational Measurement,
1967, 4, 69-80.
Jensen, A. R. How much can we boost IQ and scholastic
achievement? Harvard Educational Review, 1969, 39,
1-123.
Kenny, D. A. Cross-lagged and synchronous common
factors in panel data. In A. S. Goldberger & O. D.
Duncan (Eds.), Structural equation models in the
social sciences. New York: Seminar Press, 1973.
Kenny, D. A. A test for a vanishing tetrad: The second
canonical correlation equals zero. Social Science
Research, 1974, 3, 83-87.
Lindquist, E. F., & Hieronymus, A. N. Iowa tests of
basic skills: Manual for administrators, supervisors,
and counselors. Boston: Houghton Mifflin, 1964.
Lord, F. M. Large-scale covariance analysis when the
control variable is fallible. Journal of the American
Statistical Association, 1960, 55, 307-321.
Lord, F. M. A paradox in the interpretation of group
comparison. Psychological Bulletin, 1967, 68,304-305.
APPENDIX
To demonstrate the null hypothesis for each
statistical technique in Table 1, it must be
shown that given a zero covariance between
the treatment (T) with the dependent variable,
the null hypothesis follows. Letting C(X, Y) be
the covariance of X and Y, we begin with
analysis of covariance :
0 = C(T, X,) -
Since
C(Xlt X£ - C(Xi, T)C(X2,
it follows that
Multiplying through by
361
Lord, F. M., & Novick, M. R. Statistical theories of
mental test scores. Reading, Mass.: Addison-Wesley,
1968.
McNemar, Q. Psychological statistics (4th ed.). New
York: Wiley, 1969.
Porter, A. C. The effects of using fallible variables in the
analysis of covariance. Unpublished doctoral disserta-
tion, University of Wisconsin, 1967.
Rosenthal, R., & Rosnow, R. L. The volunteer subject.
In R. Rosenthal & R. L. Rosnow (Eds.), Artifact in
behavioral research. New York: Academic Press,
1969.
Rulon, P. J. Problems of regression. Harvard Educa-
tional Review, 1941,11, 213-223.
Sween, J. A. The experimental regression design in
inquiry into the feasibility of nonrandom treatment
allocation. Unpublished doctoral dissertation, North-
western University, 1971.
Thistlethwaite, D. L., & Campbell, D. T. Regression-
discontinuity analysis: An alternative to the ex post
facto experiment. Journal of Educational Psychology,
1969, 51, 309-317.
Weiss, C. H. Evaluation research. Englewood Cliffs,
N.J.: Prentice-Hall, 1972.
Werts, C. E., & Linn, R. L. A general linear model for
studying growth. Psychological Bulletin, 1970, 73,
17-22. (a)
Werts, C. E., & Linn, R. L. Path analysis: Psycho-
logical examples. Psychological Bulletin, 1970, 74,
193-212. (b)
Woodrow, H. The relation of verbal ability to improve-
ment with practice in verbal tests. Journal of Educa.-
tional Psychology, 1939, 30, 179-186.
and subtracting, the result is
0 = vx?C(T, X2) - C(T,
Dividing through by <TT<rx^ffxf, one gets
0 = C(T,
- C(T,
0 = prx2 —
Some may recognize the above formula as the
numerator of the formula for pz2r.Xi and
PxiT.Xi, that is, the partial correlation and
standardized partial regression coefficient of
Xt and T controlling for Xi.
The logic for analysis of covariance with
reliability correction is the same as above with
the inclusion of the reliability of the pretest
in the formula.
For raw change score analysis one begins
with
0 = C(r, X2 - Xi)
0 = c(r, *,) -C(T,XI).
362 DAVID A. KENNY

Multiplying through by f T) , the where * indicates zero mean and unit variance.
result is Multiplying through by I/or one gets
0 = C(r, 0 = ~C(T,
0 =
0 =
since X*i and X*i have unit variance. One
For standardized change score analysis should note that it need only be assumed that
0 = C(r, X*z - X*i) X*i and X*2 have equal variance.
= c(r, (Received April 22, 1974)

View publication stats