6
Validity of Measures
With Karen L . Soeken
Validity is defined as the degree to which
evidence and theory support the interpretation
entailed by proposed use of tests (American
Educational Research Association [AERA],
American Psychological Association [APA],
National Council on Measurement in Education
[NCME], 1999, p. 9). The type of validity infor-
mation to be obtained depends upon the aims or
purposes for the measure rather than upon the
type of measure per se. Hence, it is not appro-
priate to speak of a valid tool or measurement
method, but rather of accruing evidence for
validity by examining the scores resulting from a
measure that is employed for a specific purpose
with a specified group of respondents under a
certain set of conditions. For any given measure,
different aspects of validity will be investigated
depending upon the measures purpose(s). That
is, when test scores are used or interpreted in
more than one way, each intended interpretation
must be validated (AERA, APA, & NCME, 1999,
pp. 9 11).
Standards for measurement validity changed
substantially in the Standards for Educational
and Psychological Testing published by AERA,
APA, and NCME in 1999. These changes resulted
from concern that the focus on only the three
types of validity (content, criterion-related, and
construct) tended to narrow thinking and lead
to the misconception that validity is a property
of a test or measure per se. In the 1999 Stan-
dards the definition of validity was expanded
to emphasize that validity should be viewed as
a unitary concept. Validity is not a property of a
tool. It is a property of the scores obtained with
a measure when used for a specific purpose and
with a particular group of respondents.
Thus, for any given measure, one or more types
of evidence will be of interest. Evidence for the
validity of a newly developed measure requires
extensive, rigorous investigation using a num-
ber of different approaches depending upon
the purpose(s), subjects, and the conditions
under which it will be used, prior to employing
the measure clinically and/or within a research
study. In addition, evidence for validity should
be obtained within the context of each study in
which the measure is used for the collection of
data. Therefore, for any given tool or measure-
ment method, validity will be investigated in
multiple ways depending upon the purpose(s)
for measurement and evidence for validity will
be accrued with repeated use of the measure. The
Standards focus on five distinct types of evidence
based on: (1) test content, (2) response processes,
(3) internal structure, (4) relations with other
variables, and (5) the consequences of testing.
Two important points are emphasized: First,
these sources of evidence may illuminate differ-
ent aspects of validity, but they do not represent
distinct types of validity . . . it is the degree to
which all of the accumulated evidence supports
the intended interpretation of test scores for the
intended purpose (AERA, APA, & NCME, 1999,
p. 11). Second, A sound validity argument inte-
grates various strands of evidence into a coherent
account of the degree to which existing evidence
and theory support the intended interpretation
of test scores for specific uses . . . encompasses
evidence gathered from new studies and evi-
dence available from earlier reported research . . .
indicates the need for refining the definition of
the construct, may suggest revisions in the test
or other aspects of the testing process, and may
163
164 Part II Understanding Measurement Design
indicate areas needing further study (AERA,
APA, & NCME, 1999, p. 17).
Five types of evidence to be considered when
investigating the validity of a measure are
described in the Standards:
1. Evidence based on test content: The extent
to which content, including specific items
or tasks and their formats, represent the
content domain. Evidence for relevance of
the content domain to the proposed inter-
pretation of scores might include:
Extent to which content represents the
content domain (Content Validity)
Experts review of sufficiency, relevance,
and clarity of items (Content Valid-
ity, Item Analysis), extent to which the
intended construct is measured (Con-
struct Validity)
Extent to which items or subparts match
the definition of the construct and/
or purpose of the measure (Construct
Validity)
Evidence that construct underrepresen-
tation or construct-irrelevant components
may give unfair advantage to subgroups
of subjects (Construct Validity)
2. Evidence based on response processes: The
fit between the type of performances or
responses in which respondents engage
and the intended construct. Evidence that
the construct of interest is assessed and
criteria applied as intended and responses
are not the result of irrelevant or extra-
neous factors like social desirability may
include:
Interviewing subjects to assess the basis
for their responses
Observing subjects as they engage in
task performances
Studying the way in which judges or
observers employ criteria to record and
score performances
3. Evidence based on internal structure: The
extent to which relationships among items
and components match the construct as
operationally defined. Evidence based on
internal structure might include:
Factor analysis, especially Confirmatory
Factor Analysis (CFA)
Differential item function (DIF) studies
to detect item bias
Item analysis to examine interrelation-
ships
4. Evidence based on relations to other vari-
ables: The nature and extent of the relation-
ships between scores and other variables
that the measure is expected to correlate
with or predict, as well as variables that
the measure is not expected to relate with.
Extent to which these relationships are
consistent with the construct that serves as
the basis for the proposed interpretation
of scores. Evidence based on relations with
other variables may include:
Correlation with scores from another
measure of the same construct (Criterion-
related Validity)
Extent to which scores correlate with
data obtained at a later date (Criterion-
related Validity)
Differential group prediction studies
Validity generalization studies (Meta-
analysis)
Convergent and divergent validity studies
(Multitrait-multimethod)
Experimental and known group com-
parison studies
5. Evidence based on consequences of testing:
Extent to which anticipated benefits of
measurement are realized and/or unantici-
pated benefits (positive and negative) occur.
Extent to which differential consequences
are observed for different identifiable sub-
groups. Evidence based on consequences
of testing may include:
Descriptive studies
Focus groups (AERA, APA, & NCME,
1999; Goodwin, 2002, pp. 100106).
A number of validation frameworks have been
proposed that are based on these Standards.
Readers interested in learning more regard-
ing these frameworks are referred to Slaney
and Maraun (2008) who provide an overview
and evaluation of validation frameworks pro-
posed by others and propose a framework for
test analysis based upon the Standards that
addresses some of the limitations in existing
frameworks, and Baldwin and colleagues (2009)

who developed and implemented a process to
validate the 75 Core National Association of
Clinical Nurse Specialists (NACNS) clinical
nurse specialist (CNS) competencies among
practicing CNSs.
NORM-REFERENCED
VALIDITY PROCEDURES
Content Validity
Validation of the extent to which content rep-
resents the content domain is important for all
measures and is especially of interest for instru-
ments designed to assess cognition. The focus is
on determining whether the items sampled for
inclusion on the tool adequately represent the
domain of content addressed by the instrument
and the relevance of the content domain to the
proposed interpretation of scores obtained when
the measure is employed. For this reason, content
validity is largely a function of how an instru-
ment is developed. When a domain is adequately
defined, objectives that represent that domain
are clearly explicated, an exhaustive set of items
to measure each objective is constructed, and
then a random sampling procedure is employed
to select a subset of items from this larger pool
for inclusion on the instrument, the probability
that the instrument will have adequate content
validity is high.
When investigating content validity, the
interest is in the extent to which the content
of the measure represents the content domain.
Procedures employed for this purpose usually
involve having experts judge the specific items
and/or behaviors included in the measure in
terms of their relevance, sufficiency, and clar-
ity in representing the concepts underlying the
measures development. To obtain evidence for
content validity, the list of behavioral objec-
tives that guided the construction of the tool, a
definition of terms, and a separate list of items
designed to specifically test the objectives are
given to at least two experts in the area of con-
tent to be measured. These experts are then
asked to (l) link each objective with its respec-
tive item, (2) assess the relevancy of the items
to the content addressed by the objectives, and
Chapter 6 Validity of Measures 165
(3) judge if they believe the items on the tool
adequately represent the content or behaviors in
the domain of interest.
When only two judges are employed, the
content validity index (CVI) is used to quantify
the extent of agreement between the experts. To
compute the CVI, two content specialists are
given the objectives and items and are asked
to independently rate the relevance of each
item to the objective(s) using a 4-point rating
scale: (1) not relevant, (2) somewhat relevant,
(3) quite relevant, and (4) very relevant. The
CVI is defined as the proportion of items given
a rating of quite/very relevant by both raters
involved. For example, suppose the relevance of
each of 10 items on an instrument to a particular
objective is independently rated by two experts
using the 4-point scale, and the results are those
displayed in Figure 6.1. Using the information
from the figure, the CVI equals the proportion
of items given a rating of 3 or 4 by both judges:
CVI = 8/10 or 0.80. If all items are given ratings
of 3 or 4 by both raters, interrater agreement
will be perfect and the value of the CVI will be
1.00. If one-half of the items are jointly classi-
fied as 1 or 2, while the others are jointly classi-
fied as 3 or 4, the CVI will be 0.50, indicating an
unacceptable level of content validity (Martuza,
1977).
When more than two experts rate the items
on a measure, the alpha coefficient, discussed
in Chapter 5, is employed as the index of con-
tent validity. Figure 6.2 provides an example
of alpha employed for the determination of
content validity for six experts rating of the
relevance of each of five items on a measure.
In this case, column headings represent each
of the six experts ratings (AF), and the row
headings represent each of the five items (15)
rated. The resulting alpha coefficient quanti-
fies the extent to which there is agreement
between the experts ratings of the items. A
coefficient of 0.00 indicates lack of agreement
between the experts and a coefficient of 1.00
indicates complete agreement. It should be
noted that agreement does not mean that the
same rating was assigned by all experts, but
rather that the relative ordering or ranking of
scores assigned by one expert matches the rela-
tive order assigned by the other experts. When
166 Part II Understanding Measurement Design

Judge 1
(1 or 2) (3 or 4)
not/somewhat quite/very
Judge 2 relevant relevant Total
(1 or 2) not/somewhat relevant 2 0 2
(3 or 4) quite/very relevant 0 8 8
Total 2 8 10

FIGURE 6.1 Two judges ratings of 10 items.

Judges
Subjects A B C D E F Total
1 10 5 4 5 10 8 42
2 8 4 4 3 9 7 35
3 10 5 5 4 8 10 42
4 10 4 5 4 10 10 43
5 9 5 5 5 10 8 42
Total 47 23 23 21 47 43 204
Mean 9.4 4.6 4.6 4.2 9.4 8.6 40.8
Variance 0.64 0.24 0.20 0.70 0.64 1.44 8.56
alpha = ( )[ ( )]
6 1 3.86
5 8.56
= (1.2) (1 0.45)
= (1.2) (0.55)
= 0.66

FIGURE 6.2 Example of alpha employed for the determination of content validity for six judges rating of a
five-item measure.

sufficiency and/or clarity of the specific item
and/or behavior included in the measure in
representing the concept underlying the mea-
sures development is of interest, the same pro-
cedure is employed.
Content validity judgments require subject
matter expertise and therefore careful atten-
tion must be given to the selection, preparation,
and use of experts, and to the optimal number
of experts in specific measurement situations.
Readers interested in more information in
this regard are referred to Berk (1990), Davis
(1992), and Grant and Davis (1997). Content
validity judgments are different from the judg-
ments referred to in determining face validity.
Face validity is not validity in the true sense and
refers only to the appearance of the instrument
to the layperson; that is, if upon cursory inspec-
tion, an instrument appears to measure what
the test constructor claims it measures, it is said
to have face validity. If an instrument has face
validity, the layperson is more apt to be moti-
vated to respond, thus its presence may serve as
a factor in increasing response rate. Face valid-
ity, when it is present, however, does not provide
evidence for validity, that is, evidence that the
instrument actually measures what it purports
to measure.
For example, Ganz et al. (2009) examined
content validity of a survey instrument employed

in a study undertaken to describe the oral-care
practices of ICU nurses to compare those prac-
tices with current evidence-based practice and
to determine if the use of evidence-based prac-
tice was associated with personal demographic
or professional characteristics. Brodsky and
Dijk (2008), when examining the validity of
their questionnaire designed to evaluate Israeli
nurses and physicians attitudes regarding
the introduction of new nursing roles and the
expanding scope of nursing practice, employed
a panel of experts with theoretical knowledge
and practical experience in the fields being ana-
lyzed. Specifically, the panel was asked to inde-
pendently review the measures content, face
validity, and to scrutinize each item regarding
its appropriateness, comprehensiveness, and rel-
evance to the target population. Similarly, Black-
wood and Wilson-Barnett (2007) distributed
their questionnaire to assess nurses knowledge
and principles regarding protocolized-weaning
from mechanical ventilation to a panel of inter-
national experts in critical care with research
experience to comment on its content and face
validity. Other examples of content validity can
be found in Glaister (2007); Schnall et al. (2008);
Gatti (2008); Kearney, Baggs, Broome, Dough-
erty, and Freda (2008); and Hatfield (2008).
Construct Validity
Construct validity is important for all measures
especially measures of affect. The primary con-
cern in assessing construct validity is the extent to
which relationships among items included in the
measure are consistent with the theory and con-
cepts as operationally defined. The importance
of consistency between theory and operation-
alization is exemplified in the work of Foulad-
bakhsh and Stommel (2007) who address the
need for conceptual and operational consistency
in discussing the development of a Complemen-
tary Alternative Medicine Health Care Model.
Activities undertaken to obtain evidence for
construct validity include:
Examining item interrelationships
Investigations of the type and extent of the
relationship between scores and external
variables
Chapter 6 Validity of Measures 167
Studies of the relationship between scores
and other tools or methods intended to
measure the same concepts
Examining relationships between scores
and other measures of different constructs
Hypothesis testing of effects of specific
interventions on scores
Comparison of scores of known groups of
respondents
Testing hypotheses about expected differ-
ences in scores across specific groups of
respondents
Ascertaining similarities and differences
in responses given by members of distinct
subgroups of respondents
Construct validity is usually determined using (1)
the contrasted groups approach, (2) hypothesis
testing approach, (3) the multitrait-multimethod
approach (Campbell & Fiske, 1959; Martuza,
1977), and/or (4) factor analysis (Rew, Stuppy, &
Becker, 1988).
Contrasted Groups
Approach
In the contrasted groups approach, the investi-
gator identifies two groups of individuals who
are known to be extremely high and extremely
low in the characteristic being measured by the
instrument. The instrument is then adminis-
tered to both the high and low groups, and the
differences in the scores obtained by each are
examined. If the instrument is sensitive to indi-
vidual differences in the trait being measured,
the mean performance of these two groups
should differ significantly. Whether the two
groups differ is assessed by use of an appropri-
ate statistical procedure such as the t test or an
analysis-of-variance test.
For example, to examine the validity of a
measure designed to quantify venous access, the
nurse might ask a group of clinical specialists
on a given unit to identify a group of patients
known to have good venous access and a group
known to have very poor access. The nurse
would employ the measure with both groups,
obtain a mean for each group, then compare the
differences between the two means using a t test
or other appropriate statistic. If a significant