See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/235634155

Motivational and Skills, Social, and Self-

Management Predictors of College Outcomes:
Constructing the Student...

Article in Educational and Psychological Measurement June 2005

DOI: 10.1177/0013164404272493

CITATIONS READS

68 1,392

4 authors, including:

Huy Le Alex Casillas

University of Texas at San Antonio ACT, Inc.
55 PUBLICATIONS 3,164 CITATIONS 22 PUBLICATIONS 883 CITATIONS

SEE PROFILE SEE PROFILE

Steven Bernard Robbins

Educational Testing Service
132 PUBLICATIONS 3,790 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Holistic Framework View project

All content following this page was uploaded by Huy Le on 09 July 2017.

The user has requested enhancement of the downloaded file. All in-text references underlined in blue are added to the original document
and are linked to publications on ResearchGate, letting you access and read them immediately.
 10.1177/0013164404272493
EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT
LE ET AL.

MOTIVATIONAL AND SKILLS, SOCIAL, AND

SELF-MANAGEMENT PREDICTORS OF COLLEGE OUTCOMES:
CONSTRUCTING THE STUDENT READINESS INVENTORY

HUY LE
Human Resources Research Organization

ALEX CASILLAS
University of Iowa

STEVEN B. ROBBINS
RONELLE LANGLEY
ACT, Inc.

The authors used a rational-empirical approach to construct the Student Readiness Inven-
tory, measuring psychosocial and academic-related skill factors found to predict two
important college outcomes, academic performance and retention, in a recent meta-
analysis. The initial item pool was administered to 5,970 first-year college students and
high school seniors to empirically validate and cross-validate the underlying factor struc-
ture. Ten first-order and 3 second-order factors were derived, partially resembling the
original conceptual model. Future study is needed to explore the criterion and predictive
validities of the factors constituting this inventory.

Keywords: academic performance; college outcomes; academic-related skills;

noncognitive predictors; psychosocial factors; retention

The search for predictors of college success has long been a research
theme in the educational psychology literature (Hezlett et al., 2001;
Pascarella & Terenzini, 1991). This line of research has both theoretical and
practical significance. Theoretically, the identification of higher order factors

Correspondence concerning this article should be addressed to Steven B. Robbins, ACT, Inc.,
P.O. Box 168, Iowa City, IA 52243-0168; e-mail: steve.robbins@act.org.
Educational and Psychological Measurement, Vol. 65 No. 3, June 2005 482-508
DOI: 10.1177/0013164404272493
2005 Sage Publications
482
 LE ET AL. 483

associated with college success would shed light on students behaviors in

college. Practically, these factors could assist colleges by targeting key areas
for developmental intervention to reduce both the academic and the persis-
tence risk of entering students. Although the educational literature is
replete with theories and models proposing various psychosocial and/or
skill factors that may influence students college success (cf. Pascarella &
Terenzini, 1991; Tinto, 1975, 1993), the diversity of the constructs and the
lack of an integrative framework limit the development of a multidimen-
sional inventory with strong psychometric and conceptual underpinnings.
Robbins et al. (2004) recently sought to provide this integration by meta-
analytically examining the validities of various psychosocial and study skills
constructs in predicting two important college success criteria: academic per-
formance (i.e., college grade point average [GPA]) and persistence (i.e., col-
lege retention). On the basis of 109 studies, the authors identified and studied
nine broad constructs derived from educational persistence and motivational
models of college success: achievement motivation, academic goals, institu-
tional commitment, perceived social support, social involvement, academic
self-efficacy, general self-concept, academic-related skills, and contextual
influences. After controlling for the effects of traditional predictors (i.e., high
school GPA, ACT and SAT assessment scores, and socioeconomic status),
these researchers identified three psychosocial constructs that have demon-
strated incremental validity in predicting the academic performance crite-
rion: academic self-efficacy, achievement motivation, and academic goals.
Additionally, six constructs were found to be predictive of the persistence
criterion (above and beyond the effects of the traditional predictors): aca-
demic goals, academic self-efficacy, institutional commitment, academic-
related skills, social support, and social involvement.
Robbins et al. (2004) proposed that the composite of psychosocial and
academic-related skill predictors was best understood by three higher order
constructs: motivation, academic-related skills, and social engagement.
They also pointed to the limitations of the current empirical research due to
the absence of well-constructed measures subsumed under the same con-
structs and the dearth of studies examining several important motivational
constructs that rendered meta-analytical examination unfeasible. Despite
these limitations, we believe that Robbins et al.s meta-analysis provides a
unified framework for guiding the development of a comprehensive psycho-
social and skills inventory for predicting college success. Therefore, the pur-
pose of this study was to propose and develop an inventory of psychosocial
and skill factors that (a) captures the aforementioned higher order constructs,
(b) includes other important constructs missing in Robbins et al.s meta-
analysis that may be predictive of college success criteria, and (c) establishes
the foundation for the construct validation process of the resulting inventory.
To overcome the limitations of Robbins et al.s meta-analysis, we reviewed
current theories and models, including those from other relevant literatures
484 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

(i.e., personality and industrial-organizational), to identify additional con-

structs for inclusion in our measure development efforts.

Current Theories and Models of College Success

Motivation theories. Covington (2000) and Eccles and Wigfield (2002)

provided comprehensive reviews of the motivation theories that relate to stu-
dents college success. As discussed in these reviews, theories can be catego-
rized into two groups: achievement-as-drive and achievement-as-goal theo-
ries (Covington, 2000). For the former group, the key construct is motivation
to achieve (also called achievement drive; Atkinson, 1964; McCelland,
1980). Indeed, Robbins et al.s (2004) meta-analysis showed that this con-
struct is predictive of college students academic performance above and
beyond traditional predictors. Expectancy-value theories (e.g., Wigfield &
Eccles, 2000) represent the latter group. Two major constructs suggested in
these theories, self-expectancy and values, were also examined in Robbins
et al.s meta-analysis (as academic self-efficacy and academic goals, respec-
tively) and found to be the best predictors for both college outcome criteria.
In summary, it appears that the validities of key constructs suggested under
the prevalent motivational theories were appropriately covered in Robbins
et al.s meta-analysis.

Models combining motivation and skill constructs. The self-regulated

learning model (Schunk & Zimmerman, 2003; Zimmerman, 1986;
Zimmerman & Martinez-Pons, 1986) combines motivational constructs
(e.g., goals, self-efficacy, and self-esteem) and cognitive skills (e.g., self-
regulated learning strategies, including metacognitive, motivational, and
behavioral strategies) to explain the processes through which students self-
regulate their learning. As such, the model explains the processes determin-
ing students performance in college. Accordingly, both the motivational and
cognitive skill constructs suggested in the model were included in Robbins
et al.s (2004) meta-analysis.
Pintrich and colleagues (e.g., Pintrich, 1989, 2000; Pintrich & De Groot,
1990) suggested another model integrating motivational constructs derived
from expectancy-value and goal theories, as well as cognitive constructs
(e.g., background knowledge, learning strategies, and self-regulatory and
metacognitive theories), to predict students academic performance. The
model further introduced an affective construct, test anxiety, that was not
examined in Robbins et al.s meta-analysis. According to this model, test
anxiety is an important predictor of students academic performance
(Pintrich, Smith, Garcia, & McKeachie, 1993). Therefore, we included this
construct in our current measure development efforts.
 LE ET AL. 485

The educational persistence models. The two leading models in the edu-
cational literature predicting studentscollege retention were proposed by
Tinto (1975, 1993) and Bean (1980, 1985). As shown by Robbins et al.
(2004), these models have several common factors: (a) contextual influence,
which includes factors pertaining to an institution that may be likely to affect
college outcomes; (b) perceived social support; (c) social involvement; and
(d) academic engagement, which includes commitment to obtain a degree
and commitment to an academic institution. Because of their roles in the edu-
cational literature, both models have been well researched. Accordingly, the
constructs from these models also were examined in Robbins et al.s meta-
analysis. In particular, the social support, social involvement, and institu-
tional commitment factors were found to be predictive of the college reten-
tion criterion and were included in the current study.

Other Relevant Literatures

We broadened our search for potential predictors by examining the per-

sonality and industrial-organizational psychology literatures. For example,
in recent years, there has been a growing consensus in these literatures about
the basic structure of personality. Specifically, many agree that there are five
broad personality constructs: emotional stability (or neuroticism), extro-
version, conscientiousness, agreeableness, and openness (Digman, 1990;
Goldberg, 1993; John, 1990). Of these, conscientiousness, emotional stabil-
ity, and agreeableness have repeatedly been found to be predictive of two
important criteria in the organizational literature: job performance (Barrick
& Mount, 1991; Hurtz & Donovan, 2000; Saldago, 1997; Tett, Jackson, &
Rothstein, 1991) and turnover (Caligiuri, 2000; Saldago, 2002). These two
criteria are analogous to the criteria of academic performance and reten-
tion in the educational literature. Furthermore, variations of these constructs
have been found to be predictive of college performance (e.g., Malloch &
Michael, 1981; Wolfe & Johnson, 1995) and college retention (Brown &
Cross, 1993) in the educational measurement literature. Thus, it seems likely
that these personality constructs are useful predictors of college outcome
criteria. Indeed, findings from recent studies (e.g., Chamorro-Premuzic &
Furnham, 2003; Okun & Finch, 1998; Paunonen & Ashton, 2001) have
supported this assertion. Consequently, these constructs also were included
in our scale development efforts.

The Conceptual Model

We constructed a rational model to direct scale construction efforts using

the three content domains that surfaced in Robbins et al.s (2004) meta-
486 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

analysis: motivation, academic-related skills, and social engagement. The

motivation domain includes personal characteristics that help students suc-
ceed academically by focusing and maintaining energies on goal-directed
activities. The academic-related skills domain includes cognitive, behav-
ioral, and affective tools and abilities necessary to successfully complete aca-
demic-related tasks. Finally, the social engagement domain includes inter-
personal factors that influence students successful integration or adaptation
into their environment. As stated above, we expanded the model by including
constructs believed to be predictive of college success but not examined in
Robbins et al.s meta-analysis.
The initial conceptual model consists of 10 constructs grouped under
three content domains. Specifically, the motivation domain includes three
constructs: conscientiousness, goal focus, and academic self-efficacy. Con-
scientiousness partially reflects the achievement motivation construct, but it
also includes dependability (cf. Hough, 1992; Mount & Barrick, 1995). We
believed that including both constructs would improve the model, because
the student academic performance criterion is analogous to the job perfor-
mance criterion in the organizational literature (cf. Barrick & Mount, 1991;
Hurtz & Donovan, 2000; Saldago, 1997; Tett et al., 1991). The academic-
related skills domain includes four constructs: study skills, problem solving
skills, communication skills, and emotional control skills. With the excep-
tion of emotional control skills, these constructs were examined in Robbins
et al.s (2004) meta-analysis under the general domain of academic-related
skills. In our conceptual model, we further specified the constructs to facili-
tate the item writing process. Emotional control skills captures the affective
component included in Pintrichs (1989, 2000) model. It also reflects the
construct of emotional stability, which has been found to be predictive of
job performance and turnover criteria in the organizational literature (cf.
Saldago, 2002; Tett et al., 1991).
The social engagement domain includes three constructs: sociability,
social connection, and teamwork. The first two constructs are analogous to
the social involvement and social support constructs featured in Robbins
et al.s (2004) meta-analysis. Teamwork was also included in the conceptual
model, because it captures the construct of agreeableness, which has been
found to be predictive of job performance (Farsides & Woodfield, 2003;
Hurtz & Donovan, 2000; Tett et al., 1991) and turnover (Caligiuri, 2000;
Saldago, 2002) in the organizational literature. Conceivably, those criteria
are analogous to academic performance and college retention, the two col-
lege outcomes of interest. Thus, the inclusion of the teamwork construct was
expected to enhance the model for predicting college outcomes. Table 1 fea-
tures definitions for all constructs in the conceptual model.
 Table 1
Breakdown of the Original Item Pool: Scale Definitions and Number of Items

Domain Construct Definition Number of Items

Motivation Conscientiousness
Goal focus
Academic self-confidence
Academic-related Study skills
skills
Problem solving skills
Communication skills
Emotional control skills
Social engagement Teamwork
Social activity
Social connection
Total items
The extent to which a student is self-disciplined, achievement oriented, responsi- 40
ble, and careful.
The extent to which a student has functional, well-defined academic goals and is 37
committed to achieving these goals.
The extent to which a student has confidence in his or her academic abilities and 30
is willing to use these abilities to cope with academic challenges.
The ability to develop effective strategies and habits for learning in an academic 31
environment.
The ability to use a process of identifying an obstacle, considering solutions, 26
making decisions, and taking appropriate action that results in positive outcomes.
The ability to exchange information effectively with others 28
The ability to understand and effectively manage ones emotions. 27
The ability to work collaboratively with others. 28
The ability to develop and maintain relationships with others. 27
The extent to which a student (a) feels connected to his or her environment and 31
(b) has available social resources.
305

487
488 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Method

General Overview of the Inventory Development Procedure

We followed the construct validation approach (e.g., Clark & Watson,

1995; Loevinger, 1957; Nunnally & Bernstein, 1994) to develop the scales of
interest. Subsequently, a sequential exploratory-confirmatory procedure (cf.
Gerbing & Hamilton, 1996) was used to empirically determine the factor
structure of the scales and to select items. First, we sought comments on the
proposed model from experts in the subfields of education, counseling, and
personality psychology. The general consensus was that the model appro-
priately captured the constructs theoretically expected to be predictors of
college outcomes.
Thus, we began the scale development process using this conceptual
model as a foundation. Specifically, we first generated items representing the
constructs in the conceptual model (see Table 1). After some revisions, the
items were administered to samples of college and high school students. We
carried out an exploratory factor analysis to empirically examine the factors
underlying these items. Items were screened on the basis of their pattern and
structure coefficients. Next, confirmatory factor analysis was implemented
on data from a new sample to (a) confirm the factor structure and (b) reselect
the items. Finally, we attempted to determine the higher order factor structure
of the scales by using second-order factor analyses.

Item Generation Procedure

Item writing. A research team composed of three applied psychologists

wrote items representing the constructs illustrated in the conceptual model.
For each construct, the writers developed a definition and then wrote items
broadly believed to capture the construct. First, the writers generated items
independently and then met to discuss which items were to be retained and/or
revised. This procedure yielded an initial pool of 320 items. In addition, on
the basis of earlier research concerning the optimal number of response
options in Likert-type scales (Green & Rao, 1970; Matell & Jacoby, 1972),
items were set to a 6-point, Likert-type response scale ranging from strongly
disagree (1) to strongly agree (6).

Preliminary test of item clarity. To ensure that the items would be compre-
hensible to 1st-year college students, we organized the items into two forms,
each containing 160 items, and administered them to a small group of high
school seniors (N = 38). The students were asked to rate the readability of the
items (i.e., the extent to which they understood the meaning of the items)
using a 5-point, Likert-type scale ranging from very easy to understand (1) to
 LE ET AL. 489

very difficult to understand (5). On the basis of the mean ratings of item clar-
ity, we deleted or revised items. Subsequently, the revised items were pre-
sented to a group of experts in education and communication, who were
asked to comment on item clarity. The items were again revised on the basis
of this feedback. The resulting item pool consisted of 305 items. (Table 1 fea-
tures the number of items assigned to each scale.)

Study Design

The questionnaires. For various practical administrative concerns (e.g.,

time constraints, potential fatigue), we organized the items into 10 partially
overlapping forms. Specifically, the 305 items were first randomly assigned
to five nonoverlapping item clusters, each with 61 items. Subsequently, we
created 10 different questionnaires by using three different clusters for each
form. Thus, each questionnaire consisted of 183 items, which required
approximately 30 minutes for students to complete. The questionnaires were
administered sequentially to the study sample. This procedure followed the
balanced incomplete block spiraling design (cf. Zwick, 1987). It ensured that
the minimum sample size of the correlation between any two items (i.e., the
effective sample size) was approximately three tenths of the total sample size.
The maximum sample size was approximately three fifths of the total sample
size. In the terminology of generalizability theory (Cronbach, Gleser, Nanda,
& Rajaratnam, 1972), the current study design is the two-facet nested design
p:(i:j), where i, j, and p represent conditions in the universes of items, test
forms, and the population of subjects, respectively.

Test administration procedure. The questionnaires were administered to

the study sample in a group setting during class time. Students were informed
that participation was voluntary and that the questionnaire would require
approximately 30 minutes. The 10 forms of the questionnaire were spirally
distributed to students in each test administration session.

Study sample. Because the scales were constructed with the goal of identi-
fying at-risk college students, the primary population of interest was 1st-year
students at postsecondary institutions (i.e., community colleges and 4-year
universities). Thus, we selected samples that closely resemble this popula-
tion. Specifically, we recruited 1st-year students at community colleges and
universities, as well as high school seniors, to participate in the study. Partici-
pating institutions were given group summaries of their students results as
an incentive for participating.
A total of 50 institutions (22 high schools, 22 community colleges, and six
4-year universities) participated in the study. Reflecting the ACT customer
base, high schools primarily were located in the Midwest, South, Southeast,
490 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

and Southwest, whereas 2- and 4-year colleges were dispersed across all
regions (the ACT is administered to more than 5 millions students and adults
each year). The total number of questionnaires returned was 6,456 (high
school: 2,337; community college: 2,471; university: 1,648). Of these, 5,970
were usable. Data from 486 questionnaires were discarded because partici-
pants did not indicate which test form was used, rendering the forms un-
scorable, or because of random response patterns. Participants were mostly
female (57.2%) and Caucasian (66.9%), with a mean age of 20 years (SD =
5.90 years, range 16 to 68 years). Because of the relatively small sample sizes
of some ethnic and racial subgroups (e.g., n = 48 Native Americans, n = 23
Native Hawaiian/Pacific Islanders), we were unable to make group compari-
sons using each ethnic and racial groups. Instead, we grouped ethnic and
racial minorities into an overall minority group. This group was used for
some of the comparative analyses presented in subsequent sections.

Data Analyses

Step 1: Exploratory factor analysis. First, we used exploratory factor

analysis to examine the factors underlying the data. These analyses were car-
ried out on approximately two thirds of the total sample, henceforth referred
to as the exploratory sample. We used the SAS (version 8.01; SAS Institute,
1999) FACTOR procedure and specified principal-axis factoring as the
extraction method (cf. Gorsuch, 1997; Nunnally & Bernstein, 1994). The
number of factors to be retained was determined on the basis of several crite-
ria, including examination of the resulting scree plot, parallel analysis, and
factor interpretability. The factors were then rotated using the oblimin rota-
tion method, which allowed the factors to be correlated (cf. Gorsuch, 1997).
On the basis of both the pattern coefficients and structure coefficients, we
preselected items for further analysis in the next step. The selected items
were expected to maximize a clean factor structure.

Step 2: Confirmatory factor analysis. The purpose of this analysis was

twofold: (a) to confirm the factors determined in the previous step and (b) to
reselect items representing the factors. The confirmatory factor analysis was
carried out on the remaining one third of the total sample, henceforth referred
to as the confirmatory sample. We specified the measurement model on the
basis of the pattern of itemlatent factor relationships found in the explor-
atory step. Specifically, for each item, the path from its respective latent fac-
tor (i.e., regression weight for the factor or path coefficient) was allowed to be
freely estimated while the paths from other factors were constrained to be
zero. We examined the extent to which the model fit the data by using the
combination of several fit indexes (i.e., the comparative fit index [CFI], the
root mean square error of approximation [RMSEA], and the standardized
 LE ET AL. 491

root mean square residual [SRMR]; Hu & Bentler, 1999). We then reselected
the items on the basis of the magnitudes of their path coefficients using
LISREL (version 8.30) (Jreskog & Srbom, 1999). Approximately 10
items were selected for each factor.

Step 3: Analyses to determine scale properties. For these analyses, the

entire sample was used. We estimated the internal consistency reliability
(i.e., Cronbachs coefficient
) of scores on the resulting scales for the factors
determined in the previous steps. We also estimated the scales intercorre-
lations, as well as their correlations with demographic variables (i.e., age,
gender, ethnicity, and family income) and achievement scores (i.e., reported
high school GPA, ACT Assessment scores). The SAS CORR procedure was
used for these analyses.

Step 4: Second-order analyses. We examined the higher factor structure

of the scales using second-order analyses. The resulting factor structure was
then compared with the conceptual model. For these analyses, we followed
the exploratory-confirmatory procedure used in Steps 1 and 2. Specifically,
we again randomly split the entire sample into two new, independent sub-
samples. The first subsample was used to explore the higher order factor
structure underlying the factors established in the earlier steps (Steps 1 and
2). Particularly, we expected to recover second-order factors similarly
hypothesized to reflect the motivation, academic-related skills, and social
engagement constructs. Exploratory factor analysis was carried out on the
basis of the correlations of the first-order factors. Next, the second subsample
was used to confirm the factor structure by means of second-order confirma-
tory analysis (cf. Rindskopf & Rose, 1988).
Additionally, we performed measurement equivalence analyses (cf.
Byrne, Shavelson, & Muthen, 1989; Vandenberg & Lance, 2000) to further
examine the factorial equivalence of the scales across the following sets of
subgroups: (a) men versus women, (b) ethnic majority versus minority, and
(c) high school versus community college versus university students. This
analysis involved comparing hierarchically nested models by means of
multigroup confirmatory factor analysis. The factor structures of the scales
(i.e., path coefficients and interfactor covariances) were not expected to vary
significantly across the subgroups.

Results

Exploratory Factor Analysis

We randomly selected 3,300 responses (330 for each test form) from the
total 5,970 responses for this analysis. Because of our partially overlapping
492 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

study design, the effective sample size ranged from 990 (i.e., three tenths of
3,300) to 1,980 (i.e., three fifths of 3,300). Exploratory factor analysis
(principal-axis factoring) with pairwise deletion was carried out. On the basis
of the resulting scree plot (Cattell, 1966), we initially extracted 11 factors.
The factors were then rotated using the oblimin rotation method (different
rotation methods were also used to thoroughly examine the factor solutions,
including different oblique [i.e., promax and oblimin with different parame-
ters] and orthogonal [varimax and quartimax] methods, all of which yielded
essentially the same results). On item examination, we decided to retain only
10 factors because of the uninterpretability of the items belonging to Factor
11. Following the suggestion of a reviewer, we additionally conducted a par-
allel analysis (using the procedure provided by Thompson & Daniel, 1996)
to further determine the number of factors to be extracted. This analysis
resulted in 16 factors. However, out of those factors, only 10 were interpret-
able. These 10 factors were very similar to those retained in the earlier
analyses.
We reran the exploratory factor analysis specifying a 10-factor solution
using oblique rotation (using the oblimin rotation [ = 0] method). The 10
factors appeared interpretable. The 10 factors accounted for about 37.4% of
the total variance. This percentage of variance appeared to be acceptable,
given the relatively large number of items examined (i.e., 305). The factors
were low to moderately correlated with one another. The mean of the
interfactor correlations was .06 (range r = |.02| to |.42|), and the standard devi-
ation was .19. (A table with the resulting factors and factor intercorrelations
and the number of items clearly associated with each factor is available from
the authors on request.) From these results, we tentatively selected 145 items
on the basis of the relative magnitudes of their pattern coefficients (i.e., larger
than .30 on the principal factor and lower than .20 on other secondary factors)
for further (confirmatory) analysis.

Confirmatory Factor Analysis

Data for this analysis were the remaining 2,670 responses (minimum n =
800). Confirmatory factor analysis was carried out examining the model
specifying 10 latent factors with 145 items as indicators. The items were
specified to be indicators of the factors determined in the earlier step. We
used the maximum likelihood estimation method, with sample size specified
as the geometric mean of sample sizes of all correlation pairs (n = 1,035; cf.
Viswesvaran & Ones, 1995). The model showed a very good fit (@ 2 =
10,486.72, df = 10,250, p = .05, CFI = .99, RMSEA = .012, SRMR = .035),
confirming the factor structures determined in the exploratory analysis. We
then proceeded to reselect the items on the basis of the magnitudes of the
regression weights for their assigned factors, as previously stated in the
 LE ET AL. 493

Method section. Ninety-five items were selected to represent the 10 factors

(latent constructs). Table 2 features the final factors, their definitions, and
sample items and the number of items belonging to each factor.

Examination of Scale Properties

The entire sample (N = 5,970) was used in these analyses. Scale scores
were formed by averaging (instead of summing) the items belonging to each
factor. Because of the nature of our study design (see the Method section),
this procedure was necessary, because participants were not administered all
items. Means and standard deviations of the scales are presented in Table 2.
Except for three scales (i.e., Commitment to College, Communication Skills,
and General Determination), whose distributions were somewhat negatively
skewed, the distributions of most of the scales were approximately normal.
(Distributional statistics [e.g., skewness, kurtosis] and plots of the scales are
available from the authors on request.) Table 2 also shows the internal consis-
tencies (Cronbachs coefficient
) of the scales (range
= .72 to .87, median

= .82).
Scale intercorrelations and their correlations with other variables of inter-
est (i.e., demographic variables, high school GPA, and ACT Assessment
scores) are presented in Tables 3 and 4, respectively. Overall, the scales show
a good convergent-discriminant pattern, with scales correlating more
strongly with scales conceptually designed to relate more highly to one
another than with other scales. For example, the General Determination,
Academic Discipline, Goal Striving, and Commitment to College scales
were correlated more highly with one another (range r = .39 to .56, median
r = .46) than with other scales (range r = .16 to .53, median r = .34). This pat-
tern was also seen in the relation between the Social Activity and Social Con-
nection scales, which correlated more highly with each other (r = .40) than
with other scales (range r = .14 to .39, median r = .27). However, the con-
vergent-discriminant pattern was less apparent between the remaining two
sets of scales, Study Skills and Communication Skills and Academic Self-
Confidence and Emotional Control. Thus, to better examine the underlying
structure of the scales, we performed higher order analyses (featured in sub-
sequent sections).
As can be seen in Table 4, the correlations with ethnicity and reported
family income were generally small (range r = .00 to .10). Although some of
these correlations were statistically significant because of the large sample
size, their magnitudes did not seem practically significant. This pattern of
correlations suggests that the scales do not discriminate on the basis of these
important demographic variables. Additionally, most correlations between
the scales and age were small in the practical sense, with the exception of the

(text continues on page 497)

494
Table 2
Factor Definitions and Scale Properties

Factor Name, Definition, and Sample Item Number of Items
M SD

General Determination: the extent to which students are dutiful, careful, and dependable. Sample
item: When I make plans, I follow through with them. 5 .72 4.82 0.99
Academic Discipline: the extent to which students value schoolwork and approach school-related tasks
conscientiously. Sample item: I do my best in my classes. 10 .83 4.27 0.97
Goal Striving: the extent to which students (a) set important goals, (b) make efforts to achieve the goals,
and (c) are confident about their abilities to succeed. Sample item: Once I set a goal, I do my best to
achieve it. 10 .87 4.60 0.89
Commitment to College: the extent to which students appreciate the values of education and are
committed to attaining the college degree. Sample item: I am motivated to get a college degree. 10 .86 4.90 1.01
Study Skills: the extent to which students know how to approach academic related problems
systematically and effectively. Sample item: I organize my thoughts before I prepare an assignment. 12 .82 4.02 0.93
Communication Skills: the extent to which students know how to handle interpersonal problems
effectively and can work cooperatively with others in team or group settings. Sample item: Im
willing to compromise when resolving a conflict. 10 .84 4.71 0.90
Social Activity: the extent to which students are comfortable in becoming involved in social activities.
Sample item: I find it hard to talk to people I dont know well. 10 .83 4.11 1.02
Social Connection: the extent to which students are involved in the college or school environments. Sample
item: I am involved in campus activities. 11 .79 3.94 0.97
Academic Self-Confidence: the extent to which students are confident that they can perform well in school.
Sample item: I am a fast learner. 9 .76 3.96 1.04
Emotional Control: the extent to which students can effectively control their emotions and keep them from
negatively affecting other activities. Sample item: I have a bad temper. 8 .76 3.71 1.06

Note. Means and standard deviations are the averages of the items belonging to each scale, with a potential range of 1.00 to 6.00.
 Table 3
Intercorrelations Among the Final Scales

Scale Name 1 2 3 4 5 6 7 8 9 10

1. General Determination
2. Academic Discipline .41
3. Goal Striving .47 .56
4. Commitment to College .39 .46 .56
5. Study Skills .35 .44 .50 .28
6. Communication Skills .43 .38 .53 .45 .37
7. Social Activity .22 .18 .36 .29 .14 .26
8. Social Connection .21 .27 .39 .28 .27 .36 .40
9. Academic Self-Confidence .17 .33 .37 .35 .20 .20 .23 .19
10. Emotional Control .16 .29 .28 .29 .21 .30 .22 .15 .26

Note. n = 2,844 to 5,945.

495
496
Table 4
Correlations of the Scales With Demographic and Academic Variables

a b
Scale Ethnicity Income Age Gender High School GPA ACT Score

General Determination .01 .01 .10 .15 .11 .01

Academic Discipline .01 .00 .13 .24 .28 .05
Goal Striving .07 .03 .14 .15 .16 .02
Commitment to College .02 .00 .07 .23 .21 .07
Study Skills .10 .06 .21 .14 .09 .00
Communication Skills .03 .02 .07 .22 .12 .03
Social Activity .06 .05 .00 .10 .08 .03
Social Connection .06 .06 .15 .12 .20 .05
Academic Self-Confidence .01 .01 .01 .06 .32 .32
Emotional Control .02 .01 .11 .05 .10 .05

Note. n = 1,601 to 5,948.

a. Minority = 0, majority = 1.
b. Male = 0, female = 1.
 LE ET AL. 497

correlation between the Study Skills scale and age (r = .21). We thought that
this correlation might be influenced by the fact that older respondents were
more likely to be community college or university students. Therefore, we
carried out additional analyses partialing out student status (high school stu-
dent vs. community college or university student). The resulting correlation
(r = .19) was only slightly smaller than the zero-order correlation. Although
this finding ruled out the hypothesis that the correlation was due to student
status, the reason for this correlation remains unknown. If replicated, we plan
to explore this issue in subsequent research.
Several scales (i.e., Commitment to College, Communication Skills, Aca-
demic Discipline, and to a lesser extent General Determination, Goal Striv-
ing, and Study Skills) were correlated with gender (range r = .14 to .24). Spe-
cifically, these correlations indicated that women tended to score higher than
men on these scales. These findings were generally expected, given the
nature of the constructs underlying the scales and previous research findings
that men and women tend to score differently on various psychological mea-
sures, including measures of agreeableness (Budaev, 1999; Charbonneau &
Nicol, 2002; Feingold, 1994), measures of conscientiousness (Charbonneau
& Nicol, 2002; Gullone & Moore, 2000; Guthrie, Ash, & Stevens, 2003), and
study strategy measures (e.g., Hong & Milgram, 1999; Slotte, Lonka, &
Lindblom-Ylaenne, 2001). Moreover, a recent international study conducted
by the Organisation for Economic Co-operation and Development (2003)
found that on average, adolescent girls tend to score higher on measures of
motivation, self-related cognitions, and learning strategies.

Initial validity estimates using achievement indices. The relations

between the scales and high school GPA and ACT Assessment scores ap-
peared consistent with the theoretically expected pattern, thus providing
some initial evidence for the construct validity of the scales. Specifically,
Commitment to College, Social Connection, Academic Discipline, and Aca-
demic Self-Confidence were somewhat correlated with high school GPA
(range r = .20 to .32). Moreover, as theoretically expected (cf. Bandura,
1997), the Academic Self-Confidence scale was found to be somewhat corre-
lated with students ACT Assessment scores (r = .32). All correlations
reported in the text are observed correlations. Correlations between true
scores (i.e., correlations disattenuated by the effect of measurement error) are
likely to be higher.

Examining the Higher Order Factor Structure of the Scales

As described in the Method section, we used exploratory factor analysis

on the basis of the correlations among the 10 first-order factors to determine
the second-order factors. Those correlations were obtained from the confir-
498 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

matory factor analysis carried out on a new subsample (n = 3,970, with effec-
tive sample size ranging from 1,867 to 3,957). Examination of the resultant
scree plot suggested that there were 3 to 4 second-order factors underlying
the 10 first-order factors. We initially used a solution with 4 second-order
factors. On the basis of the meanings of the first-order factors that served as
indicators, the second-order factors were named Motivation (underlying the
Commitment to College, Goal Striving, Academic Discipline, and General
Determination scales), Skills (underlying the Study Skills and Communica-
tion Skills scales), Social Engagement (underlying the Social Activity and
Social Connection scales), and Self-Management (underlying the Academic
Self-Confidence and Emotional Control scales).
A second-order confirmatory factor analysis was then carried out on the
remaining holdout sample (n = 2,000, with effective sample size ranging
from 566 to 1,987). Following Viswesvaran and Oness (1995) suggestion,
we calculated the geometric mean of the sample sizes of all the bivariate cor-
relations and used the result (n = 1,060) as the sample size in our confirma-
tory factor analysis. Each scale was split into two halves to create indicators
for the first-order factors. The analysis confirmed the higher order factor
structure found in the exploratory factor analysis (fit indices: @ 2 = 695.52,
df = 154, p < .001, CFI = .927, RMSEA =.058, SRMR = .043). However, the
estimated correlation between the second-order constructs of skills and moti-
vation was quite high (r = .96). This finding suggests that the two constructs
may be the same. To explore this possibility, we carried out an additional
analysis, testing a model containing three second-order factors, with the
skills and motivation constructs combined into a single factor (henceforth
referred to as Model 2). Model 2 was nested within the model with four sec-
ond-order factors model discussed earlier (henceforth referred to as Model
1), so we could compare the fit indices of the two models to determine the
appropriateness of Model 2. In particular, we examined the difference
between the chi-square values of the two models, which itself is distributed as
a chi-square statistic with degrees of freedom equal to the difference in the
degrees of freedom of the models (Widaman, 1985). Model 2 provided a rea-
sonable fit (fit indices: @ 2 = 712.34, df = 157, p < .001, CFI = .925, RMSEA =
.058, SRMR = .044). The chi-square difference was statistically significant
(@ 2 = 16.82, df = 3, p < .001), indicating that Model 2 did not fit the data as
well as Model 1. It is worth noting that the chi-square difference statistic, just
like the chi-square statistic, is largely dependent on sample size (see Cheung
& Rensvold, 2002). With a sufficiently large sample, the chi-square statistic
provides a sensitive statistical testbut not a practical testfor model fit. In
fact, our sample size was relatively large (effective n = 1,060), which likely
inflated the small difference between the models. Recently, Yuan and Bentler
(2004) showed that the nested chi-square test is not reliable when the base
model is not correctly specified. Although we believed that our Model 1 was
 LE ET AL. 499

generally appropriate, there might have been some minor misspecifications

(e.g., unspecified correlated errors between the indicators) that affected the
result provided by the chi-square test.
On the basis of the results of simulation studies, Cheung and Rensvold
(2002) suggested using the CFI (Bentler, 1990) to compare the degree of
fit of hierarchically nested models. Specifically, these researchers recom-
mended that a difference of .01 or larger in the CFI indicates that the nested
model does not fit as well as the original model. When applying this criterion,
we found that the difference in CFI between the models was only .002, which
is considerably smaller than the suggested cutoff value. Thus, Model 2,
which is more parsimonious than Model 1, was retained.
Examining Model 2, the first-order factors indicating the motivation
second-order construct had estimated standardized path coefficients ranging
from .69 to .89 (general determination = .80, academic discipline = .80, goal
striving = .89, commitment to college = .74, study skills = .69, and communi-
cation skills = .70). For the second-order construct of social engagement, the
estimated standardized path coefficient of social activity was .71, and social
connection was .77. For the self-management second-order construct, the
estimated standardized path coefficient of academic self-confidence was
.67, and emotional control was .58. The estimated correlation between
motivation and social engagement was .64, between motivation and self-
management was .80, and between social engagement and self-management
was .59.

Potential subgroup differences. As mentioned in the Method section,

we carried out additional analyses to examine the equivalence of the factor
structure of the scales across different subgroups (men vs. women, high
school vs. community college vs. university students, and majority vs. mi-
nority). Specifically, we compared base models, which allowed parameters
to be independently estimated in each subgroup, with nested models, in
which the major parameters (i.e., second-order factor covariances, path co-
efficients from the second-order factors to the first-order factors and from
the first-order factors to the indicators) were constrained to be equal across
the subgroups. Table 5 features the results of these analyses. As can be seen,
all nested models had reasonable fit. Although the chi-square differences
were statistically significant for all the comparisons, the differences in CFI
were much smaller (from .003 to .005) than the cutoff criterion of .01 sug-
gested by Cheung and Rensvold (2002). These results indicated that the fac-
tor structure (path coefficients and covariances) was invariant across subgroups.

Discussion
The main objective of the current study was to develop an inventory of
psychosocial and skill factors that (a) captured the constructs found to be pre-
500 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Table 5
Results of Analyses Examining Factor Invariance Across Subgroups

Comparison of Fit Indexes

Base Nested
2
Subgroup Comparison Model Model @ (df)  CFI

Minority versus majority

2
@ (df) 3,786.8 (314) 3,916.7 (337) 129.9 (24), p < .01 .003
CFI .916 .913
RMSEA .062 .061
SRMR .043 .056
Men versus women
2
@ (df) 3,771.6 (314) 3,937.9 (337) 166.3 (24), p < .01 .003
CFI .911 .908
RMSEA .063 .062
SRMR .044 .049
High school versus community
college versus university
2
@ (df) 4,127.6 (471) 4,349.1 (517) 221.5 (46), p < .01 .005
CFI .911 .906
RMSEA .064 .063
SRMR .048 .055

Note. Base model: models allowing parameters to be independently estimated in each subgroup; nested model:
models constraining major parameters (i.e., second-order factor covariances, path coefficients from the
second-order factors to the first-order factors and from the first-order factors to the indicators) to be equal
across subgroups.

dictive of college success in Robbins et al.s (2004) meta-analysis, (b) cap-

tured other constructs not examined in the meta-analysis that may be predic-
tive of college success criteria, and (c) established the foundation for the
construct validation process of scores obtained from the resulting inventory.
On the basis of the study findings, we believe that these objectives were met.
In this section, we elaborate on the findings and their implications. Specifi-
cally, we discuss the first- and second-order structure of the inventory, the
relations with demographic and achievement variables, as well as the limita-
tions of the study and future research directions.

Factor Structure of the Scales and Their

Expected Criterion-Related Validities

First-order factors. Of the 10 empirically derived factors, 5 (i.e., Commit-

ment to College, Academic Self-Confidence, Social Activity, Social Con-
nection, and Emotional Control) resemble those in the original conceptual
model (i.e., Goal Focus, Academic Self-Efficacy, Social Activity, Social
Connection, and Emotional Control Skills, respectively). However, the 6th
factor, General Determination, only partially captures the original Conscien-
 LE ET AL. 501

tiousness factor, because it primarily includes items constituting the Depend-

ability subfactor of Conscientiousness. The remaining four factors appear to
be combinations of two original factors. Specifically, the obtained Study
Skills factor includes items belonging to the original Study Skills and Prob-
lem Solving Skills factors. The obtained Goal Striving factor combines items
of the original Goal Focus and Conscientiousness factors. Similarly, the ob-
tained Communication Skills factor includes items from the original Com-
munication Skills and Teamwork factors. And the obtained Academic Disci-
pline factor combines items from the original Conscientiousness and Study
Skills factors. Overall, the findings partially confirmed the factors originally
suggested in the conceptual model.
Several of the first-order factors are similar to those examined in Robbins
et al.s (2004) meta-analysis. In the motivation domain, the Commitment to
College factor appears to capture both the goal commitment and institutional
commitment constructs. Similarly, the Goal Striving factor seems to capture
the achievement motivation and goal commitment constructs. The Academic
Self-Confidence and Study Skills factors resemble the academic self-
efficacy and academic-related skills constructs, respectively, included in
Robbins et al.s meta-analysis. In the social engagement domain, the Social
Connection factor appears to represent the constructs of social support and
social involvement. Finally, the Sociability factor is highly similar to the
social involvement construct. Because comparable scales were found to pro-
vide important incremental validities in predicting college outcomes in Rob-
bins et al.s meta-analysis, we expect that the scales developed in this study
will be similarly predictive of such outcomes. This hypothesis should be
examined with future research.
The remaining four first-order factors (Academic Discipline, General
Determination, Communication Skills, and Emotional Control) appear to
represent relatively new factors not examined in Robbins et al.s (2004) meta-
analysis. It again should be noted that the scope of the meta-analysis was sub-
ject to the limitations of the literature on which it was based. As mentioned
before, on the basis of our study of the related literature, we attempted to
overcome such limitations by including items representing constructs poten-
tially predictive of the outcomes of interest. Accordingly, we expect that
future research will find that these new factors will also contribute posi-
tively to the prediction of college outcomes.

Second-order factors. We found three higher order factors underlying the

first-order factors, just as in the conceptual model. The Social Engagement
factor reflects items and scales originally suggested in the conceptual model.
However, first-order constructs forming the Motivation and Academic-
Related Skills higher order factors in the conceptual model were combined
into one single factor: Motivation and Skills. Consequently, this higher order
factor contains a range of factors that appear to tap motivation and conscien-
502 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

tiousness, as well as academic and communication skills. The other higher

order factor, Self-Management, emerged as a separate second-order factor
including Academic Self-Confidence and Emotional Control. This finding,
although initially surprising, seems consistent with recent research showing
that generalized self-efficacy, self-esteem, and emotional stability can be
grouped under the general construct of core self-evaluation (Judge, Erez,
Bono, & Thoresen, 2002). This core self-evaluation construct has been
shown to be a good predictor of job performance, turnover, and job satisfac-
tion (Judge & Bono, 2001). Together with Robbins et al.s (2004) meta-
analytic findings about the predictive abilities of academic self-efficacy for
college outcomes, the current finding suggests that the Self-Management
factor may be an important predictor of the criteria of interest.
As explained earlier, the very high correlation between the two higher
order factors Motivation and Skills led us to combine them to form the Moti-
vation and Skills factor. This finding is puzzling given that the scales consti-
tuting the original higher order factors were written to be theoretically dis-
tinct on the basis of the existing literature. In the current context, motivation
is conceptualized as psychological characteristics that drive students to
engage in the pursuit of academic-related behaviors, whereas skills are gen-
erally defined as the proficiency to perform well academically. One possible
explanation is that these factors, despite their relatively distinct research his-
tories, may be measuring the same thing. This is sometimes referred to as the
jangle fallacy (Block, 1995), in which the same construct bears different
names. From another perspective, we may be observing the incongruence
between the specificity of measurement and the motivational constructs
rather than the similarity of the motivation and skill constructs (i.e., the lack
of measurement specificity may be creating the apparent similarity).
In the past, researchers have treated the two constructs differently, and we
followed that practice when writing items to capture the constructs. Despite
conceptual distinction, some existing empirical evidence suggests that the
two constructs may be highly correlated. In their meta-analysis, Robbins
et al. (2004) found that the corrected correlations between academic-related
skills and two motivational constructs, achievement motivation and goals,
were .82 and .69, respectively. Applying the formula used for correlation of
composite scores (see Hunter & Schmidt, 1990; Nunnally & Bernstein,
1994), we estimated the correlation between academic-related skills and the
equally weighted composite of achievement motivation and goals, a proxy of
the motivation to succeed construct, may be as high as .83. This calculation
was based on the estimated corrected correlation between achievement moti-
vation and goals (r = .65), as found in Robbins et al. (2004). Because this
proxy is only a rough representation of the motivation construct, it may be
that the true correlation between the skills and motivation constructs, if prop-
erly operationalized, could be as high as the value obtained in this study (cf.
Hunter & Schmidt, 1990). Indeed, it is possible that those individuals who
 LE ET AL. 503

have the skills to succeed academically are more motivated to do well in

academic environments, which is consistent with the expectancy theory
(Vroom, 1964). Conversely, it is also possible that students who are moti-
vated sufficiently have made necessary efforts to equip themselves with the
skills needed to succeed. This is an interesting research question that requires
further examination. Thus, we believe that it is important to replicate the find-
ing in the current study and to reexamine the high magnitude of association
between the motivation and skills constructs before any firm conclusions
about the relation between the constructs can be made. On the basis of empiri-
cal evidence in the study, we have elected to retain the three second-order fac-
tor model with the Motivation and Skills factors collapsed into a single factor.

Relations With Demographic Variables

As stated in the Results section, most of the correlations between the

scales and the demographic variables of interest (age, family income, and
ethnicity) are small and, despite their statistical significance, of little practi-
cal significance. However, correlations between several scales (Commitment
to College, Teamwork, and Academic Discipline) and gender were some-
what larger in magnitude, thereby raising questions about the potentially dif-
ferential interpretations of the scales contents by men and women. This con-
cern was somewhat alleviated by our measurement equivalence analyses,
which showed that the factorial structure of the scales was invariant across
genders. However, our analyses could not provide a definitive answer to
questions such as the potentially differential predictive validity of the scales
across these groups. Such questions will be addressed when the criterion data
are available. We are in the process of collecting such data.

Limitations and Future Research

Normative data. Because of the nature of the design, participants did not
respond to all items. This limitation forced us to average available responses
for items belonging to each scale to form scale scores. Although this proce-
dure should not seriously affect our correlation-based (or covariance-based)
analyses (Zwick, 1987), the estimations of scale distributional characteristics
(i.e., mean, standard deviation, skewness, and kurtosis) may not be as accu-
rate. Accordingly, further study with a traditional, fully crossed design (i.e.,
all students responding to all items; cf. Cronbach et al., 1972) is needed to
obtain more conclusive information about the characteristics of the scales.
Moreover, a detailed sampling procedure targeting the population of interest
(i.e., community colleges, 4-year universities across different levels of selec-
tivity) should be used to obtain the normative data that are essential for the
scales evaluation and use.
504 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Criterion-related and construct validity. The main purpose of the scales

constructed in the study is to assist in identifying students who are at higher
risk for falling behind academically or dropping out. As such, validities in
predicting the college outcomes of interest are the most important criteria for
evaluating the scales. Without criterion data (i.e., data about students GPAs
and enrollment status), we could not perform the analyses to directly address
the issue. The results of the current study suggest that the scales presented in
this study may satisfactorily capture the constructs supported by the existing
literature on college outcomes. For example, in Robbins et al.s (2004) meta-
analysis, validities of the constructs examined were found to be generalizable
across different operationalizations and situations. In accordance with the
academically and professionally accepted standard pertaining to validity
generalization research (American Educational Research Association,
American Psychological Association, & National Council on Measurement
in Education, 1999; Campbell, 1990; Schmidt, Hunter, Pearlman, & Hirsh,
1985), we believe that the scales developed as part of this study will show
predictive validity for academic performance and retention. However, de-
spite our confidence, it is necessary to directly estimate the magnitudes of the
validities, as well as to collect data using other established measures to exam-
ine whether the pattern of convergent-discriminant relations with such mea-
sures is supportive of the construct validity of the current inventory. Thus,
additional research is needed to develop a more compelling construct validity
argument.

Future directions. We believe that the careful construction of the Student

Readiness Inventory is an essential first step in examining the incremental
validity of psychosocial and study skill factors when predicting college per-
formance and retention. We are currently surveying a national sample of
entering 1st-year students at both 2- and 4-year postsecondary institutions.
Because we have the students ACT Assessment records, we will be able to
control for measures of standardized achievement, high school academic
performance, and socioeconomic status before determining the role that
noncognitive factors, as measured by the Student Readiness Inventory, play
on college outcomes.
Additionally, an important question concerning the practical value of the
inventory will need to be addressed: If at-risk students are to be identified,
can educational institutions use interventions to help these students improve
their chances of success? Indeed, the literature suggests that tailored inter-
ventions can boost studentssuccess, particularly within the retention and en-
rollment management arena. For example, Hattie, Biggs, and Purdie (1996)
used meta-analytic techniques to examine 51 study skills interventions to
determine the conditions under which interventions are effective. They found
that the promotion of learner activity and contextual learning led to positive
outcomes. In another meta-analytic study of career education interventions
 LE ET AL. 505

emphasizing basic academic skills, work habits, and work values, Evans and
Burck (1992) found that the overall effect size (e.g., Cohens d) of 67 studies
was .16, producing a positive gain in academic achievement. Thus, it seems
that if at-risk students can be identified early (e.g., at the beginning of their
1st year of college), developmental interventions designed to facilitate stu-
dent success may yield significant positive outcomes. Thus, we are interested
in forming partnerships with institutions who are committed to examin-
ing the efficacy of such interventions for at-risk students identified in their 1st
year.
Finally, we are interested in exploring whether the aforementioned
noncognitive factors can influence course placement decisions for those stu-
dents who appear within the decision zone of either a more difficult or eas-
ier college course on the basis of achievement testing (e.g., the ACT Assess-
ment or COMPASS [Computer-Adaptive Placement Assessment and
Support System]). For example, when controlling for achievement test
scores, will noncognitive factors predict successful performance on a more
difficult course? That is, given two students with comparable achievement
scores, do motivational and skills, social, and self-management factors dem-
onstrate incremental validity for course placement? We hope to explore such
questions in future research.

References
American Educational Research Association, American Psychological Association, & National
Council on Measurement in Education. (1999). Standards for educational and psychological
testing. Washington, DC: Authors.
Atkinson, J. W. (1964). An introduction to motivation. Princeton, NJ: Van Nostrand Reinhold.
Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W. H. Freeman.
Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job perfor-
mance: A meta-analysis. Personnel Psychology, 44, 1-25.
Bean, J. P. (1980). Dropouts and turnover: The synthesis and test of a causal model of student at-
trition. Research in Higher Education, 12, 155-187.
Bean, J. P. (1985). Interaction effects based on class level in an explanatory model of college stu-
dent dropout syndrome. American Educational Research Journal, 22, 35-64.
Bentler, P. M.(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107,
238-246.
Block, J. (1995). A contrarian view of the five-factor approach to personality description. Psy-
chological Bulletin, 117, 187-215.
Brown, N., & Cross, E. J. (1993). Retention in engineering and personality. Educational and Psy-
chological Measurement, 53, 661-671.
Budaev, S.V. (1999). Sex differences in the Big Five personality factors: Testing an evolutionary
hypothesis. Personality and Individual Differences, 26, 801-813.
Byrne, B. M., Shavelson, R. J., & Muthen, B. (1989). Testing for the equivalence of factor
covariance and mean structures: The issue of partial measurement invariance. Psychological
Bulletin, 105, 456-466.
Caligiuri, P. M. (2000). The Big Five personality characteristics as predictors of expatriates de-
sire to terminate the assignment and supervisor-rated performance. Personnel Psychology,
53, 67-88.
506 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and orga-
nizational psychology. In M. V. Dunnette & L. M. Hough (Eds), Handbook of industrial and
organizational psychology (Vol. 1, 2nd ed., pp. 687-732). Palo Alto, CA: Consulting Psy-
chologists Press.
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research,
1, 245-276.
Chamorro-Premuzic, T., & Furnham, A. (2003). Personality predicts academic performance:
Evidence from two longitudinal university samples. Journal of Research in Personality, 37,
319-338.
Charbonneau, D., & Nicol, A. M. (2002). Emotional intelligence and prosocial behaviors in ado-
lescents. Psychological Reports, 90, 361-370.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing mea-
surement invariance. Structural Equation Modeling, 9, 233-255.
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale develop-
ment. Psychological Assessment, 7, 309-319.
Covington, M. (2000). Goal theory, motivation, and school achievement: An integrative review.
Annual Review of Psychology, 51, 171-200.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behav-
ioral measurements: Theory of generalizability for scores and profiles. New York: John
Wiley.
Digman, J. M. (1990). Personality structure: Emergence of the five-factor model. Annual Review
of Psychology, 41, 417-440.
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of
Psychology, 53, 109-132.
Evans, J. H., & Burck, H. D. (1992). The effects of career education interventions on academic
achievement: A meta-analysis. Journal of Counseling & Development, 71, 63-68.
Farsides, T., & Woodfield, R. (2003). Individual differences and undergraduate academic suc-
cess: The roles of personality, intelligence, and application. Personality and Individual Dif-
ferences, 34, 1225-1243.
Feingold, A. (1994). Gender differences in personality: A meta-analysis. Psychological Bulletin,
116, 429-456.
Gerbing, D. W., & Hamilton, J. G. (1996). Viability of exploratory factor analysis as a precursor
to confirmatory factor analysis. Structural Equation Modeling, 3, 62-72.
Goldberg, L. R. (1993). The structure of phenotypic personality traits. American Psychologist,
48, 26-34.
Gorsuch, R. L. (1997). Exploratory factor analysis: Its role in item analysis. Journal of Personal-
ity Assessment, 68, 532-560.
Green, P. E., & Rao, V. R. (1970). Rating scales and information recoveryHow many scales
and response categories to use? Journal of Marketing, 34, 33-39.
Gullone, E., & Moore, S. (2000). Adolescent risk-taking and the five factor model of personality.
Journal of Adolescence, 23, 393-407.
Guthrie, J. P., Ash, R. A., & Stevens, C. D. (2003). Are women better than men? Personality
differences and expatriate selection. Journal of Managerial Psychology, 18, 229-243.
Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learn-
ing: A meta-analysis. Review of Educational Research, 66, 99-136.
Hezlett, S., Kuncel, N., Vey, A., Ahart, A., Ones, D., Campbell, J., et al. (2001, March). The effec-
tiveness of the SAT in predicting success early and late in college: A comprehensive meta-
analysis. Paper presented at the annual meeting of the National Council of Measurement in
Education, Seattle, WA.
Hong, E., & Milgram, R. M. (1999). Preferred and actual homework style: A cross-cultural ex-
amination. Educational Research, 41, 251-265.
 LE ET AL. 507

Hough, L. M. (1992). The Big Five personality variables-construct confusion: Description ver-
sus prediction. Human Performance, 5, 139-155.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in re-
search findings. Newbury Park, CA: Sage.
Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The Big Five revisited.
Journal of Applied Psychology, 85, 869-879.
John, O. P. (1990). The Big Five factor taxonomy: Dimensions of personality in the natural lan-
guage and in questionnaires. In L. A. Pervin (Ed.), Handbook of personality: Theory and re-
search (pp. 66-100). New York: Guilford.
Jreskog, K. G., & Srbom, D. (1999). LISREL 8.30 and PRELIS 2.30. Chicago: Scientific Soft-
ware International.
Judge, T. A., & Bono, J. E. (2001). Relationship of core self-evaluations traitsself-esteem, gen-
eralized self-efficacy, locus of control, and emotional stabilitywith job satisfaction and job
performance: A meta-analysis. Journal of Applied Psychology, 86, 80-92.
Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2002). Are measures of self-esteem,
neuroticism, locus of control, and generalized self-efficacy indicators of a common core con-
struct? Journal of Personality and Social Psychology, 83, 693-710.
Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Re-
ports, 3, 635-694.
Malloch, D. C., & Michael, W. B. (1981). Predicting student grade point average at a community
college from scholastic aptitude tests and from measures representing three constructs in
Vrooms expectancy theory model of motivation. Educational and Psychological Measure-
ment, 41, 1127-1135.
Matell, M. S., & Jacoby, J. (1972). Is there an optimal number of alternatives for Likert-scale
items? Effects on testing time and scale properties. Journal of Applied Psychology, 56, 506-
509.
McCelland, D. C. (1980). Motive dispositions: The merit of operant and respondent measures.
Review of Personality and Social Psychology, 1, 10-41.
Mount, M. K., & Barrick, M. R. (1995). The Big Five personality dimensions: Implications for
research and practice in human resource management. Research in Personnel & Human Re-
sources Management, 13, 153-200.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-
Hill.
Okun, M. A., & Finch, J. F. (1998). The big-five personality dimensions and the process of insti-
tutional departure. Contemporary Educational Psychology, 23, 233-256.
Organisation for Economic Co-operation and Development. (2003). Education at a glance:
OECD indicators. Paris, France: Author.
Pascarella, E., & Terenzini, P. (1991). How college affects students: Findings and insights from
twenty years of research. San Francisco: Jossey-Bass.
Paunonen, S. V., & Ashton, M. C. (2001). Big Five predictors of academic achievement. Journal
of Research in Personality, 35, 78-90.
Pintrich, P. R. (1989). The dynamic interplay of student motivation and cognition in the college
classroom. In C. Ames & M. Maehr (Eds.), Advances in motivation and achievement: Moti-
vation-enhancing environments (Vol. 6, pp. 117-160). Greenwich, CT: JAI.
Pintrich, P. R. (2000). An achievement goal perspective on issues in motivation terminology,
theory, and research. Contemporary Educational Psychology, 25, 92-104.
Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of
classroom academic performance. Journal of Educational Psychology, 82, 33-40.
 508 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

Pintrich, P. R., Smith, D.A.F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive
validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and
Psychological Measurement, 53, 801-813.
Rindskopf, D., & Rose, T. (1988). Some theory and applications of confirmatory second-order
factor analysis. Multivariate Behavioral Research, 23, 51-67.
Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do
psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychologi-
cal Bulletin, 130, 261-288.
Saldago, J. (1997). The five factor model of personality and job performance in the European
Community. Journal of Applied Psychology, 82, 30-43.
Saldago, J. (2002). The Big Five personality dimensions and counterproductive behaviors. Inter-
national Journal of Selection & Assessment, 10, 117-125.
SAS Institute. (1999). The SAS system for Windows. Cary, NC: Author.
Schmidt, F. L., Hunter, J. E., Pearlman, K., & Hirsh, H. R. (1985). Forty questions about validity
generalization and meta-analysis. Personnel Psychology, 38, 697-798.
Schunk, D. H., & Zimmerman, B. J. (2003). Self-regulation and learning. In W. M. Reynolds &
G. E. Miller (Eds.), Handbook of psychology, Vol. 7: Educational psychology (pp. 59-78).
New York: John Wiley.
Slotte, V., Lonka, K., & Lindblom-Ylaenne, S. (2001). Study-strategy use in learning from text.
Does gender make any difference? Instructional Science, 29, 255-272.
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job per-
formance: A meta-analytic review. Personnel Psychology, 44, 703-742.
Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of
scores: A historical overview and some guidelines. Educational and Psychological Mea-
surement, 56, 197-208.
Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Re-
view of Educational Research, 45, 89-125.
Tinto, V. (1993). Leaving college: Rethinking the cause and cures of student attrition (2nd ed.).
Chicago: University of Chicago Press.
Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance
literature: Suggestions, practices, and recommendations for organizational research. Orga-
nizational Research Methods, 3, 4-70.
Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Combining psychometric meta-analysis
and structural equations modeling. Personnel Psychology, 48, 865-885.
Vroom, V. H. (1964). Work and motivation. New York: John Wiley.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-
multimethod data. Applied Psychological Measurement, 9, 1-26.
Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Con-
temporary Educational Psychology, 25, 68-81.
Wolfe, R. N., & Johnson, S. D. (1995). Personality as a predictor of college performance. Educa-
tional and Psychological Measurement, 55, 177-185.
Yuan, K. H., & Bentler, P. M. (2004). On chi-square difference and z tests in mean and covariance
structure analysis when the base model is misspecified. Educational and Psychological
Measurement, 64, 737-757.
Zimmerman, B. J. (1986). Development of self-regulated learning: Which are the key subpro-
cesses? Contemporary Educational Psychology, 16, 307-313.
Zimmerman, B. J., & Martinez-Pons, M. (1986). Development of a structured interview for
assessing student use of self-regulated learning strategies. American Educational Research
Journal, 23, 614-628.
Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational
Measurement, 24, 293-308.

View publication stats