Trusting Teachers' Judgments: A Validity Study of a

Curriculum-Embedded Performance Assessment in

Kindergarten to Grade 3

Issue/Topic: Assessment; Teaching Quality--Preparation Author(s): Meisels, Samuel; Xue,

Yange; Nicholson, Julie; DiPrima Bickel, Donna; Atkins-Burnett, Sally Publication: American
Educational Research Journal Season: Spring 2001

Background:
Assessments that rely on teacher judgments of students' academic performance are used widely
in research and classroom settings. Curriculum-embedded performance assessment is heavily
reliant on such judgments. However, the promises of such assessments have not always been
realized and have rarely been documented empirically. What is the relationship of curriculum-
embedded performance assessment to other key indicators of student achievement?

Purpose:
To determine if teacher judgments about student learning in K-3 are accurate when they are
informed by a curriculum-embedded performance assessment.

Findings/Results:

Key Finding

Are teachers' judgments about how well students are learning accurate when those judgments are
based on curriculum-embedded performance assessments such as the Work Sampling System
(WSS)? Yes, it appears that they are, particularly in the domains of literacy and mathematical
thinking in grades K-3.

Other Findings:

 Overall, the Work Sampling System (WSS), a curriculum-embedded performance

assessment, is a reliable and valid predictor of achievement ratings in Kindergarten - 3rd
grade, particularly in literacy.

 Significant associations between the curriculum-embedded performance assessment

ratings and the standardized assessment outcomes remain for 3rd graders even after
controlling for the potential effects of age, socio-economic status, ethnicity and students'
initial performance level on the standardized assessment in literacy and math.

 This performance assessment system accurately discriminates between children who are
at risk learners and those who are not, which could help strategically appropriate Title I
federal funds.

Policy Implications/Recommendations:
  High-quality, curriculum-embedded performance assessments (also known as authentic
assessments) appear to have merit for use in classrooms and can complement conventional
accountability systems that focus almost exclusively on norm-referenced data obtained from on-
demand testing situations.

 The Work Sampling System (WSS) is not intended to be used for high stakes purposes (such as
promotion or retention) and may lost its effectiveness when so applied.

 Evaluation of the reliability and validity of authentic assessments in comparison to standards-

based exams for measuring student performance, learning growth and effectiveness of a
curriculum is critical.

For full study: http://www.ciera.org/library/archive/2001-09/2001-09.htm

Research Design:
Cross-sectional, psychometric investigation

Population/Participants/Subjects:
A cross-sectional sample of 345 K-3 students enrolled in 17 classrooms in five schools in
Pittsburgh, PA.

Year data is from:

1996-97

Setting:
School

Data Collection and Analysis:

Data analysis included a) correlations between Work Sampling System (WSS) results and
students' Woodcock-Johnson-Revised (WJ-R) standard scores; b) four-step hierarchical
regressions to examine the variance in students' spring outcome scores, c) and Receiver-
Operating-Characteristics (ROC) curves to compare the accuracy of WSS in categorizing
students as at-risk learners.