Implementing Data-Informed Decision

Making in SchoolsTeacher Access,

Supports and Use
U.S. Department of Education
Office of Planning, Evaluation and Policy Development
Prepared by
Barbara Means
Cristine Padilla
!ngela DeBarger
Marianne Ba"ia
S#$ $nternational
Menlo Par", California
%&&'
This report was prepared for the U.S. Department of Education under Contract number ED-01-
CO-0040 Task 0002 with SRI International. Bernadette Adams Yates served as the project
manager. The views expressed herein do not necessarily represent the positions or policies of the
Department of Education. No official endorsement by the U.S. Department of Education is
intended or should be inferred.
U.S. Department of Education
Margaret Spellings
Secretary
Office of Planning, Evaluation and Policy Development
Bill Evers
Assistant Secretary
Policy and Program Studies Service Office of Educational Technology
!lan (insburg )imoty *. Magner
Director Director
Program and Analytic Studies Division
David (ood+in
Director
January 2009
)is report is in te public domain. !utori,ation to reproduce tis report in +ole or in part is granted.
-ile permission to reprint tis publication is not necessary, te suggested citation is. U.S. Department of
Education, Office of Planning, Evaluation and Policy Development, Implementing Data-Informed
Decision Making in Schools: Teacher Access, Supports and Use, -asington, D.C., %&&'.
)is report is available on te Department/s -eb site at
+++.ed.gov0about0offices0list0opepd0ppss0reports.tml.
On request, this publication is available in alternate formats, such as Braille, large print, or computer
diskette. For more information, please contact the Departments Alternate Format Center at
(202) 260-0852 or (202) 260-0818.
Contents
List of Exhibits..........................................................................................................................iv
Acknowledgments......................................................................................................................v
Executive Summary.................................................................................................................vii
Contents..................................................................................................................................... iii
Exhibits ......................................................................................................................................iv
Acknowledgments...................................................................................................................... vi
Executive Summary.................................................................................................................. vii
Early Findings.................................................................................................................... viii
Data Systems Available to Local Educators.................................................................... viii
Supports for Data-Informed Decision Making....................................................................x
How Schools Are Using Data........................................................................................... xii
1. Introduction and Approach......................................................................................................1
Conceptual Framework of Data-Informed Decision-Making Process and Supports............1
The Process of Using Data for Improvement...................................................................... 2
Supports for the Process of Using Data for Improvement.................................................. 3
Data Sources for the Interim Report......................................................................................6
Contents of This Report........................................................................................................ 9
2. School Data Practices............................................................................................................ 11
District Data Systems.......................................................................................................... 11
Teacher Access to Student Data Systems..........................................................................13
Types of Data Available to Teachers................................................................................ 14
Social Context for Data Use.............................................................................................. 17
Barriers to Use of Data Systems to Inform Instruction....................................................... 18
3. District and School Supports for Data-Informed Decision Making...................................... 23
Data System Integration...................................................................................................... 23
District and School Leadership for Data Use...................................................................... 23
Tools for Generating and Organizing Data......................................................................... 26
Organizational Structures Supporting Data Use at the School Level................................27
Professional Development and Technical Support............................................................29
Tools for Acting on Data..................................................................................................... 31
Summary of District and School Supports.......................................................................... 32
4. Teachers Preparation for Data Use...................................................................................... 37
iii
Development of the Data Scenarios.................................................................................... 37
Scenario Administration...................................................................................................... 40
Measures............................................................................................................................41
Procedures......................................................................................................................... 41
Preliminary Results and Findings........................................................................................42
Comparisons by District.................................................................................................... 43
Remaining Analyses.......................................................................................................... 43
Summary ............................................................................................................................ 44
5. Summary of the Early Findings.............................................................................................45
Progress in the Use of Data Systems................................................................................... 46
Lessons Learned for Implementation.................................................................................. 47
References................................................................................................................................. 51
NETTS Survey Data............................................................................................................58
Data Scenario Development................................................................................................ 89
Reliability and Scores for Data Scenario Questions........................................................... 92
Appendix A: Case Study and Survey Data Sources................................................................ 53
Appendix B: Scenario, Exhibits, Items, Rubrics, Item Score Distribution and
Mean Scores........................................................................................................61
Appendix C: Scenario Development and Reliability Information...........................................87
Ehi!its
Exhibit 1: Data Available to Teachers...................................................................................... ix
Exhibit 2: Percentage of Correct Responses, by Item Type Across Schools........................ xiii
Exhibit 1-1: Conceptual Framework for Data-Informed Decision Making...............................2
Exhibit 1-2: 200607 Case Study Districts................................................................................8
Exhibit 2-1: Data Available to Teachers in 200607...............................................................15
Exhibit 2-2: Teachers Use of Student Data............................................................................ 16
Exhibit 2-3: Teacher Experiences Using Electronic Student Data Systems in 200607.........19
Exhibit 3-1: A District and School Data Analysis Process......................................................24
Exhibit 3-2: Principal as Data Leader...................................................................................... 25
Exhibit 3-3: Activities of School-Based Support Staff Members............................................28
Exhibit 3-4: Topics for Additional Teacher Professional Development................................. 30
Exhibit 3-5: Professional Development Online and In Person................................................ 31
Exhibit 3-6: Case Study District Supports for School Use of Student Data............................ 34
Exhibit 3-7: Case Study School Supports for Use of Student Data......................................... 35
Exhibit 4-1: Data-Informed Decision-Making Components, Concepts and Skills..................39
iv
Exhibit 4-2: Examples of Items Related to Each Component................................................. 40
Exhibit 4-3: Frequency Distribution and Mean Percentage Correct for Scored
Items Overall and Within Each Data Component...............................................42
Exhibit 4-4: Total Average Scores for Schools, by District.................................................... 43
Exhibit A-1: Definitions of Terms........................................................................................... 55
Exhibit A-2: Data Available From the State Data System for Each Case Study
District in 2007................................................................................................... 56
Exhibit A-3: Case Study Data Systems................................................................................... 57
v
Ackno"ledgments
Many individuals contributed to the completion of this interim report. We are particularly
grateful to the district and school-level staff members who took time out of their busy schedules
to respond to our requests for information; in particular, the 9 districts and 18 case study schools
were generous with both their time and attention to this evaluation work. Without their efforts,
this report would not have been possible, and we deeply appreciate their assistance.
We would like to acknowledge the thoughtful contributions of the members of our
Technical Work Group in reviewing study materials and prioritizing issues to investigate. The
advisors consisted of Katherine Conoly of Corpus Christi ISD, Marty Daybell of Washington
Schools Information Processing, Aimee Guidera of the National Center for Educational
Accountability, Glynn Ligon of ESP Solutions, Ellen Mandinach of CNA Corporation, Jim
Pellegino of the University of Illinois-Chicago, Arie van der Ploeg of Learning Point Associates,
and Jeff Wayman at the University of Texas at Austin. We would also like to acknowledge the
contributions by our consultant Jere Confrey at North Carolina State University and Geneva
Haertel at SRI.
Many U.S. Department of Education staff members contributed to the completion of this
report. Bernadette Adams Yates served as project manager and provided valuable substantive
guidance and support throughout the design, implementation and reporting phases of this study.
We would also like to acknowledge the assistance of other Department staff members in
reviewing this report and providing useful comments and suggestions, including David
Goodwin, Daphne Kaplan, Tim Magner and Zeev Wurman.
We appreciate the assistance and support of all of the above individuals. Any errors in
judgment or fact are of course the responsibility of the authors.
The Study of Education Data Systems and Decision Making is supported by a large
project team at SRI. Among the staff members who contributed to the research were Sara
Carriere, Eva Chen, Larry Gallagher, Marilyn Gillespie, Torie Gorges, Ann House, Harold
Javitz, Aasha Joshi, Carlin Llorente, Nicolette Mayes, Natalie Nielsen, Christina Park, Angeline
Reyes, Corinne Singleton, Tina Stanford, Willow Sussex, Edith Yang and Viki Young. Layout
and editing were performed by Eileen Behr and Klaus Krause.
vi

Eecuti#e Summar$
)e collection, analysis and use of educational data are central to te improvement of
student outcomes envisioned by No hild !eft "ehind 1N!"2. )e use of data in educational
decision ma"ing is e3pected to span all layers of te education system4from te federal to te
state, district, scool and classroom levels. )e implementation of te 5C6B legislation as been
accompanied by a demand for data systems capable of providing a longitudinal record of eac
student/s educational e3periences and performance over time.
)e national Study of Education Data Systems and Decision Ma"ing, sponsored by te
U.S. Department of Education/s Office of Planning, Evaluation and Policy Development, is
documenting te availability of education data systems, teir caracteristics, and te prevalence
and nature of data7informed decision ma"ing in districts and scools.
8
)e study is e3amining
bot te implementation of student data systems per se and te broader set of practices involving
te use of data to improve instruction, regardless of +eter or not te data are stored in and
accessed troug an electronic system.
)e conceptual frame+or" developed for te study identifies si3 prere9uisites and supports
for data7informed decision ma"ing. 1a2 state, district and scool data systems: 1b2 leadersip for
educational improvement and te use of data: 1c2 tools for generating actionable data: 1d2 social
structures and time set aside for analy,ing and interpreting data: 1e2 professional development
and tecnical support for data interpretation: and 1f2 tools for acting on data.
)is conceptual frame+or" inspired te study/s researc 9uestions.
-at "inds of systems are available to support district and scool data7driven decision
ma"ing; -itin tese systems, o+ prevalent are tools for generating and acting on
data;
<o+ prevalent are organi,ational supports for scool use of data systems to inform
instruction;
<o+ are scool staff members using data systems; Do tey "no+ o+ to interpret
student data; <o+ is scool staff members/ use of data systems and of data more
broadly influencing instruction;
)is interim report describes findings from multiple sources. $t dra+s on case study
findings from te first round of site visits in nine purposively sampled districts, nominated on te
basis of te strengt of teir data use activities. #esearcers intervie+ed district staff members as
+ell as principals and teacers from tree scools +itin eac district. $n addition, a set of
scenarios involving ypotetical student data +ere presented to teacers at eac scool to probe
teir understanding of student data.
%
)is report dra+s also on data from secondary sources
8
)e term data-informed decision making is used trougout tis report in preference to te more common term
data-dri#en decision making in recognition of te fact tat fe+ decisions are based +olly on 9uantitative data.
%
$nformation on o+ districts, scools and respondents +ere selected for participation in te case study sample
can be found on pages =>' of tis report: additional information on te 5ational Educational )ecnology )rends
Study 15E))S2 survey sample can be found on page = and in !ppendi3 !.
vii
1spring %&&? district and teacer surveys from te U.S. Department of Education/s 5ational
Educational )ecnology )rends Study2. Using tese sources, te report describes te student
data systems available to scool staff members, o+ scool staff members are using te systems
and oter forms of student data, teacers/ understanding of data displays and data interpretation
issues, and te supports and callenges for scool7level use of student data in planning and
implementing instruction.
Earl$ %indings
The burgeoning activity around student data systems over the last five years is being
felt at the district and school levels. )e dramatic increase in teacer access to student data
systems as been documented in te t+o 5E))S teacer surveys. $n case study scools and
districts, teacers +ere receiving professional development around data use and +ere using data
reports from systems for scool improvement. Scool teams +or"ing on particular initiatives,
suc as #eading @irst, +ere using student data to guide teir actions.
Data from student data systems are being used in school improvement efforts but are
having little effect on teachers daily instructional decisions as evidenced in case study
districts. Despite progress in giving teacers access to student data, it is clear tat in many
districts, te use of locally generated data to inform instruction is an activity separate from use of
data systems containing student scores on standardi,ed tests. District and scool uses of data
systems to store, organi,e and report standardi,ed test scores typically focus on accountability
concerns and on efforts to ensure tat local curriculum and instruction are +ell aligned +it state
assessments. 5eiter te type of assessment for +ic data are available nor te time frame of
assessment activities serves te needs of classroom teacers ma"ing decisions on a daily basis.
Case study scools did offer evidence tat teacers and teacer teams +ere using data to guide
classroom instruction, but tese data generally came from assessments closely aligned +it local
instruction, and te data +ere typically not stored on te student data system containing state
assessment scores. )e integration of classroom and state assessment data in te same electronic
system is not common, even in case study districts noted for teir data systems and data7using
culture.
More specific interim findings address the issues of data system availability, supports
provided for data-informed decision making, schools use of data and teachers preparation for
using data to inform instruction.
Data S$stems A#aila!le to &ocal Educators
!ltoug teacer7reported access to student data systems is gro+ing rapidly, te data
systems temselves are currently not supporting instructional decision ma"ing at te scool
level. District data systems often cannot sare data across systems, are not user friendly, contain
limited data, and lac" instructional tools to elp teacers act on te data provided to tem.
Teacher!reported access to student data systems gre" significantly, from #$ percent
in %&&' to (# percent in %&&(. )is gro+t in teacer access +as estimated using national
estimates based on responses of teacers to te 5ational Educational )ecnology )rends Study
15E))S2 teacer survey.
viii
Even though teacher access to data systems is gro"ing rapidly, systems often lac) the
)inds of data that teachers find most useful for instructional decision ma)ing. !mong
teacers +o said tey ad access to a student data system 1E3ibit 82, te data most fre9uently
available to tese teacers +ere student attendance data 1?A percent2 and grades 1=? percent2.
Only BB percent of te teacers +it access to a student data system 1or A8 percent of all
teacers2 ad access to teir current students/ performance on bencmar" or diagnostic tests.
-ayman 1%&&B2 describes four common types of educational data systems. 1a2 student
information systems providing real7time accounting for scool functions suc as attendance, 1b2
assessment systems supporting te rapid organi,ation and analysis of a +ide array of assessment
data, 1c2 data +areouses tat lin" multiple databases to provide access to istorical data of all
types, and 1d2 instructional0curriculum management systems. )e 5E))S teacer survey data
suggest tat teacers are most li"ely to ave access to te first of tese system types. E3ibit 8
summari,es te survey findings.
Ehi!it '
Data A#aila!le to Teachers
T$pes of Data A#aila!le
(ercentage of
Teachers )ith
S$stem Access
(ercentage of All
Teachers
Attendance data 74 55
Student grade data 67 50
Performance on benchmark or diagnostic tests
taken by your current students
55 41
Spring 2006 standardized test scores for the
students you taught ast year !2005"06#
46 $4
Spring 2006 standardized test scores for your
current students
44 $$
%a 2006 standardized test scores for your
current students
$$ 25
&utipe years of standardized test scores for
indi'idua students
2( 11
)ourse enroment histories for students 22 16
Students* prior schoo!s# attended 16 12
Students* participation in suppementary
education programs !e+g+, tutoring#
- 6
E3ibit reads. !mong teacers +o ad access to an electronic student data system in %&&=>&?, ?A
percent reported tat tey ad access to attendance data troug te data system, +ic represents BB
percent of all teacers 1+it or +itout access to a data system2.
Source. %&&=>&? 5E))S teacer survey.
!t one point, proponents of data7informed decision ma"ing in education offered te vision
of a single integrated system combining multiple sources of information. !s states and districts
ave gained e3perience +it data systems, most e3perts ave come to agree tat it is more
practical to strive for interoperability among different education data systems so users can move
data bet+een systems or combine data from different systems easily rater tan try to build a
i3
single education data system tat +ould serve all purposes 1Datno+, Par" and -olstetter %&&?:
-ayman %&&B2. *nteroperability among data systems "as an issue "ithin the case study
districts. Staff members at the nine districts in the spring 2007 case study sample were all using
two or more separate systems with limited interoperability. None had a fully interoperable set of
data systems. Typically, the information needed for educational decision making was spread
across multiple systems without mechanisms for regular transport of information from one
system to another. !s a result, neiter teacers nor administrators see a compreensive record of
students/ educational e3periences and performances tat is bot longitudinal and up to date.
Barriers to data system use tat teacers cited on te 5E))S survey suggest tat a
significant proportion of teacers +it access to an electronic student data system ave difficulty
getting te "inds of data tey +ant to see out of te data system.
C
)eacers reported feeling
ampered by teir inability to use data 9ueries to get pertinent data from teir data systems and
by te limited utility of te "inds of information available in te student data systems for
deciding +at and o+ to teac 1eac reported by %' percent2. Smaller but significant
proportions of teacers reported tat tey ad trouble finding te information tey +ere loo"ing
for on te system 1%A percent2 and tat te system +as ard to use 1%& percent2. !mong scool
staff members intervie+ed as part of te case study researc, te tree most commonly cited
barriers +ere 1a2 lac" of training in o+ to use te data system or to derive instructional
implications from system data, 1b2 te lac" of time to engage in data e3ploration and reflection,
and 1c2 te +ea"ness of te available data.
Tools in data systems to help teachers improve decisions about instructional practice
are not the norm, even in case study districts identified as high data users. Data from te
case studies suggest tat only a minority of data systems incorporate resources suc as
instructional materials, model lesson plans, and formative assessment results lin"ed to
frame+or"s and curriculum guides. Only tree case study districts ad data systems tat
provided suc resources. One additional district +as developing a system +it tis capacity.
Supports for Data-Informed Decision Making
Effective data7informed decision ma"ing re9uires not only access to useful data but also
+ell7designed supports suc as leadersip to model data use and supported time for reflection on
data. !ltoug districts are beginning to provide supports suc as training and support staff for
data7informed decision ma"ing, tere is a great deal of variability in te dept and breadt of
tese supports, even among districts identified as model data users. @ederal programs ave
played an important role in elping teacers e3amine data and apply tis information to
instructional practice.
+ase study districts demonstrated their support and leadership for schools use of
data through purchasing systems, modeling data use, and providing school!based support
positions. More specifically, district supports for scools/ use of data included 1a2 ma"ing data7
informed decision ma"ing a priority and purcasing or developing a data system: 1b2 considering
ability to use data as a criterion for teacer iring: 1c2 providing training and support positions
for system implementation: 1d2 initiating district+ide, data7informed decision7ma"ing activities
C
!ll percentages reported for teacer survey respondents in te remainder of te e3ecutive summary are for
teacers +o indicated tey ad access to an electronic student data system in %&&=7&? unless oter+ise indicated.
3
in addition to professional development: and 1e2 using te data system for decision ma"ing at te
district level 1e.g., evaluating programs, principals, teacers2.
A number of federal programs have supported schools use of data to inform
instruction. Staff members at case study scools described te use of funds from )itle $,
Enancing Education )roug )ecnology 1EE))2 and #eading @irst to support scool7based
staff members +o elp teacers e3amine data and dra+ implications for instruction. #eading
@irst appeared to be particularly effective in supporting data use because #eading @irst coaces
could provide teacers +it different instructional approaces appropriate for students +it
various patterns of assessment results.
,eadership for data!informed decision ma)ing at the school level can e-tend beyond
the principal. Previous case studies of data7informed decision ma"ing in scools ave stressed
te importance of leadersip by te principal 1e.g., setting te conte3t for using student data,
providing te time for reflection on data, and communicating e3pectations for teacers/ data
use2. !ltoug tese functions are important, te case studies suggest tat tey may be
performed by individuals in a variety of Dob roles: instructional coaces, department lead
teacers, and instruction and assessment coaces +ere providing leadersip for using data in
many of te case study scools.
Districts appear to be responding to the need for professional development for
teachers that focuses on data analysis s)ills or data!informed decision ma)ing more
generally. On te %&&=>&? 5E))S teacer survey, C' percent of teacers reported tat te
professional development tey received about data7informed decision ma"ing ad prepared tem
to use data to improve student acievement. !t te same time, most teacers reported tat tey
+ould benefit from additional professional development on data7informed decision ma"ing. !ll
case study districts provided some form of district+ide professional development, but it varied in
terms of 1a2 +o received te training and 1b2 te training/s content, duration and format.
Organi.ational structures that support data use at the school level can include
designated time for teachers to revie" and discuss data in small groups, assigned support
staff, and the adoption of procedures for discussing data. On te %&&=>&? 5E))S teacer
survey, most teacers reported aving positive perceptions of support for using and interpreting
data tat tey could access from data systems. @or e3ample, a maDority of teacers +it access to
a data system 1?8 percent2 agreed or agreed strongly tat +en tey needed elp ma"ing sense of
te data in district systems, tey "ne+ someone +o could elp. Case study districts provided
even stronger support for data7informed decision ma"ing tan te typical level indicated in te
5E))S teacer survey responses. Si3 out of nine offered some sort of district7funded, scool7
based staff to support teacers/ data use. $n some scools and districts, individuals in tese
positions e3tracted relevant data from te system and guided teacers in dra+ing inferences for
action. $n oter cases, te coaces/ role involved elping teacers learn to analy,e data
temselves, +it coaces motivating teachers to examine data and providing just-in-time
informal professional development for data interpretation and connecting data to instructional
strategies.
3i
*o" Schools Are Using Data
! significant barrier to implementing data7informed decision ma"ing is a lac" of e3pertise
among scool staff members in te area of data analysis. Districts and scools are addressing tis
callenge by supporting collaborative activities to discuss student data, but more training is
re9uired troug bot pre7service and in7service programs to provide teacers +it a full range
of data literacy s"ills.
/ost teachers "ho use a student data system do so not only on their o"n but also in
collaboration "ith colleagues. !mong 5E))S teacer survey respondents, te most commonly
reported conte3t for using a student data system +as on one/s o+n 1?E percent2, follo+ed closely
by data use +it colleagues in a department or grade 1?8 percent2. $n te case study districts, te
most fre9uently cited groupings for data use +ere te grade7level team, sometimes facilitated by
a coac, and all7staff faculty meetings.
Both teachers and district staff members in the case study sample express concerns
about teachers ability to understand data. The majority of NETTS teacher survey
respondents also reported that they would benefit from additional professional development on
data-informed decision making. To better assess teachers understanding of data, project staff
incorporated data scenarios into the case studies, which were presented individually to several
teacers at eac scool. )e scenarios consisted of ypotetical sets of classroom, scool and
district data as +ell as a set of 9uestions about te data and its implications for practice. )ese
data scenarios +ere administered to a sample of teacers in eac case study scool to elicit teir
tin"ing about data and reveal te concepts and s"ills tat teacers can bring to data7informed
decision ma"ing. !n e3pert panel identified te necessary data s"ills and concepts, +ic +ere
clustered +itin five aspects of data7informed decision ma"ing. Fuestion Posing, Data 6ocation,
Data Compreension, Data $nterpretation, and Data Use 1ma"ing instructional decisions based
on data2.
Teachers responses to scenarios concerning hypothetical student data suggest that
teachers can locate the data they "ant "ithin comple- tables or graphs but often lac) other
data literacy s)ills. E3ibit % presents te average proportion of teacers earning full credit for
teir responses to data scenario items of various types.
Data 6ocation items, +ic re9uired te literal reading of a table or grap, +ere easy for
most teacers. )eacers ad some difficulty +it Data Compreension items suc as tose
involving multiple data points for bot scool and district performance over time or +ere
proportion and absolute value ad to be distinguised. Data $nterpretation items, +ic involved
dra+ing inferences from data patterns, also proved difficult. )ese findings ave implications for
in7service and pre7service teacer education programs.
3ii
Ehi!it +
(ercentage of Correct ,esponses, !$ Item T$pe Across Schools
Skill (ercent Correct Sample Item
.ata /ocation -1
0hat 1as 2ak Schoo*s a'erage
3ota &ath Score in 200$"044
.ata
)omprehension
64
2ak Schoo*s progress in
narro1ing the 5rade 4 math
achie'ement gap 1ith the rest of
the district has been in probem
so'ing rather than computation+
!3eacher must agree or disagree
and e6pain reasoning+#
.ata
7nterpretation
4-
3his year a African American
students shoud get more
intensi'e instruction in
mathematics. !3eacher must
agree or disagree and e6pain
reasoning+#
E3ibit reads. @or te Data 6ocation items, te average percentage of teacers in a scool earning
full credit on te item +as E8 percent.
5ote. Fuestion Posing and Data Use categories are not included in te table because tey +ere
represented by only one scored item eac 1percentage correct of te related items +ere B= percent
and E? percent, respectively2.
Source. %&&=>&? data scenarios.
3iii
'- Introduction and Approach
Over te past si3 years, meeting te data re9uirements of 5C6B and adapting or ac9uiring
database systems capable of generating te re9uired student data reports ave consumed muc of
te attention of district and state assessment and tecnology offices. )is +or" as been
necessary but insufficient for data7informed decision ma"ing to influence education. Data7
informed educational decision ma"ing consists of muc more tan Dust a data system. it includes
a set of e3pectations and practices around te ongoing e3amination of data to ascertain te
effectiveness of educational activities to improve outcomes for students. )e implementation of
data7informed decision ma"ing cannot occur +itout leadersip and supporting conditions suc
as tools for generating actionable data, professional development and tecnical support for data
interpretation, and time set aside for analy,ing and interpreting data. @or definitions of terms
used in tis report, see E3ibit !78 in !ppendi3 !.
)o understand te role of data systems and te supports necessary for teacers to use data
from any source 1electronic and nonelectronic2 to inform educational practice, te U.S.
Department of Education/s Office of Planning, Evaluation and Policy Development is
sponsoring a national study, te Study of Education Data Systems and Decision Ma"ing. )e
study is addressing a set of basic 9uestions.
8. -at "inds of systems are available to support district and scool data7driven decision
ma"ing; -itin tese systems, o+ prevalent are tools for generating and acting on data;
%. <o+ prevalent are organi,ational supports for scool use of data systems to inform
instruction;
C. <o+ are scool staff members using data systems; Do tey "no+ o+ to interpret
student data; <o+ is scool staff members/ use of data systems and of data more broadly
influencing instruction;
)is report describes interim findings from te first round of data collection and analysis for tat
study.
Conceptual %rame"ork of Data-Informed Decision-Making (rocess and Supports
)e conceptual frame+or" on +ic te study design and instrumentation +ere based is
presented ere as a prelude to te description of te study approac and early findings.
Proponents of data7informed decision ma"ing call on educators to adopt a continuous7
improvement perspective +it an empasis on goal setting, measurement and feedbac" loops so
teacers and administrators can reflect on teir programs and processes, relate tem to student
outcomes, and ma"e refinements suggested by te outcome data.
A
)o ma"e tis process
continuous, educators must ten implement te refined practices, again measuring outcomes and
loo"ing for places for furter refinement. !n important distinction bet+een tis practice and
A
$t sould be noted tat in business, a continuous improvement process is feasible under conditions +ere tere is
standardi,ed implementation of a set of processes and standardi,ed measurement of outcomes. Under tese
conditions, monitoring of outcome measures typically results in small, incremental improvements to te standard
process. -itin scools, standardi,ed implementation of teacing practices is not te norm, so implementing
continuous improvement processes becomes more difficult.
8
most educational programs or reforms is tat tis improvement cycle does not end +it te first
set of program refinements4it is a +ay of tin"ing and a +ay of life rater tan a discrete event
or process 1Scmo"er 8''=2.
E3ibit 878 so+s te stages in a data7informed, continuous7improvement process. plan,
implement, assess, analy,e data and reflect 1as a precursor to more planning and a refined
implementation2. !s te grapic suggests, components of data7informed decision ma"ing are
part of a continuous cycle. )e starting point may vary, and tere is no fi3ed end point. $n
addition, "ey prere9uisites and supporting conditions are essential for tis process to be
successful.
Ehi!it '-'
Conceptual %rame"ork for Data-Informed Decision Making
The (rocess of Using Data for Impro#ement
)e stages in tis cycle, toug often closely lin"ed and overlapping in e3ecution, can be
caracteri,ed as follo+s.
0eflect. !s educational decision ma"ers revie+ teir current practices and outcomes, tey
identify areas in +ic improvement is needed. ! formal needs assessment may be conducted.
%
(rere.uisites and Supporting Conditions
1+ .ata systems 2+ /eadership for
impro'ement
and use of
data
$8 3oos for
generating
data
4+ Socia
structures and
time set aside
for refection on
data
6+ 3oos for
acting on data
5+ Professiona
de'eopment
and technica
support for data
interpretation
Plan. $n response to concerns about e3isting practices or outcomes, educational decision
ma"ers may devise a plan. @or e3ample, a district concerned about lo+ or uneven reading levels
among students in te primary grades may decide to adopt a particular reading curriculum for all
its primary classrooms. ! scool may design a Gsafety netH program for students +o scored
belo+ grade level on te prior year/s matematics test. Often, a group of teacers or
administrators +ill Dointly design te ne+ process or materials.
*mplement. Once an intended educational cange is planned, it must be implemented
+itin te scool or classroom. $mplementation includes 1a2 disseminating documentation and
supporting materials concerning te ne+ approac and 1b2 training te teacers, administrators or
oter personnel responsible for ma"ing it appen. $t also includes te decisions tat tese
individuals ma"e in deciding o+ to implement te mandated or agreed7on cange +itin teir
particular scool or classroom.
Assess. !n organi,ation dedicated to continuous improvement does not Dust implement a
ne+ program or process. )e organi,ation builds in opportunities to measure te outcomes
obtained +it te educational cange. Often, tis effort +ill include some "ind of student
assessment suc as district end7of7unit tests or teacers/ 9uarterly appraisals of students/ +riting
fluency. Depending on te nature of te innovation, te measured outcomes could be of many
different types, including, for e3ample, student attendance or parent attitudes to+ard te scool
as e3pressed on a survey or troug participation in parent7teacer conferences.
Analy.e data. <aving accumulated assessment data, educators ten need to analy,e te
data in +ays tat relate te outcomes to processes and turn te data into actionable information.
Often tis analysis includes segmenting te findings by student subgroups or by groupings used
+itin te scool or classroom 1because students in different groups received different services2.
$t may also include te analysis of data trends over time and te searc for patterns in multiple
measures related to a given cange or issue.
0eflect. <aving analy,ed te data, educators ten must ma"e sense of observed canges or
te lac" of cange. )is reflection stage is +ere participants interpret te findings of te data
analysis and dra+ implications for action4tat is, +ere tey develop ideas for o+ tey can
refine and improve teir program. )e inferences dra+n in tis stage lay te ground+or" for
developing a ne+ plan.
Supports for the (rocess of Using Data for Impro#ement
Ma"ing te continuous7improvement perspective and te processes of data7informed
decision ma"ing part of te +ay in +ic educators function re9uires a maDor cultural cange.
Suc a cange +ill not occur +itout leadersip, effort and +ell7designed supports. )e bottom
portion of E3ibit 878 identifies si3 maDor types of prere9uisites and supports for data7informed
decision ma"ing. 1a2 state, district and scool data systems: 1b2 leadersip for educational
improvement and te use of data: 1c2 tools for generating actionable data: 1d2 social structures
and time set aside for analy,ing and interpreting data: 1e2 professional development and tecnical
support for data interpretation: and 1f2 tools for acting on data. )e national Study of Education
Data Systems and Decision Ma"ing, part of te 5ational )ecnology !ctivities 15)!2 tas"
C
order, is collecting data on te prevalence and 9uality of student data systems and associated
supports. )e nature of tese systems and supports are described belo+.
Data systems. 5C6B as stimulated an unprecedented level of state activity directed at
improving education data systems. @ederal re9uirements for reporting scools/ year7to7year
progress in raising acievement overall and for specific student categories ave led to an
e3amination of information system ade9uacy and te adoption or development of ne+ soft+are
systems in many states. State systems typically include student enrollment information, basic
demograpic data, special program designation 1if applicable2, and scores on state7mandated
acievement tests 1in most cases, an annual spring testing in language arts and matematics and
often a proficiency or Ge3itH e3amination re9uired for a ig scool diploma2.
B

)e 5ational Center for Educational !ccountability 15CE!2 as underta"en a systematic
revie+ of te caracteristics of state7level education data systems. Over te last four years, states
ave made maDor strides to+ard putting in place data systems tat +ill support longitudinal
analysis of student progress. $n %&&B, for e3ample, C= states used uni9ue student identification
numbers state+ide so students could be follo+ed if tey canged districts. $n %&&?, AB states did
so. !s of %&&?, a maDority of state systems also included student7level enrollment, demograpic
and program participation information: te ability to matc individual students/ test scores from
year to year to measure academic gro+t: and student7level graduation and dropout data. 6ess
prevalent +ere te inclusion of scores on college readiness e3aminations 1suc as te S!), !C)
or !dvanced Placement e3ams2: te ability to matc students/ Pre7I>8% records +it te state/s
iger education system records: and a data audit system for assessing data 9uality, validity and
reliability.
=

Scool districts typically maintain teir o+n data systems. $n addition to student scores for
state7mandated tests, +ic tey get from te state or te state7designated vendor, district
systems often include information about a student/s teacers, grades, attendance, disciplinary
infractions and scores on district tests.
$n terms of te soft+are applications designed to support access to and analysis of data
from tese systems, most of te commercial products are targeted to districts. $f districts ave
developed and implemented district+ide assessments tat students ta"e trougout te scool
year, tese products typically support scool7level access to te assessment results.
Oter soft+are applications are targeted for scool use, and tese tend to include features
suc as electronic grade boo"s and te capability to incorporate teacer7developed formative
assessments. Some of tese applications ave te capability to store e3amples of student +or" in
an Gelectronic portfolio.H
$n teory, state7, district7 and scool7level data systems could be interoperable 1U.S.
Department of Education %&&A2, and tere could be Gone7stop soppingH for educational data
coming from sources ranging from te individual scool to te U.S. Department of Education
B
N!" re9uires tat states create annual assessments tat measure +at cildren "no+ and can do in reading and
matematics in grades C troug E and tat tey test at least once bet+een grades 8& and 8% by %&&B>&= for
reading and matematics and by %&&?>&E for science.
=
See ttp.00+++.data9ualitycampaign.org0surveyJresults0 for a complete report of 5CE!/s %&&? state survey.
A
1+at one vendor caracteri,es as te Gsecretary to SecretaryH solution2. Suc a set of integrated
systems does not yet e3ist, but federal efforts are moving to+ard uniform and consolidated
reporting of data to te federal level, and some states are attempting to incorporate features in
state systems tat +ill ma"e tem more useful to districts and scools 1Palaic, (ood and van
der Ploeg %&&A2.
,eadership for educational improvement and use of data. Pioneering efforts to promote
data7informed decision ma"ing +itin districts and scools ave found tat te active promotion
of te effort on te part of te superintendent or principal is vital 1Cromey %&&&: <alverson et al.
%&&B: <erman and (ribbons %&&8: Mars, Pane and <amilton %&&=2. District and scool leaders
need to issue te Gcall to armsH for improving education and using data as a tool to bring about
tat improvement. )ypically, tey play a maDor role in framing targets for educational
improvement, setting e3pectations for staff participation in data7informed decision ma"ing, and
ma"ing resources suc as supported time available to support te enterprise.
Tools for generating actionable data. $ncreasingly, student acievement data are
available at te scool level in a form tat can be disaggregated by student category 1etnicity,
free or reduced7price lunc status, special education status and so on2. Soft+are systems to
support data7informed decision ma"ing all generate standard student acievement reports, and
many also produce custom reports for user7designated student groups 1an important feature for
scool staff members +o +ant to e3amine te effects of locally developed services for specific
student groups2. #esearc indicates, o+ever, tat often, scool staff members do not find te
"inds of data tese systems provide particularly useful for guiding instruction 1Mandinac et al.
%&&B: Mars, Pane and <amilton %&&=2. Scool staff members are frustrated by te fact tat te
data available to tem are typically data reflecting performance on a state acievement test ta"en
si3 or more monts earlier. )eacers +ant up7to7date information on teir current group of
students, not te students in te same grade level last year. )ey +ant a greater level of detail
concerning individual students/ strengts and +ea"nesses tan tey can get from standardi,ed
test scores 1)orn %&&%2. !ltoug far less common tan systems tat provide data from prior
testing, tere are e3amples of systems tat produce additional information for decision ma"ing
troug tools suc as formative assessments tat students may ta"e online. $n addition, some
system design companies are +or"ing on education information systems tat +ould integrate
data on a broad range of transactions suc as daily scool attendance, grades and even library
boo" cec"outs, +it te ultimate goal of automatically recording eac interaction a student as
+it te scool and student/s assessment and program participation data.
Social structures and time set aside for analy.ing and interpreting data. )e most
sopisticated data +areouse in te +orld +ill ave no effect on instruction if no one as4or
ta"es4te time to loo" at te data, reflect on tem, and dra+ inferences for instructional
planning. (iven tat time is one of te most basic resources in any organi,ation, tere need to be
strong e3pectations tat educators +ill ta"e te time to e3amine data and use tose data to guide
improvements in teir programs and practices. )eacer survey data so+ tat setting time aside
for suc activities is not business as usual in !merican scools 1U.S. Department of Education
%&&Eb2. Case studies of scools tat are active in data7informed decision ma"ing suggest tat
organi,ational structures tat include time set aside for revie+ing and discussing data in small
B
groups greatly increase te li"eliood tat te e3amination of data +ill be conducted and +ill
lead to +ell7informed decisions 1Coppin %&&%: Cromey %&&&2.
Professional development and technical support for data interpretation. )eacer
training generally as not included data analysis s"ills or data7informed decision7ma"ing
processes in te past 1Coppin %&&%2. @e+ administrators ave tis "ind of training eiter
1<erman and (ribbons %&&82. Moreover, te measurement issues affecting te interpretation of
assessment data4and certainly te comparison of data across years, scools or different student
subgroups4are complicated. Data misinterpretation is also a real concern 1Confrey and Ma"ar
%&&B2. @or tis reason, districts and scools are devoting increasing amounts of professional
development time to te topic of data7informed decision ma"ing. Many argue tat te practice of
bringing teacers togeter to e3amine data on teir students and relate tose data to teir
practices is a valuable form of professional development in its o+n rigt 1@eldman and )ung
%&&82. Some districts are using Enancing Education )roug )ecnology 1EE))2 professional
development funds to under+rite tese activities. $n addition, some districts tat ave been active
in tis area provide data GcoacesH or oter means for accessing tecnical e3pertise to scool
teams engaged in loo"ing at data.
Tools for acting on data. )e e3amination of data is not an end in itself but rater a means
to improve decisions about instructional programs, placements and metods. Once data ave
been analy,ed to reveal +ea"nesses in certain parts of te education program or to identify
students +o ave not attained te e3pected level of proficiency, educators need to reflect on te
aspects of teir processes tat may contribute to less7tan7desired outcomes and to generate
options for addressing te identified +ea"nesses. Some of te data7informed decision7ma"ing
systems incorporate resources tat teacers can use in planning +at to do differently. )ese
resources are typically organi,ed around state content standards and may include lesson plans,
instructional materials or descriptions of best practices 1Palaic, (ood and van der Ploeg %&&A2.
#esources for differentiated instruction can elp teacers adapt teir instructional approac to
students +it differing strengts and +ea"nesses.
Data Sources for the Interim ,eport
Findings from this report are drawn from both survey and case study data. The survey data
include two data sets from the U.S. Department of Educations National Educational Technology
Trends Study (NETTS). The primary data set for this report consists of responses of a random
sample of K12 teachers to a survey administered to 2,509 teachers in spring 2007. The teachers
were clustered in schools sampled from districts participating in a NETTS district survey. The
NETTS district survey was sent to a nationally representative sample of 1,039 districts also
surveyed in spring 2007. Both teachers and districts were asked to report on activities during the
200607 school year.
7
@or te case studies, te site selection process began +it identifying districts tat ave
been active in using student data to inform instruction. )e researc team ten +or"ed +it te
selected districts to identify appropriate scools. By focusing field+or" on districts +ere many
teacers could be e3pected to be actively loo"ing at student data, te study team increased te
?
)e response rate for te teacer survey +as E= percent, and te response rate for te district survey +as 'A
percent. !dditional information about te 5E))S survey samples can be found in !ppendi3 !.
=
li"eliood of seeing effects of data use on practice, compared +it a sample of scools dra+n at
random.
District sites +ere identified troug tree metods.
5ominations by )-( members and oter leaders in educational tecnology, researcers,
vendors and staff of professional associations suc as )e Consortium for Scool
5et+or"ing and State Educational )ecnology Directors !ssociation, supplemented by
a searc of conventionally publised and -eb7publised literature on data7informed
decision ma"ing, to identify scools and districts +it reputations as leaders in tis area
#ecommendations by ED staff members
Pone intervie+s +it relevant staff members in nominated districts
The final set of 10 districts for the first round of site visits in 200607 comprised three
districts using their own locally developed systems and seven using commercial systems (each
district using a different commercial system). They represented nine different states and
comprised six large and four medium-size districts, with student enrollments ranging from fewer
than 6,000 to more than 164,000 students.
8
Student poverty levels ranged from 13 percent to 70
percent of student enrollment, and the percentage of minority students ranged from 12 percent to
87 percent.
@or eac of te 8& districts selected for study, te study team contacted te district
representative by pone to gain recommendations for tree case study scools, based on te
follo+ing criteria.
One elementary or middle scool tat te district considers high in its data use practices
One scool tat as so+n improvement in its use of data to guide instruction
1emerging2
One scool tat is typical of te district +it respect to use of data systems
#esearcers as"ed te district to recommend, to te e3tent possible, tree scools serving
demograpically similar students at te same grade level 1eiter elementary or middle scool2.
-itin eac scool, te principal, an instructional or data coac 1if applicable2, and si3 teacers
+ere intervie+ed.
E
Conclusions based on te case study districts sould not be e3tended to all districts because tey are not a
nationally representative sample, but tey do provide more in7dept information about data use practices tat can
be brougt to bear in interpreting data gatered troug national surveys. $n particular, te generali,ability of data
from te case study sample to small districts 1enrollments of fe+er tan B,&&& students2 is limited: none of te
districts visited in %&&=>&? +as small because small districts could not satisfy te re9uirement for tree scools
from te same grade span 1i.e., tree elementary or tree middle scools2. )ese smaller districts serve
appro3imately C% percent of te student population in public elementary and secondary districts in te nation as
reported in te 5ational Center for Education Statistics Common Core of Data 1<offman %&&?2. District si,e can
ave a significant effect on te resources tat scool districts can apply to tecnology and professional
development.
?
One of te 8& selected districts 1large +it a locally developed system2 +as unable to
accommodate a site visit until te %&&?>&E scool year. )e demograpic and acievement data
for te nine districts in te %&&=>&? site visit sample are so+n in E3ibit 87%. @or eac district
visited, respondents included "ey staff members involved in te district/s data7informed decision
ma"ing activities 1e.g., cief information officers, directors of curriculum and instruction,
directors of researc and evaluation, directors of accountability, directors of professional
development2.
Ehi!it '-+
+//01/2 Case Stud$ Districts
District
and
Schools Demographics
(ercentage
Attaining +//0
(roficienc$ &e#els
State A#erage
(roficienc$ &e#els
.istrict 1
$ midde
schoos
Student enroment 9 1$2,4-2 !'ery
arge#
Percentage minority 9 74:
Percentage po'erty 9 62:
;o+ of schoos 9 211
<eading !5r+ -# 9 4-:
&ath !5r+ -# 9 11:
<eading !5r+ -# 9 41:
&ath !5r+ -# 9 26:
.istrict 2
$ eementary
schoos
Student enroment 9 164,2(5 !'ery
arge#
Percentage minority 9 50:
Percentage po'erty 9 20:
;o+ of schoos 9 2$-
<eading !5r+ 4# 9 -7:
&ath !5r+ 4# 9 -$:
<eading !5r+ 4# 9 -7:
&ath !5r+ 4# 9 -1:
.istrict $
$ eementary
schoos
Student enroment 9 1$7,7(- !'ery
arge#
Percentage minority 9 5(:
Percentage po'erty 9 $(:
;o+ of schoos 9 1((
<eading !5r+ 4# 9 (0:
&ath !5r+ 4# 9 -(:
<eading !5r+ 4# 9 -6:
&ath !5r+ 4# 9 -6:
.istrict 4
2 eementary
schoos = 1
midde schoo
Student enroment 9 $(,21$ !arge#
Percentage minority 9 -2:
Percentage po'erty 9 64:
;o+ of schoos 9 61
<eading !5r+ 4# 9 -4:
&ath !5r+ 4# 9 -6:
<eading !5r+ -# 9 --:
&ath !5r+ -# 9 -6:
<eading !5r+ 4# 9 -4:
&ath !5r+ 4# 9 --:
<eading !5r+ -# 9 -(:
&ath !5r+ -# 9 71:
.istrict 5
$ eementary
schoos
Student enroment 9 26,22( !arge#
Percentage minority 9 -7:
Percentage po'erty 9 70:
;o+ of schoos 9 55
<eading !5r+ 4# 9 $(:
&ath !5r+ 4# 9 $2:
<eading !5r+ 4# 9 6$:
&ath !5r+ 4# 9 54:
.istrict 6
$ eementary
schoos
Student enroment 9 5,5(( !med+#
Percentage minority 9 64:
Percentage po'erty 9 4$:
;o+ of schoos 9 14
<eading !5r+ 4# 9 51:
&ath !5r+ 4# 9 4$:
<eading !5r+ 4# 9 56:
&ath !5r+ 4# 9 4-:
.istrict 7
$ eementary
schoos
Student enroment 9 (,6-5 !med+#
Percentage minority 9 5-:
Percentage po'erty 9 61:
;o+ of schoos 9 1(
<eading !5r+ 4# 9 -2:
&ath !5r+ 4# 9 6$:
<eading !5r+ 4# 9 --:
&ath !5r+ 4# 9 -1:
E
Ehi!it '-+ 3continued4
+//01/2 Case Stud$ Districts
.istrict -
$ eementary
schoos
Student enroment 9 10,7-0 !med+#
Percentage minority 9 71:
Percentage po'erty 9 62:
;o+ of schoos 9 22
<eading !5r+ 4# 9 $(:
&ath !5r+ 4# 9 50:
<eading !5r+ 4# 9 64:
&ath !5r+ 4# 9 71:
.istrict (
$ midde
schoos
Student enroment 9 22,174 !med+#
Percentage minority 9 12:
Percentage po'erty 9 1$:
;o+ of schoos 9 2(
<eading !5r+ -# 9 6$:
&ath !5r+ -# 9 60:
<eading !5r+ -# 9 4$:
&ath !5r+ -# 9 42:
Note: Numbers have been used to label districts for confidentiality reasons. The same number for a district is used
throughout the report to support comparisons of data. Achievement data refer to the percentages of students who have
scored proficient or above in reading and math in comparison with state standards based on state Web-site data. The
number of district schools excludes early childhood centers and charter schools.
)eacers to be intervie+ed +ere nested +itin te purposive sample of scools. Even
toug scools +ere selected because tey used student data systems in instructional decision
ma"ing, variability +as e3pected across teacers +it respect to o+ tey +ere using data for
instructional decision ma"ing. 1Suc variability is typical for educational reforms and +as found
for data7informed decision ma"ing in earlier scool case studies 1Mars, Pane and <amilton
%&&=22. )erefore, proDect staff members re9uested tat te principal of eac case study scool
nominate tree active practitioners of data7informed decision ma"ing and tree teacers +o
represented average use. $n tis +ay, te study e3pected to capture Gbest practicesH +itin te
scool but still maintain a realistic perspective +it regard to te pervasiveness of tose
practices.
'
An additional round of case studies will be conducted in school year 200708. A second
sample of districts has been drawn from the pool of districts that remained after the initial
selection of 10 districts, supplemented by additional districts identified as being involved in data-
informed decision making activities. These districts included Broad prize winners and nominees
as well as selected districts participating in focus groups at the U.S. Department of Education to
discuss issues related to education technology (i.e., districts using data to support instructional
decision making).
10
The 200708 case study sample also includes a small district (for which an
exception was made to the requirement of having three schools serving the same grade levels).
Contents of This ,eport
)is interim report focuses primarily on describing te types of data available to scool
staff members, o+ scool staff members use electronic data systems, scool practices +it
respect to data7informed decision ma"ing, and te supports and callenges for scool use of
student data in planning and implementing instruction. $n addition, Capter A describes findings
'
$t is 9uite possible tat te teacers +om principals described as typical in terms of data use +ere in fact better
tan average for te scool. -en tis potential bias is considered along +it te fact tat te study conducted
case studies in districts tat +ere considered leaders in te instructional use of data, te reader sould be a+are
tat te teacers/ understanding of data described erein is li"ely to be better tan tat of average teacers.
8&
)e districts participating in te focus groups +ere identified troug e3pert nominations and purposively
sampled to obtain a balance by si,e 1small, medium, large2 and geograpic region.
'
from te presentation of data scenarios to teacers +o +ere as"ed a series of 9uestions related
to o+ tey could understand and use ypotetical data sets. )e capters of te report are
organi,ed around tese issues, +it survey and case study data on te same issue presented
togeter. Capter B summari,es te study/s early findings.
8&
+- School Data (ractices
Most states and districts have relied on cohort data to document academic achievement. This
type of data is often used to present results of student performance on annual assessments and to
compare schools and districts. What cohort data do not provide is information on the progress of
individual students over time or the capability to disentangle changes in achievement from
changes in the composition of the student cohort.
Recently, considerable effort has been put into building data systems at the state level that
are capable of tracking performance of students over time. For example, the Institute of Education
Sciences in the U.S. Department of Education awarded grants of $52.8 million in 2006 and $62.2
million in 2007 to state education agencies (SEAs) for the design and implementation of
statewide longitudinal data systems. The Data Quality Campaign, a national collaborative effort to
encourage and support the implementation of state longitudinal data systems to improve student
achievement, has sponsored an annual survey of states to assess how many of the 10 essential
elements of a longitudinal data system each state currently has (Data Quality Campaign 2007).
11

)e data collected in tis study suggest tat te e3tensive state7level activity around
longitudinal data systems as not yet ad a maDor effect on local use of student data. Muc of te
data available to educators comes from local data systems. $n te %&&=>&? 5E))S teacer survey,
among teacers reporting access to an electronic data system tat provided tem +it student
data, te vast maDority 1?' percent2 reported tat te data system +as provided by teir district.
*ust a tird of tese teacers 1CC percent2 reported tat te data system +as by teir scool, and
Dust 8? percent said tat it +as provided by teir state 1respondents could indicate more tan one
source2.
8%
)eacers in te nine 5)! case study districts also relied on teir districts/ electronic
information systems as te primary source for student data. Case study districts +ere located in
states +it state data systems tat represented a broad continuum in terms of implementation of a
longitudinal data system. te systems +itin tese nine states ad bet+een tree and nine of te
Data Fuality Campaign/s 8& essential elements of a longitudinal data system 1a summary is
provided in E3ibit !7% in !ppendi3 !2.
District Data S$stems
Proponents of data7informed decision ma"ing in education offer te vision of an integrated
system of systems, combining longitudinal information on eac individual student/s scool
istory, data needed for accountability as +ell as finance, personnel, food service, and even
transportation data 1U.S. Department of Education %&&A: -ayman %&&B2. )is vision is valuable
in setting a direction for policy7ma"ers, but even in districts nominated as leaders in te use of
student data systems, it is still not a reality. )ypically, te information needed for educational
decision ma"ing is spread across multiple systems +itout mecanisms for regular transport of
88
$n %&&?, among te B& states and te District of Columbia, tree ad one to tree elements, 88 ad four to five
elements, %8 ad si3 to seven elements, 8% ad eigt to nine elements, and four ad all 8& elements.
8%
Even toug teacers using a data system are most li"ely to be using a system provided by teir district, a good
portion of te data in te district/s system may ave come from te state system.
88
information from one system to anoter.
8C
-en as"ed +at system provided student data tat
scools use for decision ma"ing, staff members at te nine districts in te spring %&&? case study
sample all mentioned t+o or more separate systems. ! summary of te types of data systems used
in te case study districts is included in E3ibit !7C in !ppendi3 !.
)is state of affairs is a natural outgro+t of te multiple purposes for +ic student
information is collected and te fact tat systems ave tended to be developed 1and mar"eted2
over time around a range of specific needs. )e typical district as a data system for trac"ing
re9uired special education services, for e3ample, +ic is separate from te system tat maintains
te longitudinal enrollment istories of students generally. Some systems are geared to capturing
daily transactions suc as attendance and grades +ile oters are designed for access to
longitudinal records 1-ayman %&&B2. !ltoug it is not necessary to ave all types of data in a
single system, lac" of interoperability bet+een systems creates inefficiencies in data input, and
differences in system interfaces increase training re9uirements for district and scool staff.
-ayman 1%&&B2 defines four "inds of student data systems in +idespread use.
8. Student information systems provide real7time access to student data suc as attendance,
demograpics, test scores, grades and scedules.
%. Data +areouses are electronic data collection and storage systems tat provide access to
current and istorical data on students, personnel, finance, etc.
C. $nstructional0curriculum management systems provide a unifying frame+or" to support
access to curriculum and instructional resources suc as planning tools, model lesson
plans, creation of bencmar" assessments, lin"age to state content or performance
standards, communication and collaboration tools 1e.g., treaded discussion forums2.
A. !ssessment systems support rapid organi,ation and analysis of bencmar" assessment
data.
)e most fre9uent types of electronic data maintained by districts4based on all districts
responding to te %&&=>&? 5E))S district survey4+ere student attendance 1'= percent2, grades
1'C percent2, demograpics 1'% percent2, special education information 1'8 percent2 and course
enrollment istories 1EE percent24basic elements of a student information system. Student test
scores on state+ide assessments +ere maintained in electronic form by E= percent of districts.
@e+er districts stored electronic data on teacer 9ualifications 1=% percent2 and on participation of
students in particular educational programs suc as tose using an innovative classroom
curriculum 1?? percent2.
8A
)e nine districts in te spring %&&? case study sample +ere similar to te national sample.
)ey all ad student information systems, but, as noted above, te data typically +ere captured in
several different systems +it some limitations in interoperability. Data +areouses +it
longitudinal data +ere available in about alf of te case study districts, but curriculum
management systems +ere used by Dust t+o.
8C
5ational estimates on te types of student data systems in districts and o+ long tey ave been available +ill be
provided troug te %&&?>&E 5)! district survey 1Fuestions C and A2.
8A
$nformation on te types of data in district information systems +ill be updated troug te %&&?>&E 5)! district
survey 1Fuestion B2.
8%
One teme tat emerged from te case studies +as te e3tent to +ic multiple systems
+ere used to ouse education data, +itout necessarily aving any efficient procedures 1or
sometimes +itout any capability at all2 for connecting data from different systems. One district
described te use of four different data systems. one for state and district assessment scores, one
for student demograpic and enrollment information, one for discipline reports, and one for
special education. !cross case study districts, it +as typical to find separate systems for
longitudinal data, teacer professional development data and special education data. 5one of te
districts ad an integrated data system and data +areouse of te "ind described in te %&&A
5ational )ecnology Plan 1U.S. Department of Education %&&A, AA2.
Teacher Access to Student Data S$stems
!mong te nationally representative sample of teacers ta"ing te 5E))S teacer survey,
rougly tree79uarters 1?A percent2 reported aving access to an electronic student data system in
scool year %&&=>&?, up from AE percent in te 5E))S teacer survey conducted in %&&B.
8B

!ltoug tis trend in te survey data is encouraging, te scool case studies provided an
opportunity to get a deeper understanding of data system access issues. )e 9uestion of data
access is actually surprisingly comple3. Does access mean tat a teacer can get te data out of
te system erself or Dust tat tere is a +ay tat se can get data from te system if se as"s
someone for it; -at if a teacer is allo+ed to see some "inds of data for is students but not
oter "inds, or +at if e can see data for some of is students but not all; Does a teacer ave
access if se can get only standard reports and cannot frame any of er o+n data 9ueries;
Teachers in all of the case study districts had some form of access to student data, but
teachers could get assessment data for their students out of the data system themselves in only
four of the nine districts.
16
Many districts consider having relevant data extracted for teachers to
be a more efficient, reliable practice than trying to train teachers to extract data for themselves.
Teachers could extract some but not all assessment data for their students in an additional two
districts. In three districts, teachers received data reports from the district office but could not
extract student data on their own.
17

!ltoug case study districts +ere selected because of teir activity in promoting data
systems, teacers in te selected scools reported little firstand use of data systems. $t +as rare
for case study teacers to describe accessing student assessment data from a system temselves
for any purpose oter tan analysis of teir students/ responses to individual assessment items. $n
t+o7tirds of te districts, tere +ere one or more staff members at te scool site +o +ould
produce data reports for teacers. $n a number of cases, tis practice +as a deliberate decision on
8B
)e %&&B 5E))S teacer survey +as administered to a nationally representative sample of I>8% teacers in fall
and +inter %&&B. )eacer respondents +ere as"ed to report on activities in te %&&A>&B scool year. )eacers +ere
sampled from 'B? scools +itin districts selected for te 5E))S district survey. )ese are te same districts tat
participated in te %&&? 5E))S district survey 1see U.S. Department of Education %&&Ea for additional
information on tese surveys2.
8=
!mong teacers sampled from si3 case study districts +o responded to te 5E))S teacer survey, E& percent
reported aving access to an electronic data student data system in %&&=>&? compared +it ?A percent of teacers
in non7case7study districts.
8?
$nformation on te access tat scool staff ave to student data systems +ill be updated troug te %&&?>&E 5)!
district survey 1Fuestions '>882.
8C
te part of te district to provide tis resource, eiter because teacers +ere vie+ed as already
overburdened or because tere +ere doubts about teacers/ ability to use te system or understand
te data by temselves. Even a district tat began by purcasing a laptop for every teacer to use
for accessing te data system concluded tat teacers +ould not do so +it any fre9uency on teir
o+n and ended up creating assessment and instructional coac positions to elp teacers connect
data to instruction. Staff members in one case study district made it clear tat tey did not tin"
tat scool access to student data systems +ould be a good practice. District staff members
expressed concern that principals might use data from the system inappropriately to evaluate
teachers.
T$pes of Data A#aila!le to Teachers
!mong te ?A percent of 5E))S teacer survey respondents +o said tat tey ad access
to a student data system, te most common types of data available to teacers +ere attendance
data 1?A percent2 and student grade data 1=? percent24te same types of data tat districts
reported fre9uently aving in teir electronic data systems. )eacers also reported access to data
on teir current students. ?? percent said tat tey could get access to standardi,ed test scores
1spring and fall %&&=2 for te students tey +ere teacing currently, and over alf 1BB percent2 ad
performance data on bencmar" or diagnostic tests ta"en by teir current students. 6ess tan alf
1A= percent2 of teacers +it access to a student data system reported aving access to data for
students tey ad taugt te preceding year. ! muc smaller percentage of teacers 1%' percent2
reported being able to access multiple years of test scores for individual students. Only E percent
said tat tey could get information on any supplemental services teir students +ere receiving.
E3ibit %78 summari,es tese survey findings.
!noter perspective on teacers/ access to data from a student data system can be gained by
e3amining responses of district personnel completing te 5E))S district survey. 6ess tan alf of
te districts 1A= percent2 responding to te survey reported tat teir teacers are provided +it all
or most of te data available for teir students. @orty7four percent indicated tat tey provide
teacers +it only a limited set of data on teir students. !noter ' percent of districts reported
tat tey do not ma"e any data available to teacers on te students in teir classrooms.
8A
Ehi!it +-'
Data A#aila!le to Teachers in +//01/2
T$pes of Data A#aila!le
(ercentage of
Teachers )ith
S$stem Access
(ercentage of All
Teachers
Attendance data 74 55
Student grade data 67 50
Performance on benchmark or diagnostic tests
taken by your current students
55 41
Spring 2006 standardized test scores for the
students you taught ast year !2005"06#
46 $4
Spring 2006 standardized test scores for your
current students
44 $$
%a 2006 standardized test scores for your
current students
$$ 25
&utipe years of standardized test scores for
indi'idua students
2( 11
)ourse enroment histories for students 22 16
Students* prior schoo!s# attended 16 12
Students* participation in suppementary
education programs !e+g+, tutoring#
- 6
E3ibit reads. !mong teacers +o ad access to an electronic student data system in %&&=>&?, ?A
percent reported tat tey ad access to student attendance data troug te data system, +ic
represents BB percent of all teacers 1+it or +itout access to a data system2.
Source. %&&=>&? 5E))S teacer survey.
)e purposes for +ic teacers reported using data from student data systems are so+n in
E3ibit %7%. )e most common purposes reported by teacer survey respondents +ere informing
parents about student progress 1=E percent2,
8E
trac"ing individual student scores 1=B percent2, and
estimating +eter students are ma"ing ade9uate progress 1=A percent2. !dditional purposes cited
by a maDority of teacers +it data system access +ere trac"ing oter measures of student
progress 1B' percent2 and identifying s"ill gaps for individual students 1BB percent2. 1Unless
otherwise indicated, all percentages reported for teacher survey respondents in the remainder of
this report are for the 74 percent of teachers who reported that they had access to an electronic
student data system in 200607.)
8E
)eacers responded tat tey used an electronic student data system for a specified activity for one of a set of
specified fre9uencies during te %&&=>&? scool year 1i.e., a fe+ times, once or t+ice a mont, or once a +ee" or
more2.
8B
Ehi!it +-+
Teachers5 Use of Student Data
11
7
10
(
$
5
2
$
$
2
4
20
17
20
15
15
1$
-
11
10
-
(
$7
41
$4
$5
$7
26
$4
2(
26
26
2$
1(
20
1(
24
20
2-
$$
25
$0
$6
$4
1$
15
17
17
25
27
24
$2
$1
2-
$0
0: 10: 20: $0: 40: 50: 60: 70: -0: (0: 100:
7nform parents about student progress
3rack indi'idua student test scores
>stimate 1hether students are making ade?uate progress
3rack other measures of student progress
7dentify ski gaps for indi'idua students to gi'e each student materia
taiored to his@her ski profie
.etermine 1hether the cass or indi'idua students are ready to mo'e on to
the ne6t instructiona unit
3rack standardized test scores by grade
7nform curricuum changes
>'auate promising cassroom practice
7nform student pacement in courses or specia programs
.ecide 1hether to gi'e students testAtaking practice
(ercentage of teachers 2nce a 1eek or more 2nce or t1ice a month A fe1 times ;e'er .ata not in system
E3ibit reads. !mong teacers +o ad access to an electronic student data system in %&&=>&?, 88 percent
responded tat tey used data to inform parents about student progress once a +ee", %& percent once or t+ice a
mont, C? percent a fe+ times, and 8' percent never: 8C percent indicated te system did not provide data for tis
purpose.
5ote. )otals may not add to 8&& percent because of rounding.
Source. %&&=>&? 5E))S teacer survey.
)e e3tent to +ic teacers conduct tese data use activities is li"ely to be influenced by
teacers/ perceptions of system capabilities. @or e3ample, about a tird of teacers reported tat
te system tey ad access to did not provide data to inform curriculum canges 1C% percent2 or to
evaluate promising classroom practices 1C8 percent2. ! potential source of ambiguity in teacers/
data use reports is teacers/ varying interpretations of teir roles 1as +ell as te role of data2 in
functions suc as curriculum cange and evaluation of practices: tese functions may be seen as
district responsibilities. $n addition, districts may ave policies tat run counter to teacers/ use of
data for some "inds of decision ma"ing. @or e3ample, =C percent of teacers reported "no+ing
o+ to use student data to refine +at and o+ tey teac, but =& percent also reported tat
regardless of +at te data tell tem, tey need to "eep up +it state7 and district7mandated
pacing plans.
$n te case study scools, teacers commonly described te use of data for placing students
in different classes or programs and as a basis for grouping students +itin classes for
differentiated instruction. )eacers reported tat te use of assessment data for tis purpose ad
increased teir confidence in teir decisions. !s one reported, GBefore, +e ad noting to bac" up
our professional Dudgment. 5o+ +e can pinpoint +ere a student as a problem. $t as increased
our +or"load, but it as made us better at diagnosing.H )eacers also reported using data 1from
8=
bencmar" assessments2 to decide +eter to reteac. $t +as less common for teacers to report
te use of data as a stimulus for reflection on teir o+n teacing approac, but instances of tis
use +ere observed in four districts. @or e3ample, at one scool, teacers use a team approac to
problem solving. )eacers +or" togeter in teir grade7level teams to analy,e and tal" about data.
$f teacers find tat teir classes are performing at lo+ levels on a particular standard compared
+it tose of some of teir colleagues, tey +or" +it oter teacers to get ideas for +ays to
better teac tat standard.
! use of data reported in case study scools tat +as not included in te 5E))S survey
item +as for scool improvement planning. $ntervie+ees described cases in multiple districts in
+ic te state mandated annual improvement planning, and eiter te district or te principal
brougt in scool data as part of tis process.
Social Contet for Data Use
!mong 5E))S teacer survey respondents, te most commonly reported conte3t for using
a student data system +as on one/s o+n 1?E percent2, follo+ed closely by data use +it
colleagues in a department or grade 1?8 percent2. @ifty7nine percent of teacers said tat tey ad
used data as part of a district7run activity for staff members from teir scool. *ust %E percent ad
done so as part of a district activity for teacers from multiple scools, suggesting tat districts
vie+ te scool as te appropriate unit for professional development in tis area.
$n eac of te case study districts, te study team found scools +ere teacers loo"ed at
data on teir o+n and in groups. )e most fre9uently cited groupings for data use +ere te grade7
level team, sometimes facilitated by a coac, and all7staff faculty meetings. Oter types of
groupings in +ic teacers used data included cluster team meetings, curriculum groups
1department teams in middle scools2, scool+ide special7purpose teams, and cross7grade teams.
$n one district, teacers at te ig7data7use scool are actively involved in data discussions
during +ee"ly grade7level meetings, +ere tey e3amine data collectively and use te data to
elp tem form cross7class groupings for reading. !dditionally, tere are cross7grade meetings
during +ic teacers discuss student scores, s"ills tat students are struggling +it, and o+
teacers sould +or" collaboratively across grades. )ese cross7grade meetings +ere initiated
+en data identified lo+ mat scores as a scool+ide problem. )e scool as also set up its o+n
server +ere teacers can sare instructional materials, assessment data and student +or" and
+ere primary7grade teacers can store reading assessment data tat are not "ept on te district/s
electronic data system.
)eacers at a scool in anoter district are encouraged to use data collaboratively and on an
ongoing basis. )e assistant principal and reading specialist retrieve data montly from te
district data system for eac tested grade level and put it into a spreadseet so scool staff
members can e3amine student performance by teacer, by grade, by subgroup or by educational
program. )ese spreadseets are updated after every test +indo+ in reading and mat so teacers
can bring te reports to teir team meetings to discuss student grouping. )+ice a +ee", te scool
olds grade7level team meetings +ere teacers bring teir o+n data and discuss +ic students
need academic support, +ic students can graduate from academic support, +ic students need
coacing in an accelerated class, and +ic students need to be moved out of te accelerated
class. )eacers find it very important to learn o+ +ell students in oter classes are doing
8?
because tis information is useful +en tey plan teir lessons collaboratively. $n addition, grade7
level teams meet +it te principal tree times a year to discuss data and instructional strategies
4o+ +ell students ave performed overall, o+ te academic support +or"s in te classroom,
+at ne+ goals tey ave set, and +at tey need to acieve teir obDectives. )e principal at tis
scool believes tat teacers sould "no+ o+ to access and use data. )o support tis process
over te past tree years, te principal as provided all teacers +it scool7based training on
o+ to use te data system and E3cel.
6arriers to Use of Data S$stems to Inform Instruction
Mandinac et al. 1%&&B2 ave identified a set of factors tat contribute to teacers/ use of
tecnology tools for data7informed decision ma"ing. easy access: sort feedbac" loops 1bet+een
ta"ing assessments and receiving results2: information compreensibility: fle3ibility 1allo+ing
data to be e3amined in different +ays2: and alignment +it standards, instructional goals and
classroom practice. )eacers participating in te 5E))S survey and tose in case study scools
cited sortcomings in tese areas.
! sense of +at issues are getting in te +ay of teacers/ use of data can be gleaned from
teacers/ ratings of attitude7related statements on te 5E))S teacer survey 1displayed in E3ibit
%7C2. )e biggest concern e3pressed by surveyed teacers +as +it teir ability to use a system to
form data 9ueries. -eter teacers feel indered because tey do not "no+ o+ to frame 9ueries
or because te system tey ave does not support user 9ueries, only CC percent of teacers said
tat tey could form data 9ueries to get pertinent data from te system. )e oter most commonly
cited issues +ere te limited utility of te "ind of information available in te student data
systems for deciding +at and o+ to teac 1cited as a barrier by %' percent2: difficulty finding
te information tey are loo"ing for on te system 1%A percent2: and difficulty in using te system
1%& percent2. !mong scool staff members +o +ere intervie+ed as part of te case study
researc, te tree most commonly cited barriers to scool use of data +ere 1a2 lac" of training in
o+ to use te data system or to derive instructional implications from it, 1b2 te lac" of time to
engage in data e3ploration and reflection, and 1c2 te +ea"ness of te available data. State
assessment data +ere critici,ed as coming too late in te year to be very useful in some districts
and as being too gross a level to provide guidance on anyting more specific tan maDor topics or
proficiency categories. District bencmar" assessment data are collected more fre9uently and
made available +ile teacers are still responsible for te e3aminees, but teacers in multiple case
study districts 9uestioned te 9uality of teir districts/ tests and teir alignment +it te
curriculum.
8E
Ehi!it +-7
Teacher Eperiences Using Electronic Student Data S$stems in +//01/2
Statement
(ercentage of
Teachers )ho Agreed
or Strongl$ Agreed
7 am capabe of forming data ?ueries !asking specific
?uestions and getting the pertinent data from the
system#
$$
3he student data a'aiabe are not reay that hepfu in
deciding 1hat or ho1 to teach
2(
7 ha'e troube finding the information 7 1ant on the
eectronic student data system
24
3he eectronic student data system a'aiabe to me is
hard to use
20
E3ibit reads. !mong teacers +it access to an electronic student data system,
CC percent felt capable of forming data 9ueries.
Source. %&&=>&? 5E))S teacer survey.
District staff members +o +ere intervie+ed as part of te case study site visits provided
teir opinions concerning te barriers to data system use and data7informed decision ma"ing.
District staff members voiced concerns similar to tose articulated by teacers. Staff members at
seven of te nine case study districts cited limitations in teir data systems, including lac" of
system operability, cumbersome processes for generating custom reports, and lac" of tecnology
at te scool level tat +ould support teacers/ access to and use of te data system. Staff
members at seven districts cited limitations in te nature or timing of te data in te systems. !s
+it teacers, te maDor complaints from district staff members +ere tat te standardi,ed test
data do not provide enoug information about students/ specific s"ills and +ea"nesses to support
instructional planning, tat te system does not ave data from formative 1mid7year2 assessments,
and tat state test data come too late 1typically in te fall for te prior spring/s e3aminations2 to
support scool planning. E9ually common among district respondents +as concern tat scool
staff members 1bot principals and teacers2 lac" te s"ills needed to use te data system and to
analy,e data effectively 1seven districts2. Oter common district concerns 1eac e3pressed in five
districts2 +ere lac" of buy7in from scool 1and some district2 staff members, lac" of assessments
+ell aligned +it curriculum standards or for primary grades, and lac" of scool7level staff
members +o could elp teacers learn to access and use data from te system. Staff members at
four districts e3pressed concern +it resource re9uirements: administering assessments,
implementing a data system, entering data and discussing data +ere all cited as activities
consuming large amounts of staff time.
8'
!n issue underlying many of te tensions around local use of data systems is te tradeoff
bet+een standardi,ation and customi,ation. )e more po+erful soft+are systems for storing,
manipulating and reporting student data re9uire maDor e3penditures, not only for soft+are
8'
)e 5)! district survey +ill provide national estimates on current barriers to e3panding data7informed decision7
ma"ing practices 1Fuestion %B2 and areas +ere districts and scools need more support for data system use and
data7driven decision ma"ing 1Fuestion %A2.
8'
purcase but also for te ard+are needed to ost it and to provide scool7level access, for staff
and teacer training around system use, and so on. On te one and, district investments in suc
systems typically are predicated on te assumption of a uniform set of reporting and data use
practices across scools. )eacers and scools, on te oter and, often feel tat te "inds of
information available in a district system do not address te 9uestions and decisions most
important to tem. Scool staff members often +or" +it less formal, ome7gro+n data sets suc
as student performance on scool7developed final e3aminations tat are "ept in electronic
spreadseets. Districts li"e to see district+ide bencmar" assessment data in te district+ide
system, but 1as noted above2 teacers often e3pressed misgivings about te 9uality of te district
assessments and terefore te usefulness of te data.
)ere may be strategies for resolving tis tension. $n one case study district, implementation
of teir student information system made district staff members a+are of teacers/ need for
formative assessments and of te lac" of district+ide tools for measuring end7of7course
acievement. )e data system prompted a district effort to ave te curriculum specialist in eac
content area +or" +it teacer representatives from eac scool to craft end7of7course
assessments tat all scools +ould use. $mplementation of tese assessments and a ne+ data
system designed to support teacer access to te resulting data is e3pected in scool year %&&?>
&E.
Districts do not necessarily +ant teacers to ave direct access to student data. !ltoug
tose advocating for educators/ use of data for decision ma"ing describe active involvement by
teacers in data7based in9uiry 1Brunner et al. %&&B2, scool districts are far from unanimous in
promoting tis model. Even among te nine districts "no+n for teir promotion of data7informed
decision ma"ing tat constituted te case study sample, only four ad direct teacer access to te
data system as an active goal. !n additional four districts vie+ed teacer access as a goal for te
longer term, after te system is made more user friendly or after more teacer training. One
district said tat it ad decided to limit teacers/ access to te data system out of concern tat data
+ould be used inappropriately. Even principals in tat district ad access only to standard reports.
<aving scool7based staff members +o can act as bridges bet+een teacers and te data
system appears to be an important facilitator. Si3 of te nine case study districts ave scool7
based positions to support teacers/ use of student data. ! sevent district instituted district7level
coac positions to elp scools +or" +it data. )ese positions are eld by various types of staff
members in different districts. )e potential do+nside to tis practice, as noted by Boudett and
Moody 1%&&B2, is te ris" tat te generation and e3amination of data +ill come to be regarded as
te responsibility of a single person +itin te scool, +itout involving te scool staff as a
+ole. )e benefit of aving tis position is tat tis staff person can promote data use among
teacers, provide access to data tat teacers oter+ise +ould not see, and model data analysis
and reflection. @or te most part, individuals in tese positions in scools in te case study sample
+ere involved in +or"ing +it teacers on instructional issues. $n several districts, an
instructional coac or advisor led te scool7level data use activities, sometimes in connection
+it a particular initiative suc as #eading @irst. $n some cases +ere tere +as no scool7based
position to support data use, principals or vice principals +ere trained on data system use and ten
e3pected to lead teir teacers in tese activities.
%&
!ltoug teacer access to student data systems as increased over time, te types of data
available to teacers are not necessarily useful for instructional decision ma"ing. Survey and case
study data suggest tat many barriers still e3ist to te effective use of data by teacers, including
te need for more user7friendly data systems, coupled +it te need for additional training on
o+ to use tese systems and o+ to analy,e data. $n addition, many of te supports re9uired for
data7informed decision ma"ing are lac"ing. )e district and scool supports tat are available to
teacers are described in greater detail in te ne3t capter.
%8
7- District and School Supports for Data-Informed Decision Making
)e ideali,ed model of data7informed decision ma"ing supports tat are described in
Capter 8 provides a frame+or" for e3amining te practices of districts and scools +it respect
to data use. )is capter e3plores te issues of data system integration, district and scool
leadersip for data use, tools for generating and organi,ing data, and tools for acting on data tat
+ere found in actual practice in te case study districts and scools.
Data S$stem Integration
!ltoug scools are being e3orted to base teir decisions on data, te systems available
to tem +ere not necessarily designed +it teir needs foremost in mind. Case study data
suggest tat most student data systems +ere built to address uses oter tan scool7level
instructional decision ma"ing. Moreover, te multiplicity of systems used by local educational
agencies +or"s against te integration of different "inds of information 1e.g., discipline and
attendance data, program placement information, district bencmar" and state test scores2 and
re9uires scool personnel to go troug te ramp7up time to learn to use multiple systems.
!ll te districts in te case study sample ad student data systems, but none ad a fully
integrated set of data systems. One district, for e3ample, ad an instructional management
system tat included te district curriculum standards, curriculum guides, instructional resources,
all student assessment data and student demograpics. ! data +areouse stored te district/s
longitudinal data, and an online professional development data system +as used to develop
teacer professional development plans. !noter district used tree data systems. )e student
information system provided istorical test information for eac student, including past state and
district tests 1but not current scores2. ! separate -eb7based transactional system +as used by
scools to transmit to te district information suc as attendance, scool lunc data and scores on
district bencmar" tests. )is system included a grade boo" tat +as available, but not re9uired,
for teacer use. ! separate system oused special education data.
%&

$n general, it +as 9uite common not only for districts to ave separate systems for
transaction capture 1attendance and grades2 and assessment data but also for tese systems to be
detaced from instructional resources geared to standards. <ence, altoug te t+o 5E))S
teacer surveys suggest tat teacer access to data systems increased dramatically bet+een %&&B
and %&&?, te systems temselves do not necessarily support integrated planning and self7
evaluation.
District and School &eadership for Data Use
Districts +ere nominated for case study on te basis of teir leadersip in te use of data
for decision ma"ing, and not surprisingly, evidence of leadersip in tis area +as found in all of
tem. Case study districts demonstrated teir support and leadersip for scools/ use of data
troug a number of actions.
Ma"ing data7informed decision ma"ing a priority and purcasing or developing a
system tat supported tese activities
20
)e district +as in te process of developing a data +areouse to lin" data from all tese systems.
%C
Considering ability to use data as a criterion for teacer iring 1One district ad
developed intervie+ 9uestions on te use of data for instructional decision ma"ing and
used tese +en iring teacers.
%8
!noter ad developed tis "ind of 9uestion for use
+en iring district staff members.2
Providing training and support positions for system implementation 1and trac"ing
usage2.
District+ide data7informed decision ma"ing activities in addition to professional
development 1!n e3ample of tis "ind of activity is provided in E3ibit C78.2
Using te data system for decision ma"ing at te district level 1e.g., evaluating
programs, principals, teacers2.
Ehi!it 7-'
A District and School Data Anal$sis (rocess
>'ery year, schoos in this mediumAsize district get together to conduct Bdata digsC 1here schoo
eadership teams ha'e the opportunity to re'ie1 and discuss their state test data+ 3he teams use
a coaborati'e in?uiry process caed the .ataA.ri'en .iaogue to anayze and interpret their
data+ 3he process, adapted from Druce 0eman and /aura /ipton !2004#, can be used 1ith any
kind of data from any source+ 3he process incudes four steps8
PredictEacti'ate and engage interest in the data, access prior earning, name frames of
reference, and estabish common ground for diaogue+
>6poreEinteract 1ith the data, ook for patterns and trends, identify data facts and
surprises, make obser'ations 1ithout inference, identify ?uestions raised by the data, and
de'eop probem statements+
>6painEgenerate theories of causation, stay open to mutipe possibiities, deepen
thinking and identify Broot causesC rather than symptoms, make inferences about data,
and identify the additiona data that 1i 'aidate the theories of causation+
3ake ActionEmo'e from probems to soutions based on 'aidated theories of causation,
identify goas and specific reated action steps, and identify data to be monitored to
determine 1hether action steps ead to the soution+
A schoos 1ere initiay pro'ided training on the process by the State )onsortium for .ataA.ri'en
.iaogueE.istrict - is the ead district+ 3he consortium 1as estabished 1ith support from a
F$+5 miion, threeAyear grant from the S>A 1ith federa >>33 funds+
7nformation from the data digs is used to inform the de'eopment of the schoo*s impro'ement
pan, 1hich is a re?uirement of the state accreditation process+ Schoos are then encouraged to
continue to re'ie1 their data throughout the year by means of a monthy action pan that
impements and monitors progress on goas in the schoo impro'ement pan+ Schoo eaders 1ho
ha'e embraced the use of dataAinformed decision making use the same data in?uiry cyce to
de'eop their monthy action pans and to 1ork 1ith gradeAe'e teams to monitor student
progress and progress on schoo impro'ement goas+ Principas reported that the data digs ha'e
heped them to structure con'ersations 1ith their staff members about data+ 3eachers can ook at
student progression in a particuar ski or content area as a 1ay to monitor their o1n instruction+
%8
!ny increase in entry7level re9uirements as te potential to screen out Dob candidates +it oter+ise desirable
attributes. <o+ever, no evidence +as identified, one +ay or anoter, specific to any entry7level re9uirements for
teacers/ ability to use data.
%A
)+o districts +ere Dudged to ave +ea"er leadersip for data use tan te oter seven
because one ad a very compartmentali,ed approac to data use 1tere +as little saring across
departments even toug te district +as an advocate of data use2 and te oter district +as Dust
beginning to address a more compreensive approac to data use at te district level. 5eiter
district is currently providing scool7based staff members to support teacers/ use of data 1bot
districts ave faced declining resources tat ave forced support staff cutbac"s2. !ltoug bot
districts are ac9uiring ne+ data systems tat +ill enance data accessibility and analysis
capabilities, currently, neiter as a plan for o+ it +ill provide te necessary professional
development and ongoing support to scool staff members to use tese ne+ systems.
!t te scool level, principals can so+ leadersip +it regard to data use by setting a
personal e3ample of data use, establising e3pectations for data7informed decision ma"ing and
+or"ing +it teacers to interpret data. 1!n e3ample of suc a principal is provided in E3ibit C7
%.2 !cross case study scools, principals so+ed strong leadersip in less tan alf of te
scools. $n t+o of nine districts, principals at all tree scools set an e3ample of data use. )ere
+as no clear pattern evident bet+een principal leadersip and scool type 1typical, emerging and
ig2 in te case study sample, altoug principals +o acted as data leaders +ere sligtly more
common in ig7use and emerging scools tan in typical scools. Principal leadersip may
appear less strongly related to data7informed decision7ma"ing activity in tis study/s sample tan
in prior case studies because of te presence of oter scool leaders or coaces assigned to +or"
+it teacers on data use.
Ehi!it 7-+
(rincipal as Data &eader
3he principa at a schoo identified for its e6empary use of data uses a broad range of data,
incuding students* report cards and grades+ She beie'es that grades, ike beha'ior and
attendance, are a refection of student engagement+ She aso ooks at enroment data !e+g+, if
students seected the schoo by choice# and re'ie1s indi'idua target students* data 1ith their
teacher teams+ 3he principa fees that teachers need data in their hands, and e'en though they
ha'e access to the district data system, she ceans and organizes schooAe'e standardized
testing data from the system for her teachers+ Ger phiosophy is that it is much more important
that teachers 1ork 1ith the data to understand it rather than 1ork to organize it+ She meets 1ith
teachers t1ice a month to tak about the instruction behind the test scores and pro'ides ongoing
data to support them+ As the schoo year progresses, schoo staff members are encouraged to
use a 'ariety of data from indi'iduaized education programs !7>Ps# and from inAhouse, district
and speciaized benchmark assessments to monitor students 1ithin their casses and
departments+ 3eachers are e6pected to use these data to refect on and discuss 1ith their gradeA
e'e teams 1hat is and 1hat is not 1orking instructionay+ 3eachers aso base their initia
referras to ha'e students assessed for specia education on this re'ie1 process+ As a resut, a
cuture of dataAinformed decision making has taken root at the schoo+
Scools set e3pectations for data use +en tey set aside time during scool ours to
discuss data. !ll scools in te case study sample provided at least minimal opportunities to
discuss data 1e.g., at department meetings or during a common planning time available to
teacers of a particular grade or subDect2. )e amount of time available for tese discussions
varied +idely across districts and +as only loosely correlated +it scool+ide data practices.
%B
One district provided opportunities to discuss data in all scools t+o to tree times per +ee". $n
anoter district, te ig7use scool used common preparation times and montly professional
development meetings to tal" about data. )is type of discussion occurred less fre9uently at
departmental meetings in te emerging and typical scools in te same district, +ere principals
often sared data only at te beginning of te scool year.
Scool leaders can also encourage data use by +or"ing +it teacers to elp interpret teir
data. $n some cases, te principal filled tis role: in oter cases, specialists and lead teacers did
so. !s researc on scool leadersip as so+n 1e.g., Camburn, #o+an and )aylor %&&C2, +en
tere are subDect leaders in te scool, tey can play a role e9ual to tat of te principal in terms
of influencing instruction. $n some case study districts, designated instructional or data coaces
too" on4or at least sared +it te principal4te role of data leader. -en coaces +ere
present in a scool, teacers +ere more li"ely to +or" +it te coaces tan +it teir principals
on data interpretation, but in some of tese scools, principals supplemented teacer meetings
+it te coac +it teir o+n meetings +ere tey led te data discussions +it teacers. $n one
district tat did not ave scool7based coaces 1tey ad been lost to budget cuts2, te principal
in te scool nominated by its district as GigH in terms of data use +as more active in +or"ing
+it teacers to revie+ teir data and discuss instructional implications tan +ere te principals
in scools described by te district as GemergingH or GtypicalH in terms of data use. $n general,
o+ever, scool7level supports for data use +ere more similar +itin tan across districts.
Tools for 8enerating and 9rgani:ing Data
!noter +ay for districts to promote data7informed decision ma"ing is to provide tools for
generating and organi,ing data. Data systems tat provide student acievement data in a form
tat can be disaggregated by student category troug standard or custom reports elp reduce te
burden of analy,ing data for scool staff members. Some systems can provide data not only from
standard and formative assessments but also from daily transactions suc as scool attendance or
from receipt of special services suc as tutoring.
%%

)e case study districts represent a range of system capabilities +it regard to providing
formative assessment data, but all provided some form of reporting of student scores bro"en
do+n by s"ill areas tat scools could use for planning purposes. @our districts ad data systems
tat incorporated student scores on interim assessments tat +ere aligned +it te state
standardi,ed tests, and a fift district ad some of its interim assessment data available troug
its data system. )+o districts provided interim assessments tat +ere not aligned +it te state
test, and anoter t+o districts +ere developing bencmar" assessments tat +ill be available in
te data system at some future date. Even toug all districts ad systems tat allo+ed analysis
of student performance in specific s"ill areas, some district systems +ere more robust tan
oters. @or e3ample, in one district, s"ill areas +ere sometimes covered by only one item on te
assessment, ma"ing it easy for teacers to dismiss te assessment data. $n anoter district, s"ill7
level analysis of test data +as possible for grades C and above, but not for primary7grade
students. )e systems in t+o districts provided standard reports by s"ill area, and tey system in
a tird district allo+ed custom reports to be generated.
Online assessments +ere less common in case study districts, +it only t+o districts
%%
)e %&&?>&E 5)! district survey +ill provide prevalence estimates for tools for gatering data 1Fuestion E2.
%=
providing online assessments for any subDect. Several oter districts tat did not provide online
assessments did ave oter tecnology supports for assessment. )ree ad digital item ban"s tat
allo+ed scools to develop teir o+n formative assessments, eiter in addition to or in place of
district bencmar" tests. Oter districts used andeld computers, +ic allo+ed teacers to
enter data as tey administered tests and to generate reports instantly. $n te tree districts +ere
use of andeld computers +as a relatively common practice, andelds +ere used for reading
assessments in te early grades.
9rgani:ational Structures Supporting Data Use at the School &e#el
Organi,ational structures tat support data use at te scool level can include 1a2 time set
aside for teacers to revie+ and discuss data in small groups, 1b2 designated support staff and 1c2
te adoption of procedures for discussing data. On te %&&=>&? 5E))S teacer survey, most
teacers reported aving positive perceptions of support for using and interpreting te data tat
tey could access from data systems. ! maDority of teacers +it access to a student data system
1?8 percent2 agreed or agreed strongly tat +en tey needed elp ma"ing sense of te data in
te system, tey "ne+ someone +o could elp, and AB percent reported tat someone else
usually e3tracted te relevant data from te system for tem.
Of tose teacers +o indicated on te 5E))S %&&=>&? teacer survey tat tey ad
access to a student data system, BE percent reported tat tey received support for using data to
guide instructional decisions from professional development received at teir scool and
B= percent reported receiving encouragement for using data from teir scool principal. !
9uarter of tese teacers 1%B percent2 also reported receiving support from a consultant or mentor
teacer s"illed in data analysis.
-or"ing in small groups can also support teacer use of data. Of te teacers +o
indicated tat tey ad access to a data system on te %&&=>&? 5E))S teacer survey, ?&
percent said tat tey +ere comfortable aving teir colleagues around +en tey e3amined
performance data for teir students, and B' percent said tat tey "ne+ o+ to +or" +it
colleagues in using student data to monitor progress and set goals.
$n many districts, teacers +ere not compensated for te time tey spent +or"ing +it data.
6ess tan a 9uarter of teacers 1%C percent2 ad time available during te regular day for
e3amining data, and a maDority 1B' percent2 reported accessing te data system on teir o+n
time. Only 8% percent reported tat tey ad oter paid time set aside for e3amining student data
and using data to guide decision ma"ing about practice.
%C

Case study districts provided stronger support for data7informed decision ma"ing tan te
typical level indicated in te 5E))S teacer survey responses. !s mentioned in te preceding
capter, a maDority of case study districts ad used scool7based staff members to act as bridges
bet+een teacers and te data system. Si3 out of nine offered some sort of district7funded,
scool7based staff to support teacers/ data use, but tese staff varied in number, amount of time
available and e3pertise. One district ad Dust t+o district7level coaces to serve all 8& of its
elementary scools, +ile oters ad a coac assigned to eac scool. Scool7based coaces
%C
)e %&&?>&E 5)! district survey +ill provide prevalence estimates for district supports for scool use of data
systems to inform instruction 1Fuestion %&2.
%?
+ent by different titles suc as data facilitator, tecnology specialist, system certified trainer or
staff development teacer and often ad oter responsibilities suc as instruction or serving as
te curriculum specialist in reading, mat or early cildood education. $n some cases, tere +as
additional personnel or more staff time provided to scools troug te support of federal
programs suc as )itle $ #eading @irst. )e activities of scool7based support staff members are
described furter in E3ibit C7C.
Ehi!it 7-7
Acti#ities of School-6ased Support Staff Mem!ers
2ne key to the use of data to impro'e instructiona practice at the eementary e'e in this
arge district !.istrict 4# is the a'aiabiity of fuAtime instructiona ad'isors !supported by 3ite 7
funds# and certified data system trainers+ 7nstructiona ad'isors combine e6pertise in
accessing data, understanding data, and designing instructiona practices to address student
needs identified by a'aiabe data+ )ertified trainers are schooAbased staff members 1ho are
trained by the district to support the use of the district*s data system+ 3rainers are in'o'ed in
ongoing professiona de'eopment by the district and an annua fuAday certification update
during the summer+ )ertified trainers initiay pro'ided training to their coeagues on ho1 to
get data out of the system, then mo'ed to ho1 to read data, and are no1 focusing on taking
about data, using data and re'ie1ing data+ 3he e6tent to 1hich the certified trainer roe is
combined 1ith the instructiona ad'isor roe !not possibe in e'ery schoo# affects the
a'aiabiity of support to schoo staff members+ 3hat is, 1hen certified trainers are fuAtime
teachers, the time a'aiabe for supporting their coeagues is more imited+
31o arge districts ha'e instituted a combination of schooAbased support staff and district
poicies such as common panning periods focused on data discussions to encourage dataA
informed decisionAmaking practices at their schoos+
.istrict 2 is focusing its efforts on schooAbased professiona de'eopment through teams of
speciaists pro'ided to each schoo+ 3here is a fuAtime technoogy speciaist in e'ery schoo
1ho has responsibiity for training schoo staff members on using the district data system as
1e as on the use of technoogy in instruction and the use of assessment toosH the speciaists
ha'e recei'ed training on the district benchmark assessments+ !Principas and secretaries are
aso trained on ho1 to access the system+# 7n addition, there are assessment and instructiona
coaches 1ho support schoo staff members+ A high schoos ha'e fuAtime assessment
coaches 1ho 1ork 1ith teacher eaders and principas on data access, data interpretation and
instructiona panning+ 3he assessment coaches aso hep manage testing systems+ 3here are
20 instructiona coaches 1ho pro'ide fuAtime support to the neediest schoos, 1hie other
schoos share a coach+ 7nstructiona coaches ha'e e6pertise in curricuum and pro'ide
teachers 1ith instructiona strategies based on test resuts+ )oaches aso train and support
teachers on ho1 to use the district data system and anayze their data+ A district eementary
schoos cose eary on &onday afternoons so teachers can meet either 1ithin their grade e'e
or across grade e'es to e6amine data and coaborati'ey pan instruction+
.istrict 5 pro'ides iteracy and math coaches to a its eementary schoos !athough their
numbers are shrinking because of decining funding from federa programs#+ )oaches are
responsibe for carifying data reports 1ith teachers, re'ie1ing data trends, and modeing
essons and practices for teachers+ Principas are re?uired to print out and dei'er data reports
to their teachers on their students* academic performance+ 3he district has incorporated data
discussions into 1eeky teacher common panning times !one hour e'ery 1eek#+ 7n eementary
schoos, these common panning times amost a1ays in'o'e reading data dispays,
interpreting data, and using data to change cassroom practices+ )oaches aso hod afterA
schoo 1orkshops to ook at schoo data+
%E
(rofessional De#elopment and Technical Support
)eacer professional development focusing on data analysis s"ills or data7informed
decision ma"ing is a relatively recent penomenon, but districts appear to be responding actively
to te need for tis "ind of teacer support. !lmost tree79uarters 1?C percent2 of all districts
responding to te %&&=>&? 5E))S district survey indicated tat tey supported professional
development in te past 8% monts Gto elp teacers and administrators in data7driven decision
ma"ing.H
#ecent years ave seen an increase in te proportion of teacers receiving professional
development on data7informed decision ma"ing. On te %&&B 5E))S teacer survey, C& percent
of teacers said tat tey ad received professional development from teir scool on tis topic,
and ' percent said tey ad received tis "ind of support from anoter source. On te %&&?
survey, te proportions receiving professional development on data use rose to AC percent of all
teacers for professional development from one/s scool and 8B percent for development from
oter sources.
)eacer survey responses suggest tat teacers receiving professional development in tis
area vie+ it as beneficial. On te %&&=>&? 5E))S teacer survey, C' percent of teacers +it
access to a data system reported tat te professional development tey received about data7
informed decision ma"ing ad prepared tem to use data to improve student acievement.
!t te same time, most teacers reported tat tey +ould benefit from additional
professional development on data7informed decision ma"ing 1see E3ibit C7A2. @ifty7eigt
percent of teacers +it access to a data system tougt tat additional professional development
on o+ to develop diagnostic assessments for teir classes +ould be beneficial, and BB percent
said tat tey +ould li"e additional professional development on adDusting te content and
approac used in teir class in ligt of student data. !lmost alf 1AE percent2 reported te need
for more professional development on te proper interpretation of test score data, and over a
tird 1CE percent2 for more professional development on o+ to formulate 9uestions tat can be
addressed +it data.
%A
<alf 1B& percent2 reported tat tey could benefit from additional
professional development on o+ to identify types of data to collect to monitor scool progress
against goals for improvement, and AA percent reported a need for more professional
development on te mecanics of using teir data system. Over a tird 1C? percent2 of teacers
+it access to a data system said tat tey +ould li"e professional development on tecni9ues
for collaborating +it colleagues on te use of data.
%A
!s noted earlier, only CC percent of teacers reported feeling capable of forming 9ueries of te data system
available to tem. $t may be tat some teacers feel tat tey can formulate 9uestions tat can be addressed +it
data 1and ence do not see a need for more professional development on tis topic2, but find teir data system
difficult to use because of interface issues.
%'
Ehi!it 7-;
Topics for Additional Teacher (rofessional De#elopment
Topic
(ercentage of
Teachers )ho
)anted More
(rofessional
De#elopment
Go1 to de'eop diagnostic assessments for your cass 5-
Go1 to adIust the content and approach used in your cass in ight of
student data
55
Go1 to identify types of data to coect to monitor schoo progress
against goas for impro'ement
50
Proper interpretation of test score data 4-
3he mechanics of using the eectronic data system 44
Go1 to formuate ?uestions that can be addressed 1ith data $-
3echni?ues for coaborating 1ith coeagues on the use of data $7
E3ibit reads. !mong teacers +it access to an electronic student data system, BE percent indicated
tat tey +ould benefit from additional professional development on o+ to develop diagnostic
assessments for teir class.
Source. %&&=>&? 5E))S teacer survey.
5E))S survey data also so+ tat teacers/ desire for professional development depends
on teir personal confidence in using a data management system, te perceived support for
system use, and +eter or not teir scool is ma"ing ade9uate yearly progress 1!KP2. @or
e3ample, a significantly iger percentage of teacers in scools not meeting !KP indicated tat
tey +ould benefit from additional professional development in si3 of te seven types of training
listed in te survey compared +it teacers in scools ma"ing !KP. )eacers +o e3press a
lo+er confidence in teir ability to interpret data and to use a data management system are more
li"ely to believe tat tey could benefit from furter professional development.
%B
!ll case study districts provided some form of district+ide professional development on
data use, but it varied in terms of +o received te training as +ell as te training content,
duration and format. @or e3ample, E3ibit C78 1presented earlier in tis capter2 describes o+ a
district provides school leadership teams with the opportunity to come together annually to
review and discuss their state test data and provides them with the tools to carry out this process
on an ongoing basis to monitor their school improvement goals. In another district, university
staff members were brought in to provide professional development to kindergarten through
grade 3 teachers on how to disaggregate and interpret early literacy assessment data. !n e3ample
of district7supported professional development tat combines an online professional
development system and in7person support is provided in E3ibit C7B.
%B
)e survey included a bloc" of 9uestions as"ing teacers to reflect on teir use of data management systems
using a B7point 6i"ert response scale. !dditional information on tese data can be found in Teachers$ Use of
Student Data Systems to Impro#e Instruction %&&' to %&&( 1U.S. Department of Education %&&Eb2.
C&
Ehi!it 7-<
(rofessional De#elopment 9nline and In (erson
Professiona .e'eopment 2nine !P.2# 1as buit Iointy by .istrict $ and a pri'ate company+
3his eectronic system dei'ers training on demand to a district staff members and ao1s
them not ony to 'ie1 their professiona de'eopment pan for the year but aso to manage
their training+ 3he district incudes re?uired courses in the system, and staff members can
monitor their courseAtaking progress+ Professiona de'eopment information is taiored to each
indi'idua staff member according to his or her position through targeted announcements,
information and highighted dispays of re?uired and recommended courses+ 7n addition, the
P.2 0eb site incudes training on ho1 to use the system !P.2 Gep#, professiona
de'eopment resources, and a ist of professiona de'eopment conferences+
As part of its ongoing professiona de'eopment acti'ities, the district aso assigns a staff
de'eopment teacher to e'ery schoo+ Ge or she focuses on heping teachers impro'e their
instructiona practice+ Staff de'eopment teachers in an eementary schoo, for e6ampe, might
meet 1ith the thirdAgrade teachers as they are panning a esson, obser'e a thirdAgrade
teacher, 1ork on instruction and esson panning, or mode a esson for some teachers+ Staff
de'eopment teachers aso 1ork 1ith teachers to e6amine and anayze data and to design
instructiona programs based on data+
)ree of te case study districts +ere focusing teir professional development activities on
data use primarily at te scool level. )e presence of scool7based support staff members
provides opportunities for professional development to be ongoing and tailored to individual
teacer needs. !s noted above, si3 case study districts provided some sort of scool7based staff
to support data use. )eir role included providing professional development to teacers troug
mentoring and serving as a resource to teacers on o+ to analy,e and interpret data. Scools
also spent some of teir professional development time on analy,ing scool7 and grade7level
data, supported by informal data7related discussions at grade7level or department meetings.
One district +as e3ploring te issue of scool7 versus district7based professional
development, +ile t+o districts planned to maintain a balance of district7 and scool7based
training. !noter district +as moving to+ard more tailored training on o+ to use te district
data system. -en user statistics indicated tat tere ad been a decline in use of te system, te
district decided to offer professional development to individual scools based on teacer needs
1e.g., o+ to e3tract te data tey need from te system2.
Tools for Acting on Data
Data systems can elp promote data7informed decision ma"ing by providing tools to elp
teacers improve decisions about instructional practice. Some of te resources provided include
instructional materials, model lesson plans, and formative assessment results lin"ed to
frame+or"s and curriculum guides.
%=
Data from te case studies suggest tat tese types of tools
are currently not common resources in district systems. Only tree case study districts ad data
systems tat provided suc resources, and one district +as developing a system +it tis
capacity. $n one of te tree districts, its commercial data system offered modules tat included
%=
)e %&&?>&E 5)! district survey +ill provide prevalence estimates for tools for acting on data 1Fuestions ? and
E2 and te fre9uency +it +ic district staff use teir data systems to carry out data7driven decision ma"ing
activities 1Fuestions 8E and 8'2.
C8
curriculum and lesson planning features along +it an assessment ban" tat allo+ed te district
to administer bencmar" tests tied to state standards and to display results to so+ +ic
students passed a particular obDective, generating s"ills and item analysis reports. $n anoter
district, te data system contained curriculum information aligned +it state and local standards,
instructional frame+or"s for teacers, and a curriculum guide to elp teacers +it teir pacing
and content coverage. )eacers could access assignments and lesson plans online so tey could
sare resources to supplement instructional materials.
)o lin" formative test data +it instructional resources, te data system must ave bot
components. Only four of te nine case study districts ad bencmar" assessment data on teir
systems 1one additional district ad some of its bencmar" data on te system2, and only one of
tese districts ad instructional resources on te system tat +ere aligned +it te assessments.
Scool staff members in several districts used systems outside te district for tis purpose. @or
e3ample, one of te districts used a 5ort+est Evaluation !ssociation 15-E!2 test as a
bencmar" assessment, and te 5-E! -eb site provides instructional resources tied to eac
s"ill area in its assessments.
One of te barriers illuminated by te case study data +as te need for greater alignment
bet+een te district curriculum and bencmar" assessments. 6ac" of alignment presented
callenges to districts trying to elp scools lin" formative assessment data to instructional
practice. One case study district ad undergone a curriculum audit by te state tat identified as a
+ea"ness te lac" of a clear, compreensive curriculum aligned +it standards. !s a result, te
district +as developing a uniform curriculum lin"ed to state standards and supported by
bencmar" assessments tat +ere tied to bot te curriculum and state standards. )e audit +as
also te impetus for te district/s selecting a data system +it a curriculum management
component so instructional resources +ould be available to teacers for s"ill areas in +ic teir
students are identified as being belo+ proficiency on te basis of bencmar" data. Until te ne+
district curriculum is completed, teacers cannot ta"e full advantage of tis component. $n
anoter district, tere +as a common district+ide curriculum, but te bencmar" assessments
+ere not +ell aligned +it te curriculum: te mat assessment did not follo+ te district
curriculum scope and se9uence, and te language arts assessment did not cover all te s"ills in
te district curriculum, ma"ing it more callenging to support instructional decision ma"ing. )e
district reported tat tese alignment problems +ould be rectified by a ne+ data system tat +as
under development.
Summar$ of District and School Supports
Case study districts supports for data-informed decision making are summarized, by
district, in Exhibit 3-6. For each type of support, a district was given a rating: the support was
widely available in the district (coded as 2); the support was partially present, for example,
professional development provided to some staff members but not all (coded as 1); or the support
was not currently present in the district (coded as 0). The districts are ordered by size, from
largest to smallest. To some extent, the larger districts (those with student enrollments of 25,800
or more) provide a greater number of supports than the medium-size districts. The mean number
of fully present supports in very large and large case study districts is 5, compared with an
average of 3 in medium-size districts. But there are exceptions to the general pattern. District 1, a
very large district of more than 130,000 students, provides the smallest number of widely
C%
available supports because of recent resource constraints (e.g., budget cuts eliminated
instructional coach positions). District 8, a medium-size district, has benefited from external
funding sources and expertise (e.g., EETT funds, regional consortia) and provides the largest
number of supports compared with the aforementioned very large district.
Exhibit 3-7 summarizes the supports available at each case study school, organized by
district. Each school was rated on the extent to which each type of support was available at the
school: the support was fully present (coded as 2); the support was partially present, for example,
a principal who occasionally worked with teachers (coded as 1); or the support was not present
(coded as 0). Analysis indicated that there was greater consistency within districts than within
school type (high, emerging, typical) across districts. The average number of fully present
supports was 3.89, 4.11, and 3.78 for high, emerging, and typical schools, respectively. Averages
across schools within individual districts ranged from 1.33 to 5.33. This pattern suggests that
districts play a pivotal role in determining the level of support that teachers receive for the use of
data systems and making data-informed decisions.
CC
Ehi!it 7-0
Case Stud$ District Supports for School Use of Student Data
District
=o- ' + 7 ; < 0 2 > ?
Support
District
Si:e @& @& @& & & M M M M
Strong district eadership
for data use
1 2 2 2 2 2 1 2 2
A'aiabiity of data
disaggregated by student
groups and ski e'es
1 2 2 1 2 1 2 1 1
.istrict1ide dataAinformed
decisionAmaking acti'ities
0 0 1 2 1 0 0 2 0
Professiona de'eopment
for data system
impementation
1 2 2 2 1 2 1 2 2
Pro'ides schooAbased
staff members to support
dataAinformed decision
making
0J 2 2 2 2 2 0 2 0
5i'es teachers direct
access to e6tract data
from system
1 2 2 2 0 0 0 1 2
.istrict benchmark or
formati'e assessment
data on the system
2 2 2 0 2 1 2 1 0
System incudes inks
from assessment resuts
to instructiona resources
0 0 2 0 2 0 0 0 0
Students can take
assessments onine
0 2 1 0 0 0 0 0 0
3ota number of supports
fuy present
1 7 7 5 5 $ 2 4 $
Exhibit reads: In District 1, strong district leadership for data use was partially present in 200607that is, only
some departments (not all) were strong advocates for data use.
5otes. District si,e categories include very large 1L62, +it more tan 8C&,&&& students, large 162, and medium
1M2. Codes indicate te degree to +ic eac support +as present in te district. & M not present, 8 M partially
present, % M fully present0+idely available. )e N indicates tat te district used to provide scool coaces, but tis
position +as not present in %&&=>&? because of budget cuts.
CA
Ehi!it 7-2
Case Stud$ School Supports for Use of Student Data
School Support
District ' District + District 7 District ; District <
* E T * E T * E T * E T * E T
Principa sets e6ampe for data
use
2 2 2 2 2 1 1 2 0 2 2 2 2 2 2
Principa 1orks 1ith teachers
on data
2 2 2 2 2 1 0 2 0 0 2 2 0 0 0
SchooAbased staff members
a'aiabe to support data use
and@or other schoo eaders
1ork 1ith teachers on data
0 1 0 2 2 2 2 2 2 2 2 2 2 2 2
Specific procedures in pace to
discuss@act on data
1 0 0 1 0 0 2 1 1 1 1 1 1 2 2
Schoo has o1n professiona
de'eopment on data use
2 1 2 2 2 2 2 2 2 1 2 2 1 1 1
3ime aotted during schoo
day for teachers to 1ork 1ith
data
2 2 2 2 1 2 2 2 2 2 2 2 2 2 2
3ota number of supports fuy
present
4 $ 4 5 4 $ 4 5 $ $ 5 5 $ 4 4
School Support
District 0 District 2 District > District ?
* E T * E T * E T * E T
Principa sets e6ampe for data
use
2 2 1 0 2 0 2 2 2 2 2 2
Principa 1orks 1ith teachers
on data
2 2 0 0 1 1 2 2 2 2 2 0
SchooAbased staff members
a'aiabe to support data use
and@or other schoo eaders
1ork 1ith teachers on data
2 2 2 2 0 0 2 2 2 1 0 0
Specific procedures in pace to
discuss@act on data
1 1 1 1 0 0 2 2 2 1 1 1
Schoo has o1n professiona
de'eopment on data use
2 2 2 1 0 1 1 1 2 2 0 2
3ime aotted during schoo
day for teachers to 1ork 1ith
data
2 2 2 2 1 2 2 2 2 2 2 2
3ota number of supports fuy
present
5 5 $ 2 1 1 5 5 6 4 $ $
Exhibit reads: In District 1, at the high-data-use school, the principal set an example for data use and also worked with teachers on using data.
5otes. Codes indicate degree to +ic eac support +as present in te scool during %&&=>&?. & M not present, 8 M partially present, % M fully present. Scools
+ere defined by district staff members for teir level of data use. < M ig data use, E M emerging data use, ) M typical data use for te district.

3
5


;- Teachers5 (reparation for Data Use
!s noted in te preceding capter, staff members in case study districts e3pressed
reservations about teacers/ ability to obtain student data from systems or to ma"e sense of
student data reports provided to tem. Understanding te nature of teacers/ proficiencies and
difficulties in tis arena is important for te design of bot data7informed decision7ma"ing
processes 1e.g., +eter or not to ave specialists serve as mediators bet+een teacers and
data systems2 and of teacer education and professional development programs. )o provide
insigts into areas of teacer strengt and +ea"ness +it respect to using data, proDect staff
incorporated data scenarios into case study activities as a conte3t for probing teacers/ ability
to engage in various aspects of data interpretation and use. )e data scenarios +ere presented
individually to si3 teacers at eac scool.
%?

De#elopment of the Data Scenarios
)e rationale beind te development of te data scenarios +as to ave a standard set of
prompts tat could be used to elicit teacers/ tin"ing about student data and reveal te
concepts and s"ills tat tey can bring to data7informed decision ma"ing. Study staff
members assembled a group of internal and e3ternal e3perts in assessment and data7informed
decision ma"ing, including t+o assessment e3perts, an e3pert on te use and functionalities of
student data systems, a leading researcer in te area of matematics education, and t+o
researcers +o ad performed doctoral or postdoctoral researc on te use of student data
systems to inform educational decision ma"ing.
%E

-or"ing +it tis group, te study/s principal investigator first identified maDor
processes involved in using student data to inform scool7level decisions. )ese processes
included te follo+ing.
Question posing. $n cases +ere a data system is available for e3ploration and
educators must decide o+ to use it in a +ay tat can improve teir practice, te
educator needs to be able to frame a 9uestion tat can be addressed +it available
data. Doing so re9uires an appreciation of te "ind of 9uestion tat can be
ans+ered +it data and alignment bet+een te 9uestion/s intent and te measure
and student group specified in te data 9uery.
Data comprehension. Data are typically displayed in tables, graps or printouts,
+ic can be 9uite comple3, and finding te desired data element is not a trivial
matter. @urter, te educator needs to understand te data display so e or se can
ans+er te 9uestion, G-at do te data say;H
Data interpretation. Beyond te issue of literal data location and compreension,
particular elements of data compreension re9uire at least a 9ualitative
understanding of basic concepts in statistics: for e3ample, an educator +o does
not appreciate te effect of an e3treme score on a sample mean can be misled by
%?
Before te site visit, principals +ere as"ed to select si3 teacers to participate in te data scenario intervie+s.
tree teacers +o actively use data to inform decision ma"ing and tree teacers +o represent average data
use +itin te scool.
%E
)is group comprised )-( members *eff -ayman and Ellen Mandinac, as +ell as *ere Confrey 1ten of
-asington University2 and S#$ staff members Eva Cen, (eneva <aertel, and Li"i Koung.
C?
data sets +it outliers.
Data use. @or student data and data systems to ave a positive influence on
instruction, data must be not only located and properly interpreted but also lin"ed
to appropriate instructional options. )e educator must ave alternative
instructional strategies or coices available and a +illingness to use data to inform
decisions.
@or eac of tese components of data7informed decision ma"ing, e3pert7group members
identified specific s"ills and concepts tat teacers sould ave to e3ecute tis aspect of data
use successfully.
!fter identifying "ey s"ills and concepts for using data, te group brainstormed
e3amples of situations or 9uestions tat +ould call on eac of te concepts and s"ills in
9uestion. Screen sots from actual data systems and 9uestions tat ad been used in prior
researc or teacer education also +ere revie+ed by te group as possible item models.
)e list of priority concepts and s"ills developed by te e3pert group and te e3ample
situations calling on tose concepts and s"ills +ere ten used by te researc team as a
starting point for generating eigt data scenarios 1see !ppendi3 B2. Draft scenarios +ere
revie+ed by an assessment e3pert and a matematics education e3pert from te group for
plausibility, accuracy and alignment +it te identified s"ills and concepts.
One of tese revie+ers pointed out tat data compreension items could be classified into
t+o types. Some items called simply for finding te correct data element in a display. )ese
+ere subse9uently labeled data location items. Oters added a degree of difficulty by
re9uiring te manipulation of displayed data 1suc as te addition of t+o percentages or te
computation of a proportion2 to ans+er a 9uestion about +at te grap or table so+ed.
!fter te revie+, te term data comprehension +as reserved for tis latter item type. E3ibit
A78 so+s te final set of concepts and s"ills identified by te group of e3perts.
)e assessment scenarios +ere pilot7tested first +it former teacers on S#$/s staff and
ten +it practicing teacers, +it revisions made after eac round of pilot testing. )o cover
all of te identified s"ill areas +itout e3tending intervie+s to an intolerable lengt,
developers created tree protocol Gforms,H eac +it five of te eigt scenarios. )e amount
of content on te various forms +as balanced to acieve rougly e9uivalent average
administration times, estimated at C& minutes eac.
CE
Ehi!it ;-'
Data-Informed Decision-Making Components, Concepts and Skills
Component Target ConceptASkill
Kuestion Posing Aigns ?uestion 1ith purpose and data
%orms ?ueries that ead to actionabe data
Lnderstands iterati'e nature of data e6poration
Lnderstands 'aue of mutipe measures
.ata /ocation %inds ree'ant ces in a compe6 tabe
%inds ree'ant ces in a compe6 figure
.ata )omprehension &anipuates data from a compe6 tabe or graph to support
reasoning
&aps bet1een data in a tabe and a prose representation of
the data
&aps bet1een data in a figure and a prose representation of
the data
Lnderstands a histogram as distinct from a bar graph
7nterprets a contingency tabe
.istinguishes bet1een ongitudina and crossAsectiona data
>'idences data comprehension monitoring !metacognition#
.ata 7nterpretation Lnderstands the ad'antages and disad'antages of using
disaggregated subgroup data 's+ indi'idua student data
Attends to distribution and e6treme ?uarties, not Iust mean or
proportion abo'e cut score
Appreciates effect of a fe1 e6treme scores on the mean
<eaizes that more items on a scae or members in a sampe
produce more precise, reiabe estimates
Lnderstands measurement error and 'ariabiityH resuts not
identica on e'ery testing
Lnderstands that student cohorts differ from year to year
.ata Lse Lnderstands ho1 to differentiate instruction based on data
Seeks subscae and item data that can be mapped to
curricuum
Lnderstands 'aue of formati'e assessments
C'
Scenario Administration
-itin eac of te %? scools participating in te %&&=>&? case study site visits,
principals +ere as"ed to identify si3 teacers for participation in intervie+s and focus groups.
)ey +ere as"ed to coose tree teacers +o +ere active in scool use of data to inform
instruction and tree teacers +o +ere typical of te scool teacing staff in tis regard. )e
data scenarios constituted te second part of te individual intervie+s for tese teacers. @or a
maDority of scools, si3 teacers participated in te study as planned. @or a fe+ scools, it +as
possible to meet +it only t+o to four teacers because of scedule conflicts. $n total, 8A?
teacers from %? scools responded to a set of data scenarios in spring %&&?. Eac form
included scenarios addressing multiple components of data7informed decision ma"ing.
Scenarios contained a combination of open7ended and structured7response items. )e data
scenarios varied in te number of items tey contained, but eac included items tat could be
scored using eiter dicotomous or partial7credit rubrics.
%'
E3amples of items representing
eac stage in using educational data for decision ma"ing are presented in E3ibit A7%. !ll
items and rubrics, as +ell as item allocations to forms, are included in !ppendi3 B.
Ehi!it ;-+
Eamples of Items ,elated to Each Component
Component Item (rompt %ull-Credit ,esponse
Kuestion
Posing
So no1 in Manuary 2007, 1hat specific
data 1oud you 1ant to get from this
system to hep you decide ho1 to
impro'e your fourth graders*
performance4
3eacher picks a ogica group and
seects a ogica measure for that
group+
.ata /ocation 0hat 1as 2ak Schoo*s a'erage 3ota
&ath Score in 200$"044
3eacher pro'ides correct ans1er
from graph 1ithin 5Apoint range+
.ata
)omprehension
2ak Schoo*s progress in narro1ing the
grade 4 math achie'ement gap 1ith the
rest of the district has been in probem
so'ing rather than computation+
!3eacher must agree or disagree and
e6pain reasoning+#
Agrees+ 3eacher describes data
iustrating that the gap bet1een
2ak Schoo and district scores
decreased more for probem
so'ing than for computation+
.ata
7nterpretation
3hese data suggest that ne6t year*s
thirdAgrade Asian@Pacific 7sand girs 1i
score better than other third graders on
this test+
3eacher disagrees and e6pains
that students in a subgroup 'ary
from year to year and you cannot
generaize based on Iust a singe
Asian student the prior year+
.ata Lse 3eachers shoud obtain a detaied
breakdo1n of ast year*s test resuts by
item or content standard+
3eacher agrees 1ith a reasonabe
e6panation about why ha'ing
detai is important+
%'
!nalysis of oter aspects of te protocols is ongoing, using a 9ualitative coding sceme to capture additional
information about teacers/ tin"ing. @indings from te 9ualitative coding +ill be presented in te final report.
A&
Measures
Multiple 9uestions +ere associated +it eac data scenario. Most 9uestions +ere
designed to get at a specific concept or s"ill and to ave rigt and +rong ans+ers. ! teacer/s
response to eac of tese 9uestions +as assigned a 8 if correct and a & if incorrect. @or some
items, tere +ere intermediate or partially correct responses for +ic a .B& +as assigned. $n
addition, some 9uestions +ere more open7ended, deliberately designed to elicit teacers/
tin"ing. #esponses to tese 9uestions are not included in te scores reported ere. )eacer
responses to tese open7ended 9uestions ave been transcribed and are being coded for
evidence of types of data7informed decision7ma"ing s"ills 1e.g., indications of compreension
monitoring, descriptions of o+ data +ould be used to differentiate instruction2.
(rocedures
*ntervie"er training. Site visitors +ere involved in a full7day training session tat
included an overvie+ of te study/s conceptual frame+or", te data systems used by eac
district, and te various data collection activities, including te data scenarios. Site visitors
+ere so+n a video of te administration of te data scenarios to illustrate proper
administration tecni9ues and ten given te opportunity to practice administering te data
scenarios to one anoter. )is activity +as follo+ed by a discussion period to ans+er
9uestions tat arose as a result of te practice session.
Data scenario intervie"s. )eacers participated in AB7minute intervie+s +it t+o
researcers. !ppro3imately te first 8B minutes of te intervie+ involved 9uestions
concerning te teacer/s personal e3perience +it te district/s data system, including
decisions e or se ad made on te basis of student data. )eacers ten responded to items
from one of te tree data scenario forms. )ey +ere told tat te study +anted to investigate
o+ different "inds of data displays are understood by teacers and +ere as"ed to tin" out
loud as tey loo"ed at te various data presentations and responded to 9uestions about tem.
-en te data scenarios +ere presented, one researcer +as responsible for as"ing teacers
9uestions from te assigned data scenario form +ile te oter researcer too" notes.
1)eacers +ere randomly assigned to forms before te intervie+.2 Eac form +as
administered to t+o teacers +itin a scool, if all si3 teacers from te scool participated in
te study. )eacers +ere provided +it copies of te graps, tables and screen soots
included in eac scenario. )eacers +ere also provided +it paper, pencils and calculators
tey could use as tey +ised 1e.g., to carry out basic aritmetic calculations2. !ll intervie+s
+ere audio7recorded and transcribed to facilitate later analysis.
Scoring. Before scoring, researcers revie+ed eac transcript to identify te beginning
and end of te discussion of eac item. Eac item segment +as coded +it an item
identification number in !)6!S.ti, a 9ualitative data analysis program. !)6!S.ti +as used
to produce data reports by item 1i.e., all responses for a given item2 to facilitate te scoring of
a single item at a time. @or eac item, raters revie+ed te rubric using a sample of transcripts
and ten refined te rubric on te basis of te range of teacer responses to te item.
)+o raters scored eac item, and at least C& percent of all item responses +ere double7
coded to permit te e3amination of interrater agreement. $nterrater agreement +as '& percent
or iger for %' items, bet+een E& percent and '& percent for four items, and ?B percent for
A8
one item. !ll discrepancies bet+een raters/ scores +ere resolved by consensus bet+een te
raters.
@or t+o of te scenarios 1tose focusing on Fuestion Posing and Data Use2, all but one
or t+o of te scenario7based 9uestions +ere open7ended. )e responses to tese and oter
open7ended 9uestions ave been transcribed and are being coded for evidence of te s"ills and
concepts of interest.
(reliminar$ ,esults and %indings
E3ibit A7C so+s te mean score 1percentage of full credit2 for eac data component for
+ic five or more items could be scored as rigt or +rong. )e data scenarios temselves
and te distribution of responses to tose 9uestions tat +ere scored as correct or incorrect are
included in !ppendi3 B. )e proportion of respondents giving correct ans+ers for eac
assessment item and information concerning te reliability of te data scenario assessment
items as a set are included in !ppendi3 C.
Ehi!it ;-7
%re.uenc$ Distri!ution and Mean (ercentage Correct for Scored Items 9#erall
and )ithin Each Data Component
Component
=um!er of
Scored
Items
Mean =um!er of
,espondents per
Item
Mean
(ercentage
Correct
Standard
Error
3ota score $$ -4 67 1+1$
Kuestion Posing 1 -5 ;A ;A
.ata /ocation 10 72 -1 1+61
.ata )omprehension 16 (0 64 1+5$
.ata 7nterpretation 5 -- 4- 2+60
.ata Lse 1 77 ;A ;A
Exhibit reads: )e average percentage of a scool/s teacers responding correctly across all CC data scenario
items scored +as =? percent.
5ote. 5! M $nsufficient data 1fe+er tan B items2.
)e data in E3ibit A7C suggest tat teacers are generally capable of finding specific
pieces of information in a data table or grap 1te mean percentage correct for Data 6ocation
items +as E8 percent2 but tat teir ability to translate bet+een a statement and a set of data
declines +en tey must do computation or ma"e multiple comparisons using te numbers
provided 1for Data Compreension items, te mean +as =A percent2. -en teacers are as"ed
to go one step furter and to reason about data issues tat involve basic statistical concepts
suc as variability, measurement error or distribution, teir li"eliood of ma"ing a correct
inference is still lo+er 1AE percent correct2.
)e data scenarios provide insigt into te nature of teacers/ misconceptions related to
use of student data. On Data $nterpretation items calling for teacers to derive predictions
from data, for e3ample, a maDority of teacers relied on subgroup means +itout
A%
consideration of te scores for specific students at ris". -en loo"ing at a scool/s or
district/s multiyear data, a maDority of teacers lac"ed te understanding tat te students in a
particular grade coort differ from year to year. -en reasoning about te acievement trend
of an individual scool over time, teacers, on average, did not consider district+ide trends
+en interpreting year7to7year canges in te scool/s scores. )e ongoing 9ualitative
analysis of teacer responses +ill elucidate furter teacers/ reasoning about data.
Comparisons !$ District
Because different teacers responded to different data scenarios, depending on te form
administered to tem, teacer scores +ere aggregated to te scool level. Mean scool scores
+ere e3amined by district to determine +eter tere +ere any district7level differences in
performance 1E3ibit A7A2. $n general, te nine districts appeared comparable +it respect to
their teachers average performance on the data scenarios, but two districts did stand out as
potentially different. District 1 had the highest mean score, and District 5 had the lowest
score. A one-way ANOVA revealed that variance by district was not statistically significant
[F(8,18) = 2.41, p = .06,
2
= .52].
Ehi!it ;-;
Total A#erage Scores for Schools, !$ District
0+7(
0+67
0+61
0+5(
0+6(
0+70
0+65
0+66
0+66
0+00
0+10
0+20
0+$0
0+40
0+50
0+60
0+70
0+-0
0+(0
1 2 $ 4 5 6 7 - (
.istrict 7.
3
o
t
a

A
'
e
r
a
g
e

S
c
o
r
e
Exhibit reads: The total average score for schools in District 1 was 79 percent
correct.
,emaining Anal$ses
)e analyses presented in tis report provide initial insigt into variation in data7
informed decision7ma"ing s"ills for scools and districts. )e final report +ill contain
findings from te 9ualitative coding of teacer responses to open7ended data scenario
9uestions and provide greater detail about te types of "no+ledge and s"ills tat teacers use
as tey develop 9uestions, read data tables and graps, interpret data, and use data to ma"e
AC
instructional decisions. !lso, for te final report, te teacer data set +ill be e3panded +it
additional teacers intervie+ed in %&&?>&E and +it responses of small groups of teacers
responding collectively rater tan as individuals to te same or similar scenarios.
Summar$
)e data in E3ibit A7C suggest tat teacers generally can locate specific information in
a data table or grap but tat tey e3perience increasing difficulty if tey need to manipulate
te data in some +ay or ma"e comparisons bet+een multiple data points. 5ecessary
manipulations re9uired only basic aritmetic 1subtraction, division2, and teacers +ere
provided +it paper and pencil and +it calculators tey could use if tey +ised. Even +it
tese tools, cognitive researc +ould suggest tat te cognitive load re9uired for "eeping in
mind te issue at and, multiple data points, and intermediate products of data manipulations
ave a role in te observed deterioration of teacers/ accuracy 1!yres %&&82. Upcoming
administrations of data scenarios to small groups of teacers +ill test +eter greater
accuracy in data compreension is acieved by spreading te compreension burden across
multiple individuals.
$n terms of te ability to interpret data, teacers demonstrated limited understanding of
basic statistical concepts suc as variability and reliability. )eacers appeared to ave
particular difficulty 1a2 using individual students/ scores, rater tan teir subgroup mean, as
te basis for predictions and instructional prescriptions and 1b2 "eeping in mind te fact tat
student groups differ from year to year +en loo"ing at multiyear data.
AA
<- Summar$ of the Earl$ %indings
)e concept of using data as a basis for teir decisions is novel to many scool staff
members, and tere are multiple obstacles to institutionali,ation of tis practice. By collecting
data at te scool and district levels, te researc team sougt a Gbottom7upH perspective on
te 9uality of te supports for data7informed decision ma"ing coming from te state and
national levels. Policy7ma"ing in tis area began +it te assumption tat federal
re9uirements and national campaigns 1e.g., te Data Fuality Campaign, grants to states for
enancement of teir data systems2 +ould lead to iger79uality state data systems, +ic in
turn +ould ave a positive influence on te use of data in districts, scools and classrooms.
Although federal and state policies have spurred districts and schools to initiate more
data-informed activity, the study found no evidence of a tightly coupled system. )e vast
maDority of te activity around data systems and data use in districts and scools involves
district, not state data systems, according to te 5E))S survey data. )+o of te case study
districts +ere in states +it systems tat use a uni9ue student identifier tat te district system
also used, facilitating transfer of data across systems. But for te most part, te oped7for
efficiencies to be gained from integrating data systems at te state, district and scool levels
are not apparent from te vantage point of scools and districts. More progress in tis area can
be e3pected as ne+ state systems come on line.
Direct district use of a state7provided soft+are system appears to be relatively
uncommon. )is practice +as found in only one of te nine case study districts. $n tis case,
te state made available a performance improvement mapping system tat te district could
use to obtain test items lin"ed to areas +ere te scool needed to improve. $n anoter case,
te district +as using a state soft+are system for its data +areouse, but te state ad since
moved to a ne+ soft+are system +it additional capabilities, and te district +as left +it an
GorpanH system.
!mong te nine case study districts 1dra+n from nine different states2, only one cited
training on data7informed decision ma"ing made available troug its state education agency.
$n tis case, principal training and some funds for laptop purcases +ere made available by
te state.
$t sould be noted tat a tigtly coupled system is not necessarily te ideal solution in all
cases. )e fact tat no suc system +as found in any of te case study districts, selected on
te basis of perceived e3cellence in data use, suggests tat implementation of suc systems is
difficult. Muc of te value provided by a tigtly integrated system can arguably be provided
also by less tigtly coupled systems.
Case study informants were more likely to describe federal funds supporting data use
activities than state funds doing so. ! number of federal programs ave provided funds tat
ave paid for e3tra staff positions or oter activities to promote a data7using culture. $n one
district, for e3ample, )itle $ funds paid for full7time scool7based instructional advisors +o
elped teacers ac9uire and interpret data: te funds also supported te ig7speed $nternet
access needed to use te data system. !noter district used some of its )itle $ funds for
AB
professional development on te use of D$BE6S
C&
data for instructional decision ma"ing.
Scools in tree districts used #eading @irst grant funding for activities suc as collecting and
trac"ing longitudinal data on primary students/ reading s"ills and for funding reading coaces
+o elped teacers incorporate assessment data into teir practice. !n EE)) grant to a state
consortium for scool use of data +as led by one of te case study districts: te consortium
as provided professional development on data7informed decision ma"ing at no cost to te
district.
University partners are an additional class of facilitators. One district is +or"ing +it a
literacy proDect run by a teacers/ college to learn more about "inds of student data tat can be
collected and o+ tey can be used to guide subse9uent instruction. !noter district ad a
university come in to +or" +it its teacers on o+ to disaggregate and interpret data from a
ponological a+areness assessment.
(rogress in the Use of Data S$stems
)is study encompassed bot district and scool use of student data systems and teir
data use practices to improve instruction. !cross districts and scools in te case study
sample, use of locally generated data to inform instruction and use of electronic data systems
containing student scores on standardi,ed tests appeared to be t+o parallel initiatives rater
tan a single, integrated effort. District and scool use of standardi,ed test scores obtained
from data systems typically focused on accountability concerns and ma"ing sure tat local
curriculum and instruction +ere +ell aligned +it te state/s assessments. 6ocally generated
data often came from instruments connected +it an early literacy program or bencmar"
assessments. )e t+o data activities +ere not necessarily integrated, and scools ma"ing
progress in one of tese areas +ere not necessarily doing muc +it te oter.
Staff members at four of te nine case study districts reported aving identified areas in
need of improvement, corrections needed in curriculum alignment, or areas ripe for ne+
programs on te basis of analysis of data from teir data systems. )+o of te districts
described more mature data use, including using data on an ongoing basis to evaluate +ic
of teir initiatives +ere +or"ing and +ic +ere not. One of tese districts +on a state a+ard
as one of te most improved districts in te state.
)e 5E))S teacer survey and first round of case study data collection ma"e it clear
tat te implementation of data systems and practices for teir use in decision ma"ing at te
scool level is still in its infancy. Bet+een te t+o 5E))S teacer surveys, te proportion of
teacers reporting access to a student data system rose by nearly B& percent. )is large a
cange over a t+o7year period is 9uite remar"able, but te optimism it engenders must be
tempered by concerns over te perceived value of te data to +ic more teacers ave
access. On te latest 5E))S survey, te rougly t+o7tirds of teacers +it access to a
student data system +ere divided in teir perspective on +eter or not scool use of data ad
paid off. A% percent agreed tat teir scool ad been improved troug te use of data, +ile
AB percent neiter agreed nor disagreed +it te statement. Similarly, te case study
C&
)e Dynamic $ndicators of Basic Early 6iteracy 1D$BE6S2 are a set of standardi,ed, individually
administered measures of early literacy development. )ey are designed to be sort 187minute2 fluency
measures used to regularly monitor te development of pre7reading and early reading s"ills.
A=
intervie+s suggest tat e#en in districts )ith a reputation for leadership in using data,
electronic data systems are *arely influencing classroom-le#el decision making. <o+ever,
more progress can be seen in te use of data, coming from oter sources suc as diagnostic
tests included in early reading programs, to sape instruction.
Staff members at all %? case study scools described te use of student data tat +as not
contained on teir data system to guide instruction. @re9uently mentioned +ere data from
early reading assessments given in primary grades 1for +ic state assessment data are
usually unavailable2, performance reports from computer7based instruction, running records,
and e3aminations constructed by scool departments.
&essons &earned for Implementation
)e reflections of te case study informants and te pattern of activity across teir
districts and scools +it different practices suggest a number of guidelines for scools and
districts embar"ing on te implementation of data7informed decision ma"ing.
8. Districts are finding that providing school-level data coaches is an important support for
school-level use of data to inform instruction. $n te maDority of case study districts, some
or all scools ad on7site staff members designated to elp teacers retrieve data from te
data system, interpret data and ma"e instructional decisions based on data. 5ot all districts
started out +it tis intention, but it appears to be an emerging best practice tat bot
encourages more use of data and lessens te li"eliood of misinterpretation of data.
%. A common curriculum and curriculum-aligned benchmark assessments increase the
likelihood that school staff members will make extensive use of a districts data system.
State assessments are typically administered once a year, and scool revie+ and reflection
on te resulting data is typically also a once7a7year event. )eacers in all nine case study
districts loo"ed more fre9uently at interim, bencmar" or end7of7course assessments. $n
some cases, tese data +ere available on a district7provided system, but in many cases tey
+ere accessed on a test developer -eb site, eiter as reports generated by te assessment
soft+are itself or troug a scool7developed database. )o te e3tent tat te district data
system contains more current data tat teacers find relevant to teir instructional
decisions, teacers +ill ave a greater motivation to use te system.
+. !f teacher use of data is the goal, then it is desirable to have curriculum and instruction
staff members involved in the initiative. )e use of student data to inform instruction is
necessarily a systemic effort, lin"ing assessment and accountability, professional
development, information tecnology, and curriculum and instruction functions tat are
typically separate offices +itin medium7si,e and large districts. !ll of tese offices need
to support te initiative. $n t+o of te case study districts, te data use initiative +as led by
te assessment and accountability office, and oter district offices +ere not deeply
involved. $n tese cases, te data system +as not being used to sape instruction.
A. "eacher buy-in for the data system and its use should be sought early and maintained
continuously. !ttention to teacer buy7in +as not uniform across te case study districts.
On one end of te continuum +ere districts tat solicited input from a fe+ principals or
A?
teacers regarding +at tey +anted in te system and ten proceeded to procure and
adopt one. !t te oter end of te continuum +as a district tat instituted a nine7mont
process of convening sta"eolder groups, including teacers, to elp select te ne+ system
and involved te union in te media campaign for its launc. Several districts tat ad not
involved teacers early on +ere dealing +it teacer suspicions tat te data system +ould
be used to identify +ea" teacers and tat data +ould be used against tem.
B. #rofessional development should include training on how to interpret data and how to
translate data into changes in instructional practice. )eacers in seven of te nine case
study districts cited insufficient training on o+ to engage in data7informed decision7
ma"ing practices, o+ to read and interpret te data tat are given to tem, and o+ to
translate data into actionable plans for instruction. )eacer responses in te data scenario
intervie+s suggest tat tese concerns are +ell7founded. )eacers could read data tables
and graps but ad difficulty framing a 9uestion tat could be ans+ered troug a data
system 9uery, and +en tey did ave data in front of tem, tey ad a ard time dra+ing
defensible interpretations or inferences based on te data. On te 5E))S teacer survey, a
large proportion of respondents felt tey could benefit from additional professional
development around te use of data. BE percent tougt tey could benefit from
professional development on o+ to develop diagnostic assessments for teir students,
C8
BB
percent on o+ to adDust te content and approac used in teir class on te basis of data,
B& percent on +at data to collect to monitor scool progress against improvement goals,
and AE percent on o+ to interpret data.
=. District policies should be examined to identify and remove policies and procedures that
undermine teachers use of data to inform instruction. Despite a commitment to te
promotion of teacers/ use of data in ma"ing instructional decisions and individuali,ing
and optimi,ing students/ educational e3periences, districts often ave policies in place tat
run counter to suc efforts. !n e3ample is te imposition of district pacing plans tat do
not provide time for individuali,ation or selective reteacing based on bencmar" or
formative assessment findings. On te 5E))S )eacer Survey, of tose teacers +it
access to a student data system, =& percent said tat no matter +at te system tells tem
about teir students/ learning, tey must "eep up +it state or district pacing plans.
?. $chool leaders need to build teachers mutual trust to a point where teachers are
comfortable working with colleagues to examine data that reflect on their teaching
performance. Muc of te e3amination of data in case study scools occurred in small
groups4during common planning time or grade7level or department meetings. )is
practice +as particularly +ell ingrained in te scools identified as Gig useH by teir
districts. Staff members in seven districts said tat data7informed decision7ma"ing
practices ad moved scool staff members to+ard more open conversations about
instructional practice and increased opportunities to learn from one anoter. One principal
noted, GUsing data eliminates distracters and "eeps everyone focused on +ere tey need
to go. Data also elps to eliminate some of te interpersonal issues tat can arise: it/s not
about +o you are, it/s about +at te data so+s.H Some scools are actually
C8
)eacers are being encouraged to use diagnostic assessments but often e3press frustration +it te fact tat
tey do not ave diagnostic instruments available to tem tat matc teir instructional program.
AE
disaggregating assessment data by teacer to find areas +ere tere is a mar"ed difference
across classrooms so teacers can learn from peers +it te best outcomes in tat area.
Clearly, suc activity re9uires a ig degree of mutual trust, a caracteristic tat scool
reform researc as found to be present in urban scools tat ma"e significant
improvements 1Bry" and Scneider %&&%2. )e principals at t+o of te case study scools
noted tat tey needed to spend several years building up an atmospere of trust and a
Gblame7freeH culture before teir teacers could loo" at data togeter onestly.
)e 5E))S teacer survey data also underscore te importance of peer support in
creating a culture of data use. )eacers +o reported feeling +ell7supported by teir
colleagues in teir efforts to use data +ere more li"ely tan oter teacers to use data in +ays
related to instruction, for e3ample, to identify student s"ill gaps.
Data7informed decision ma"ing is not a simple intervention not only because it involves
so many aspects of education 1e.g., assessment, curriculum, accountability, information
tecnology2 but also because it re9uires fundamental improvements in te degree of mutual
trust and canges in te +ay teacer time is used. )e survey and case study data presented in
tis interim report suggest tat te movement to incorporate student data in local education
decision ma"ing is real 1many districts, scool leaders and teacers are ma"ing good7fait
efforts2, but tere is a significant distance to go before it becomes +ell e3ecuted in practice.
)e picture in %&&=>&? +as one tat so+ed various pieces of te comple3 re9uirements
getting put in place in different locations but no single scool or district demonstrating a
totally integrated, consistent and pervasive continuous7improvement process. Mutual trust
may prove to be te glue needed to old togeter te district and scool practices tat involve
using data to improve instruction and acievement.
A'
,eferences
!yres, P. 6. %&&8. Systematic matematical errors and cognitive load. ontemporary
,ducational -sychology %.1%2. %%?>AE.
Boudett, K. P., and L. Moody. 2005. Organizing for collaborative work. In Data wise, ed. K. P.
Boudett, E. A. City, and R. J. Murnane, 1128. Cambridge, Mass.: Harvard Education Press.
Brunner, C., C. @asca, *. <ein,e, M. <oney, D. 6igt, E. Mandinac, and D. -e3ler. %&&B.
6in"ing data and learning4)e (ro+ 5et+or" study. /ournal of ,ducation for Students
-laced at 0isk, 1&1C2. %A87%=?.
Bry", !. S., and B. Scneider. %&&%. Trust in schools: A core resource for impro#ement2 5e+
Kor". #ussell Sage @oundation.
Camburn, E., B. #o+an, and *. E. )aylor. %&&C. Distributed leadersip in scools. )e case of
elementary scools adopting compreensive scool reform models. ,ducational ,#aluation
and -olicy Analysis %'1A2. CA?>?C.
Coppin, *. %&&%. Data use in practice: ,3amples from the school le#el2 Paper presented at te
annual meeting of te !merican Educational #esearc !ssociation, 5e+ Orleans.
Confrey, *., and I. M. Ma"ar. %&&B. Criti9uing and improving te use of data from ig7sta"es
tests +it te aid of dynamic statistics soft+are. $n Scaling up success: !essons learned from
technology-*ased educational impro#ement, ed. C. Dede, *. P. <onan, and 6. C. Peters, 8'E>
%%=. San @rancisco, Calif.. *ossey7Bass.
Cromey, !. %&&&, 5ovember. Using student assessment data. -at can +e learn from scools;
Policy $ssues 5o. =. Oa" Broo", $ll.. 5ort Central #egional Educational 6aboratory.
Data Fuality Campaign. %&&?. 0esults of %&&( N,A sur#ey of state --1% data collection issues
related to longitudinal analysis. ttp.00+++.data9ualitycampaign.org0surveyJresults0 1clic"
on individual states to retrieve %&&? information: accessed *anuary %, %&&E2.
Datno+, !., L. Par", and P. -olstetter. %&&?. Achie#ing )ith data: 4o) high-performing
school systems use data to impro#e instruction for elementary students2 6os !ngeles, Calif..
University of Soutern California, Center on Educational (overnance.
@eldman, *., and #. )ung. %&&8. Using data based in9uiry and decision7ma"ing to improve
instruction. ,0S Spectrum 151C2. 8&>8'.
<alverson, #., *. (rigg, #. Pricett, and C. )omas. %&&B. The ne) instructional leadership:
reating data-dri#en instructional systems in schools2 Madison, -isc.. -isconsin Center for
Education #esearc, University of -isconsin.
<erman, *., and B. (ribbons. %&&8. !essons learned in using data to support school in6uiry and
continuous impro#ement: 7inal report to the Stuart 7oundation. CSE #eport BCB. 6os
!ngeles, Calif.. UC6! Center for te Study of Evaluation. !vailable at
ttp.00+++.cse.ucla.edu0products0#eports0#EBCB.pdf 1accessed December 8=, %&&E2.
B8
<offman, *. %&&?. Num*ers and types of pu*lic elementary and secondary education agencies
from the ommon ore of Data: School year %&&'8&. 9N,S %&&(-+'+:. U.S. Department
of Education. -asington, D.C.. 5ational Center for Education Statistics.
ttp.00nces.ed.gov0pubsearc0pubsinfo.asp;pubidM%&&?CBC 1accessed December 8=, %&&E2.
Mandinac, E. B., M. <oney, D. 6igt, *. <ein,e, and 6. #ivas, 6. %&&B. )ecnology7based tools
tat facilitate data7driven decision ma"ing. $n To)ard sustaina*le and scala*le educational
inno#ations informed *y the learning sciences, ed. C. I. 6ooi, D. *onassen, and M. $deda,
%=?>?A2. !msterdam. $OS Press.
Mars, *. !., *. @. Pane, and 6. S. <amilton. %&&=. Making sense of data-dri#en decision making
in education2 Santa Monica, Calif.. #!5D.
Palaic, #. M., D. (. (ood, and !. van der Ploeg. %&&A. State education data systems tat
increase learning and improve accountability. Policy $ssues 5o. 8=. 5aperville, $ll.. 6earning
Point !ssociates.
Scmo"er, M. *. 8''=. 0esults: The key to continuous school impro#ement2 !le3andria, La..
!ssociation for Supervision and Curriculum Development.
)orn, C. !. %&&%. Data use in the classroom: The challenges of implementing data-*ased
decision-making at the school le#el2 Paper presented at te annual meeting of te !merican
Educational #esearc !ssociation, 5e+ Orleans.
U.S. Department of Education, Office of Educational )ecnology. %&&A. To)ard a ne) golden
age in American education: 4o) the Internet, the la) and today$s students are
re#olutioni;ing e3pectations2 -asington, D.C.. !utor.
U.S. Department of Education, Office of Planning, Evaluation and Policy Development. %&&Ea.
National educational technology trends study: !ocal-le#el data summary. -asington, D.C..
!utor.
444. %&&Eb. Teachers$ use of student data systems to impro#e instruction %&&' to %&&(.
-asington, D.C.. !utor.
-ayman, *. C. %&&B. $nvolving teacers in data7driven decision ma"ing. Using computer data
systems to support teacer in9uiry and reflection. /ournal of ,ducation for Students -laced
At 0isk 1&1C2. %'B>C&E.
-ellman, B., and 6. 6ipton. %&&A. Data-dri#en dialogue: A facilitators guide to colla*orati#e
in6uiry. Serman, Conn.. Mira Lia 66C.
B%
Appendi A
Case Stud$ and Sur#e$ Data Sources
BC

Ehi!it A-'
Definitions of Terms
Term Definition
.ata systems >ectronic information systems to assist in the organization and
management of data+ 3hey consist of hard1are and soft1are that
pro'ide many different functions to users such as storing current and
historica data, rapidy organizing and anayzing data !e+g+, e6amining
reationships 1ithin data, specifying data subgroups#, and de'eoping
presentation formats or interfaces+
7nteroperabiity 3he abiity of different data systems or soft1are packages to
communicate 1ith one another+
Soft1are appications Soft1are appications consist of a 1ide range of speciaized products
to faciitate access to data, data anaysis and interpretation, and
presentations of data !e+g+, formatted reports, graphing functions#+
Schoos 7nteroperabiity
%rame1ork !S7%#
.ata e6change standards for N"12 education soft1are de'eoped to
promote data sharing bet1een different appications 1ithout further
soft1are inter'ention+ S7% 1as aunched in 1((- as a nonprofit
initiati'e of the soft1are and information technoogy industry to ensure
that N"12 instructiona and administrati'e soft1are appications 1ork
together more effecti'ey+
.ataAinformed decision
making
A process that integrates the anaysis of educationa data, typicay
stored in educationa data systems, to support decisions intended to
impro'e teaching and earning at the schoo and cassroom e'es+
3he practice entais reguar data coection and ongoing
impementation of impro'ements+
/ongitudina student
data
.ata on indi'idua students coected o'er time that ao1s users to
compie an academic history for each student+ 3his type of data
enabes more robust anayses of student performance to hep
differentiate instruction and impro'e student achie'ement+
Kuery too Soft1are that ao1s for customized and adAhoc data re?uests such
as Bdri do1nC capabiity to efficienty e6amine a subset of data at a
grade, cassroom or student e'e+
.ata ?uery A re?uest for specific records from a data system+
3ransaction capture <eaAtime accounting of daiy schoo functions such as attendance
and schoo unch counts+
>ectronic grade books 2nine toos to hep teachers manage cassroom acti'ities !e+g+,
generating seating charts, recording grades and test scores,
managing attendance, tracking skis and standards, panning essons,
and generating report cards#+
>ectronic portfoios 2nine storage of student 1ork sampes+
>ectronic
communication toos
3oos to faciitate communication of information such as use of eAmai,
de'eopment of 0eb sites, and eectronic discussion groups or
Bmessage boardsC 1here users from mutipe ocations can discuss
issues+
%ormati'e assessment Assessment conducted during instruction to pro'ide feedback that
can be used to adIust ongoing teaching and earning to impro'e
student outcomes+ %ormati'e assessment can be contrasted 1ith
summati'e assessment, 1hich takes pace after a period of
instruction, to Iudge ho1 much earning has occurred+
BB
Ehi!it A-+
Data A#aila!le %rom the State Data S$stem for Each Case Stud$ District in +//2
District ' + 7 ; < 0 2 > ?
Data Elements in State
Information S$stem
CA @A MD TB ,I MA =C C9 M9
Lni?ue state1ide student identifier
that connects student data across key
databases across yearsO
Pes Pes ;o Pes Pes Pes ;o Pes ;o
Abiity to match indi'idua students*
test records from year to year to
measure academic gro1thO
Pes Pes ;o Pes Pes Pes Pes Pes Pes
State can measure yearAtoAyear
gro1th for indi'idua students in any
subIect !state tests in the same
subIect in consecuti'e years and is
abe to connect data across years#
;o Pes ;o Pes Pes Pes Pes Pes Pes
StudentAe'e enroment,
demographic, and program
participation informationO
Pes Pes Pes Pes Pes Pes Pes Pes ;o
StudentAe'e transcript information,
incuding information on courses
competed and grades earnedO
;o ;o ;o Pes ;o ;o Pes ;o ;o
A teacher identifier system 1ith the
abiity to match teachers to studentsO
;o ;o ;o ;o Pes ;o ;o ;o ;o
7dentifies 1hich schoos produce the
strongest academic gro1th for their
students
Pes Pes ;o Pes Pes Pes ;o Pes ;o
Nno1s 1hat achie'ement e'es in
midde schoo indicate that a student is
on track to succeed in rigorous
courses in high schoo
;o ;o ;o Pes ;o ;o ;o ;o ;o
A state data audit system assessing
data ?uaity, 'aidity and reiabiityO
Pes Pes Pes Pes ;o Pes Pes Pes Pes
;umber of the 10 .K) essentia
eements presentO
7 7 $ ( 6 - 7 6 4
Exhibit reads: In District 1, in 2007, the state data system has a unique statewide student identifier that connects student data across key databases across years.
Note: An asterisk identifies six of the 10 essential elements most relevant to the data issues discussed in the case studies. There are four additional elements not
listed.
Source: Data Quality Campaign (2007).
B=
A
-
3
Ehi!it A-7
Case Stud$ District Data S$stems
3urn/eaf Achie'ement &anagement System ser'es as the data
1arehouse and assessment systemH eS7S is the student information
systemH ;0>A onine database for anaysis and reporting toos and
instructiona materias+
eSchoar is the data 1arehouseH S7&S is student information systemH
StarQDase for student recordsH 3est0iz managed by data&etrics
Soft1areH 3P. eectronic registrarH P7&S performance impro'ement
mapping system from the S>AH >dine for reporting teacher data to
students and parents+
Schoo;et Account !report generation#H Assess !assessment bank#
and Aign !data 1arehouse# moduesH Pentamation is the student
information systemH Specia >ducation &anagerH discipine and
attendance systems+
>ducation .ecision Support /ibrary is the data 1arehouse !>.S/ is
/>A de'eoped#H Denchmark Assessment and <eporting 3oo !DA<3
1as Iointy de'eoped by />A and Princeton <e'ie1#H Schoos
Administrati'e Student 7nformation !SAS7# from Pearson, >ducationa
)urricuum Assessment <esource 3oo !>)A<3 is being de'eoped by
/>A to repace DA<3#+
S7&S is the state data system and ser'es as the />A data 1arehouse
!1i be repaced by ;) 07S>#H )A)ASS for specia education dataH
.7D>/S onine database for anaysis and reporting toos !data
upoaded from handheds#+
.ata 0arehouse 1as de'eoped by the />AH 7nstructiona
&anagement System !7&S designed by A.&7;+)2&#H Professiona
.e'eopment 2nine !P.2 1as buit by />A and a pri'ate company#+
<egistration !Reg 1as de'eoped by the />A# is a student information
system and data 1arehouseH Socrates )<& !the ibrary data system#H
Phonoogica A1areness /iteracy Screening !PA/S# 0eb site for
anaysis and reporting toos+
7nfinite )ampus houses student data and has a suite of reporting toos
and parent portaH >d5ate assessment system is being pioted+
)ognos !ac?uired by 7&D# is the />A*s data 1arehouseH Range is a
0ebAbased transactiona system to enter data !attendance, unch,
district test scores#H Student Assessment .ata System !SA.S#
pro'ides historica test data and standard reportsH >ncore is the
specia education system !/>AAde'eoped systems#+
B?
=ETTS Sur#e$ Data
)e primary survey data included in tis report come from te U.S. Department of
Education/s 5ational Educational )ecnology )rends Study 15E))S2. 8,&C' district
tecnology directors surveyed during spring %&&? and %,B&' teacers surveyed in spring %&&?.
)e teacers +ere clustered in scools sampled from districts participating in te 5E))S
district survey.
C%
)e 5E))S district sample of 8,&C' districts +as nationally representative
+it respect to poverty status, student enrollments, and location 1urban or rural status2. )e =&
largest urban scool districts across te country +ere selected +it certainty 1i.e., included in
te sample from te outset2. Districts composed entirely of special education scools and
vocational7tecnical scools, as +ell as independent carter scools tat are teir o+n
districts, +ere e3cluded from te district sampling frame because of teir dissimilarity to
GtypicalH districts. )e teacer sample +as created by dra+ing a probability sample of '?B
scools from respondents to te district survey, stratified by scool type 1elementary or
secondary2, and poverty level 1ig or lo+2. Scools +ere randomly sampled in proportion to
te number of teacers and in inverse proportion to district si,e to produce a sample of
scools +ose selection probabilities +ere rougly independent of te si,e of teir district/s
enrollment.
<iger7poverty scools +ere oversampled 1%CC of te '?B scools2 to obtain more
precise data about teir tecnology use. @or scools, Giger povertyH +as defined as above a
specified cutoff in terms of te percentage of students +o +ere eligible for te free or
reduced7price lunc program. )e dividing line bet+een iger7poverty and lo+er7poverty
scools +as selected to ensure tat for eac scool type 1elementary, middle or ig scool2,
tere +ould be te same number of teacers in te iger7poverty and te lo+er7poverty
groups, as reported in te 5ational Center for Education Statistics Common Core of Data
1CCD2. Elementary scools +it %'.? percent of teir students eligible for free or reduced7
price lunces +ere classified as iger poverty. @or middle and ig scools, te poverty
tresolds +ere %A.C percent and 8B.' percent, respectively.
5E))S researcers obtained teacer rosters for te '?B scools +itin te districts
selected for te 5E))S district survey. )o be eligible for te teacer sample, a teacer ad to
be teacing at te same scool in te scool year before survey administration 1i.e., teacers
ne+ to te scool +ere e3cluded2. )eacers +o did not teac core academic subDects also
+ere omitted from te sample. )argets of four teacers from eac of te scools +ere
randomly selected for te teacer sample. )e final teacer sample in %&&? consisted of 8,??'
teacers from E=B scools.
Response rates were 94 percent for the district survey in 2007 and 85 percent for the
teacher survey. Sampling weights were applied to the teacher data to obtain nationally
representative estimates. The district and teacher surveys were initially administered during
the 200405 school year to the same sample of districts and schools that provide the basis of
comparison between 2007 and 2005 for the teacher survey data. In fall 2005, 6,017 teachers
were surveyed with a response rate of 82 percent; the larger sample was designed to provide
C%
Districts +ere sampled from among te 8%,AEC districts tat received federal Enancing Education )roug
)ecnology 1EE))2 funds in %&&C and from an additional %,%C' districts tat ad not received EE)) funds.
BE
robust, school-level estimates of technology use. For additional information on the
comparisons between 2007 and 2005 teacher survey responses, see Teachers Use of Student
Data Systems to Improve Instruction: 2005 to 2007 (U.S. Department of Education 2008b).
B'
Appendi 6
Scenario Ehi!its, Items, ,u!rics, Item Score Distri!ution and Mean Scores
=8
Scenario A
;o1 7*m going to describe a hypothetica situation and a computerAbased student data system to you and
7*d ike to see 1hat kind of information you*d ike to get from the system+
Suppose it*s Manuary 2007 and you*re one of the fourthAgrade teachers in a schoo that 1as surprised by
fourth graders* reati'ey o1 performance on the state reading test ast year !spring 2006#+ Pour principa
has encouraged you to use student data to gain insights into ho1 you can get higher 5rade 4
achie'ement this year+
3he data system a'aiabe to you Sshow Figure A screen mockupT contains data on both current !2006"
07# fourth graders and ast year*s !2005"06# fourth graders !show Student Groups table, pointing to
student groups as you name them# as 1e as other student groups+ %or each student, the data system
has scores on ast spring*s state reading test and scores on a district test gi'en in the fa !point to
relevant cells in Student Variables table#+ 7t aso has other information about students that can be used to
create subgroups 1ithin a grade if you 1ant to see ho1 different subgroups compare+ %or e6ampe,
ethnicity, gender, and 1hether the student is eigibe for free or reduced price unch !%<P/#+
=C
%igure A8
Student 5roups
200506 Third graders 200506 Fourth graders 200506 Fifth graders
200607 Third graders 200607 Fourth graders 200607 Fifth graders
Student Uariabes
Ethnicity 200506 Grade 3 state spring reading achievement score
Gender 200506 Grade 4 state spring reading achievement score
Free or reduced-price lunch applicant 200506 Grade 5 state spring reading achievement score
Year entered district 200607 Grade 3 district fall reading test
Grade 3 teacher 200607 Grade 4 district fall reading test
Grade 4 teacher 200607 Grade 5 district fall reading test
Assessment System 2utput
=A
A-'- So no1 in Manuary 2007, 1hat specific data 1oud you 1ant to get from this system to hep you
decide ho1 to impro'e your fourth graders* performance4 !Follow-up probes+# 0hat data 1oud
you 1ant to see first4 3e me 1hat group of students and 1hat test score or student
characteristic you 1oud ike to see for those students+
,u!ricC
'C 3eacher picks a ogica group !either 2005"06 fourth graders or 2006"07 fourth graders# A=D
seects a ogica measure for that group !either Spring 2006 5rade 4 state test scores or %a
2006 5rade 5 district fa test scores for 2005"06 fourth graders 2< either Spring 2006 5rade $
state scores or %a 2006 5rade 4 district scores for 2006"07 fourth graders+
/-<C 3eacher identifies group and scorer can infer measure based on teacher response+H 9,
3eacher picks a ogica group !either 2005"06 fourth graders or 2006"07 fourth graders#H 9,
3eacher seects a ogica measure for that group !either Spring 2006 5rade 4 state test scores or
%a 2006 5rade 5 district fa test scores for 2005"06 fourth graders 2< either Spring 2006
5rade $ state scores or %a 2006 5rade 4 district scores for 2006"07 fourth graders+
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers
Score %re.uenc$ 3(roportion4
M / /-< '
-5 26 !+$1# 2$ !+27# $6 !+42# +56
A-+- 0oud you ike to make any other ?ueries of the data system4 Are there other kinds of data you
1oud ike to ha'e to hep impro'e this year*s fourth graders achie'ement that you don*t see
represented in this system4
=o ,u!ricC Coded onl$, not scored
=B
Scenario 6
3his !show Figure B# is the kind of data tabe that some student data systems produce+ 7*m going to ask
you to find some information from the tabe and then 7*d ike for you to te me on a scae of 1 to 10, 1ith
10 being e6tremey difficut, ho1 hard or easy it 1as to find the information in the tabe+ <eady4
%igure D
2005"06 Score /e'esE>ngish /anguage Arts !>/A#
English Language Arts
Hamilton Elementary
Grade Gender Ethnicity
Number of
Students
Tested
Percent
of Tested
Students
Mean
Scale
Score
Number Students at
Each Proficiency Level
Belo
Basic Basic Proficient Advanced
!
Female
African American 18 26% 439 5 7 5 1
Asian/Pac Islander 1 1% 610 0 0 0 1
Latino 17 24% 428 5 6 5 1
!ite 34 49% 449 4 13 11 6
"otal Female 70 100% 444 14 26 21 9
#ale
African American 18 23% 436 6 6 5 1
Asian/Pac Islander 2 3% 452 0 1 0 1
Latino 31 40% 430 8 7 14 2
!ite 27 35% 448 6 11 7 3
"otal #ale 78 100% 438 20 25 26 7
"
Female
African American 18 24% 441 3 8 5 2
Asian/Pac Islander 2 3% 462 1 0 1 0
Latino 36 47% 436 8 12 12 4
!ite 20 26% 472 2 7 8 3
"otal Female 76 100% 447 14 27 26 9
#ale
African American 16 23% 442 2 8 5 1
Asian/Pac Islander 0 0% $A 0 0 0 0
Latino 29 42% 438 5 12 10 2
!ite 24 35% 456 3 13 5 3
"otal #ale 69 100% 445 10 33 20 6
#
Female
African American 19 26% 463 2 6 8 3
Asian/Pac Islander 1 1% 317 1 0 0 0
Latino 34 47% 452 4 14 10 6
!ite 19 26% 507 1 6 7 5
"otal Female
73 100% 467 8 26 25 14
#ale
African American 17 23% 449 2 6 6 3
Asian/Pac Islander 3 4% 560 0 0 1 2
Latino 34 46% 448 7 13 11 3
!ite 20 27% 468 3 6 8 3
"otal #ale 74 100% 467 12 25 26 11
==
6-'- 0hat 1as the mean !or a'erage# scae score for the /atino fifthAgrade girs 1ho took the test4
Go1 easy 1as it to find this information in the tabe4
,u!ricC
'C 452
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
4( 5 !+10# ;@A 44 !+(0# +(0
6-+- 0hich student group had the highest a'erage or mean >/A scae score in 5rade 44 Go1 easy
1as it to find this information in the tabe4
,u!ricC
'C B0hite femaesC or B1hite girsC
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
4( 6 !+12# ;@A 4$ !+--# +--
6-7- Go1 many African American thirdAgrade boys took the test4 Go1 easy 1as it to find this
information in the tabe4
,u!ricC
'C 1-
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
4- 0 ;@A 4- !1+00# 1+00
6-;- 2'era, based on the 5rade $ data in this tabe, 1oud you say that there 1as a difference
bet1een boys and girs in >/A test performance4
=o ,u!ricC Coded onl$, not scored
=?
Scenario C
3his !show Figure C# is the kind of data chart that some student data systems produce+ 7*m going to ask you to find some
information from the graph and then 7*d ike for you to te me on a scae of 1 to 10, 1ith 10 being e6tremey difficut, ho1
hard or easy it 1as to find the information in the chart+ <eady4
%igure )
>/A Dar 5raph
8,ADE 7 8,ADE ; 8,ADE <
n 9 ;umber of test takers
6
8
S
r
i

C-' 3II4- /ooking at this graph, can you find the a'erage !or mean# >ngish /anguage Arts scae score for
the /atino fifthAgrade girs 1ho took the test4 Go1 easy 1as it to get this information from the
chart4
,u!ricC
'C Any ans1er from 44( to 455
/C 7ncorrectH 3eacher does not kno1H Scorer cannot concude 1hether ans1er is correct or
incorrect
n Teachers Score %re.uenc$ 3(roportion4
/ /-< '
(7 25 !+26# ;@A 72 !+74#
C-+- )an you find the mean score for Asian@Pacific 7sander fourthAgrade boys 1ho took the test4
Go1 easy 1as it to get this information from the chart4
,u!ricC
'C B;oC or Bnone took the testCH B3here*s no data+C
/C 7ncorrect or B3here*s no information+CH 3eacher does not kno1+H Scorer cannot concude
1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(7 $0 !+$1# ;@A 67 !+6(# +6(
C-7- 0hich student group had the highest a'erage >/A score in 5rade 44 Kuestion8 Go1 easy 1as
it to get this information from the chart4
,u!ricC
'C B0hite femaesC or B1hite girsC
/-<C B0hiteCH Borange bar for 1hiteC

/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(5 4 !+04# 10 !+11# -1 !+-5# +(1
C-;- Go1 many African American thirdAgrade boys took the test4 Go1 easy 1as it to find this
information in the tabe4
,u!ricC
'C 1-
/C 7ncorrectH 3eacher does not kno1H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(5 10 !+11# 0 -5 !+-(# +(0
='
Scenario D
3his !show Figure # is the kind of data tabe that some student data systems produce+ 7*m going to ask
some ?uestions about ho1 you 1oud interpret the data in this tabe+
%igure .
2005"06 Score /e'esE>ngish /anguage Arts !>/A#
English Language Arts
Hamilton Elementary
Grade Gender Ethnicity
Number
of
Students
Tested
Percent
of
Tested
Students
Mean
Scale
Score
Number Students at
Each Proficiency Level
Belo
Basic Basic Proficient Advanced
!
Female
African American 18 26% 439 5 7 5 1
Asian/Pac Islander 1 1% 610 0 0 0 1
Latino 17 24% 428 5 6 5 1
!ite 34 49% 449 4 13 11 6
"otal Female 70 100% 444 14 26 21 9
#ale
African American 18 23% 436 6 6 5 1
Asian/Pac Islander 2 3% 452 0 1 0 1
Latino 31 40% 430 8 7 14 2
!ite 27 35% 448 6 11 7 3
"otal #ale 78 100% 438 20 25 26 7
"
Female
African American 18 24% 441 3 8 5 2
Asian/Pac Islander 2 3% 462 1 0 1 0
Latino 36 47% 436 8 12 12 4
!ite 20 26% 472 2 7 8 3
"otal Female 76 100% 447 14 27 26 9
#ale
African American 16 23% 442 2 8 5 1
Asian/Pac Islander 0 0% $A 0 0 0 0
Latino 29 42% 438 5 12 10 2
!ite 24 35% 456 3 13 5 3
"otal #ale 69 100% 445 10 33 20 6
?&
D-'- Suppose you*re 1orking 1ith thirdAgrade teachers at this schoo and they*re interested in
e6amining ho1 their students performed in terms of the anguage skis measured on this test+
0hat does the data in this tabe indicate about 1hether boys and girs performed differenty in the
third grade4 SGet open responseT
=o ,u!ricC Coded onl$, not scored
D-+- 2N+ ;o1 7*m going to read a series of statements that peope might make about different aspects
of the 5rade $ data in this tabe+ 7*d ike you to te me for each statement 1hether you agree or
disagree and the reasons 1hy+ <emember to think aoud as you*re deciding on your ans1er+

a+ A maIority of third graders at this schoo ha'e not achie'ed proficiency in >/A as measured by this test+
,u!ricC
'C Agree S57: are ess than proficientH -5@14- 9 !14J26J20J25#@!70J7-#T
/-<C Agree !no e6panation#
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(7 16 !+16# 2 !+02# 7( !+-1# +-2
b+ 7n 5rade $, boys 1ere more ikey than girs to score Deo1 Dasic on this assessment+
,u!ricC
'C Agree 25+6: boys V 20: girs !20@7- V 14@70#
/-<C Agree !20 boys V 14 girs#H Agree !no e6panation#
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(6 1- !+1(# 47 !+4(# $1 !+$2# +5-
c+ 2f those students 1ho scored Deo1 Dasic in 5rade $, most 1ere /atino+

,u!ricC
'C .isagree !!ess than hal" of a of the students 1ho scored Deo1 Dasic in 5rade $ are /atino+#
/-<C .isagree 1ith no e6panation
/C AgreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(6 71 !+74# 1 !+01# 24 !+25# +26
?8
D-7- ;o1 et*s assume that you*re a fourthAgrade teacher and these 5rade 4 data are for midAyear performance on
a benchmark test+ 7f there is a particuar group of fourth graders you think 1i be most ikey to ha'e troube
scoring basic or abo'e on the state test at the end of the year, coud you point out their data in this tabe4 Sif
appropriate probe 1ith one of the foo1ingT 0hich group 1oud you be concerned about and 1hat data
trigger that concern4 2< 0hy don*t you think the subgroup data in the tabe point to a priority need for any
particuar subgroup4
,u!ricC
'C 3hose scoring Deo1 Dasic
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(6 -6 !+(0# 5 !+05# 5 !+05# +0-
=oteC
7tem responses did not correate 1ith those to other .ata 7nterpretation items+ Scores not incuded in
anayses+
D-;- ;o1 et*s go back to the 5rade $ data+ <emember that these are for ast year*s third graders+ 7f there ha'e
been no maIor changes in the schoo*s student body, teachers, or curricuum 1oud you e6pect that8
a+ 2n the basis of ast year*s test scores, girs can be e6pected to score higher than boys 1hen this test is
gi'en to this year*s third graders+
,u!ricC
'C .isagree S5rade $ girs and boys scored about the same, e+g+, !21J(#@70 9 !26J7#@7-T
/-<C .isagree 1ith no e6panation
/C AgreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
($ 44 !+47# 0 4( !+5$# +5$
b+ 3hese data suggest that ne6t year*s thirdAgrade Asian@P7 girs 1i score better than other third graders on
this test+
,u!ricC
'C .isagree !)annot predict ne6t year*s scores based on the one 5rade $ Asian@P7 student
represented in the tabe#
/-<C .isagree 1ith no e6panation
/C AgreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
($ $( !+42# 2 !+02# 52 !+56# +57
?%
Scenario E
3his !show Figure ## is a different kind of data dispayEa bar graph of 5rade 4 mathematics achie'ement separated into t1o components
!computation and probem so'ing# as 1e as their tota, for a schoo and its district for each of three years+ Again, 7*m going to ask you to find
some information on the dispay and then te me ho1 easy or hard that 1as to do on a scae from 1 to 10 1ith 10 being Be6tremey difficut+C
%igure >+ 3rend .ata Dar 5raph
7
3
E-'- 0hat 1as 2ak Schoo*s a'erage 3ota &ath Score in 200$"044
,u!ricC
'C Any specific ans1er from $$$ to $$-
/-<C Uague, but correct ans1er range pro'ided !e+g+, Bbet1een $$0 and $40C#
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(1 4 !+04# $ !+0$# -4 !+(2# +(4
E-+- 0hat 1as the difference in the district*s tota math score in 2005"06 compared 1ith 200$"044
,u!ricC
'C Any ans1er from 1 to $ pointsH Ba fe1 pointsC or Bup a itte bitC or Babout the sameC or BsimiarC
/-<C Be6acty the sameCH Bthe sameC
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(1 4 !+04# 10 !+11# 77 !+-5# +(0
;o1 7*m going to ask you some ?uestions about ho1 you 1oud interpret the data in this chart+
E-7- /ooking at the chart as a 1hoe, 1hat 1oud this data te you about fourth graders* mathematics
achie'ement at this schoo4 SGet open response$=o ,u!ricC Coded onl$, not scoredT 2N+ ;o1
7*m going to read a series of statements that peope might make about the data in this graph and
7*d ike you to te me for each one 1hether you agree or disagree and the reasons 1hy+ <emember
to think aoud as you*re deciding on your ans1er+
a+ 2ak Schoo does better than the district as a 1hoe in 5rade 4 mathematics+
,u!ricC
'C .isagree !2ak Schoo tota math scores are o1er than the district*s math scores+#
/-<C .isagree 1ith no e6panation
/C AgreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+

n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(1 2 !+02# - !+0(# -1 !+-(# +($
?A
E-7 3continued4
b+ 2ak Schoo has made some impro'ement in 5rade 4 mathematics o'er this time period+
,u!ricC
'C Agree !2ak Schoo tota math scores are sighty higher in 2005"06 than in 200$"04+#
/-<C Agree 1ith no e6panation
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(1 $1 !+$4# $ !+0$# 57 !+6$# +64
c+ <eati'e to the district as a 1hoe, 2ak Schoo fourth graders ha'e been getting better in their
probem so'ing skis+
,u!ricC
'C Agree !based on clearly identi"iable comparison of probemAso'ing scores for 2ak Schoo and
the district#
/-<C Agree !Dased ony on 2ak Schoo dataH no cear, identifiabe comparison 1ith district#H Agree
1ith no e6panation
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(2 $- !+41# 10 !+11# 44 !+4-# +5$
d+ <eati'e to the district as a 1hoe, 2ak Schoo fourth graders ha'e been getting better in their
computation skis+
,u!ricC
'C .isagree !Dased on clearly identi"iable comparison of computation scores for 2ak Schoo and
the district#
/-<C .isagree !Dased ony on 2ak Schoo dataH no cear, identifiabe comparison 1ith district#H
.isagree 1ith no e6panation
/C AgreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-- $2 !+$6# - !+0(# 4- !+55# +5(
?B
E-7 3continued4
e+ 2ak Schoo*s progress in narro1ing the 5rade 4 math achie'ement gap 1ith the rest of the district
has been in probem so'ing rather than computation+
,u!ricC
'C Agree !5ap bet1een 2ak Schoo and district scores decreases more for probemAso'ing than
for computation+#
/-<C Agree 1ith no e6panation
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-( 26 !+2(# 6 !+07# 57 !+64# +67
E-;- Suppose 2ak Schoo had started using a ne1 mathematics program at the beginning of the 2004A05
schoo year 1hie the rest of the district continued 1ith the od program+ /ooking at these data, 1hat
are your thoughts about the ne1 curricuum4 SGet open response$=o ,u!ricC Coded onl$, not
scoredT 0hich of these statements 1oud you agree 1ith and 1hy4
a+ 3he math program appeared to impro'e achie'ement the first year it 1as used but the benefit
disappeared the ne6t year+
,u!ricC
'C .isagree !2ak Schoo is sti performing better in 2005"06 than it 1as in 200$"04+#D Agree 1ith
reser'ations !3eacher takes issue 1ith 1ord BdisappearC+ %or e6ampe, B7 agree, it did impro'e
achie'ement the first year and scores are o1er in the second year, but the benefit did not totay
disappear+C#
/-<C .isagree 1ith no e6panation
/C AgreeH 3eacher does not kno1+ Scorer cannot concude 1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-- 5$ !+60# 1 !+01# $4 !+$(# +$(
?=
E-; 3continued4
b+ 3he ne1 math program appeared to hep students 1ith their computation skis+
,u!ricC
'C .isagree !Dased on comparison of computation scores for 2ak Schoo and the districtH 5ap
stays reati'ey constant each year+#
/-<C .isagree !ook at 2ak Schoo ony#H .isagree 1ith no e6panationH Agree 1ith reser'ations
!i+e+, Bgap is pretty consistent, but the program heped a itte bitC#
/C Agree and focus ony on 2 years of data !0$"04 and 04"05#H Agree and say that B05"06 is
better than 0$"04C but do not say Bsighty betterCH 2ther agreeH 3eacher does not kno1+H Scorer
cannot concude 1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(0 6( !+77# 15 !+17# 6 !+06# +15
c+ Pou can*t be sure 1hether the program is ha'ing an effect because there may be differences
bet1een the different casses of fourth graders+
,u!ricC
'C Agree !e+g+, ackno1edges that there are differences in the popuations of students each year#
/-<C Agree 1ith no e6panation
/C .isagree !e+g+, Program 1i ha'e an effect regardess of the student popuation in'o'ed+# H
3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(0 22 !+24# 20 !+22# 4- !+5$# +64
=oteC
7tem responses did not correate 1ith those to other .ata 7nterpretation items+ Scores not incuded in
anayses+
??
E-; 3continued4
d+ Scores mo'e around from year to year, but the ne1 math program appears promising and shoud
be monitored for more years+
,u!ricC
'C Agree !e+g+, some indication that program benefits studentsH need more than 2 years of data to
determine if a program is 1orking#
/-<C Agree 1ith no e6panation
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(1 10 !+11# 15 !+16# 66 !+7$# +-0
E-<+ Are there other issues or possibe e6panations that shoud be taken into account as 1e4
=o ,u!ricC Coded onl$, not scored
?E
SCE=A,I9 %
Suppose you*re in a meeting to discuss 2005"06 reading data from the state assessment for your
schoo*s third grade and they hand out this data dispay+ !Show Figure F%#
%igure %
Gistogram
Grade !
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
%elo& %asic %asic Proficient Ad'anced
Le'el
P
e
r
c
e
n
t
a
(
e
?'
%-'#'- Dased on this chart, 1hat percentage of the schoo*s third graders ha'e achie'ed proficiency4
,u!ricC
'C Any ans1er from 44 to 46
/-<C >'idence that correct math is being used, but no fina ans1er !e+g+, B7 need to add $-: and
6:+C#
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
47 2- !+60# 1 !+02# 1- !+$-# +$(
%-'#+- Dased on this chart, 1hat percentage of the schoo*s third graders 1ere ess than Proficient in
reading4
,u!ricC
'C Any ans1er from 64 to 66
/-<C >'idence that correct math is being used, but no fina ans1er !e+g+, B7 need to add 2$: and
42:+C#
/C 7ncorrectH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
4( 12 !+24# 6 !+12# $1 !+6$# +6(
%-+- 2ne of your coeagues, after ooking at these data says, B3here*s something 1rong 1ith this chart+C
0oud you agree4 0hy or 1hy not4
,u!ricC
'C 7f agrees A;. says that Bnumbers don*t add to 100C or Bnumbers add to more than 100C
/C 7ncorrect !)an*t find anything 1rongH 7dentifies something ese as B1rongC#H 3eacher does not
kno1+H Scorer cannot concude 1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
(6 42 !+44# ;@A 54 !+56# +56
E&
Scenario 8
Suppose you*re teaching in a district that re?uires students to attain eighthAgrade proficiency in
mathematics in order to enro in Agebra 7 in high schoo+ /ooking at students* performance the
preceding year, you found the resuts in this tabe !show Figure G# for the /atino and African
American students 1ho make up your schoo*s entire student body+
W A score of 65: on the district math test is considered proficient, and
W A schoo is considered Bo1 performingC if ess than 50: of students in any student
subgroup reach proficiency+
%igure 5
2005"06 Achie'ement in 5rade - &athematics
5roup ;umber of Students &ean &ath Score Percentage
Proficient
;umber Proficient
/atino 22- 67+5 61 1$(
African American $1 66 4- 15

E8
8-'o- 0hat do these data te you about ho1 1e students are doing at your schoo4
,u!ricC
'C Any correct obser'ation or inference from the tabe, incuding BAbout 60: of /atino students
are proficient+C BA smaer proportion of African American than of /atino students are proficient+C
B3he a'erage score in the schoo is Iust a itte abo'e the proficiency criterion+C
/C 1"2 correct genera statements, but ceary is not reading the tabe correcty !e+g+, B/atinos are
doing better than African Americans, but both groups are not proficient+C#H 2ther 7ncorrect !.o not
reference anything in the tabe, 'ery 'ague#H 3eacher does not kno1+H Scorer cannot concude
1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-2 24 !+2(# ;@A 5- !+71# +71
8-'- ;o1 7*m going to read you some statements again and 7*d ike you to te me 1hether you agree or
disagree 1ith each and 1hy+
a+ 3he maIority of our eighth graders are proficient in eighthAgrade math+
,u!ricC
'C Agree S5(: 9 154@25( 9 !1$(J15#@!22-J$1#T
/-<C Agree 1ith no e6panationH Agree 1ith incorrect reasoningH .isagree 1ith different definition
of maIority !e+g+, says 5(: is not a maIorityH may state correct numbers in'o'ed in cacuation,
but sti disagree#
/C 2ther .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-5 16 !+1(# 41 !+4-# 2- !+$$# +57
b+ 2ur schoo is not getting enough African American students to proficiency, but /atino students are
meeting the re?uired performance standard+
,u!ricC
'C Agree !61: for /atinos is greater than the 50: re?uirementH 4-: for African Americans is ess
than the 50: re?uirement#
/-<C Agree 1ith no e6panation
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-6 $0 !+$5# - !+0(# 4- !+56# +60
E%
8-' 3continued4
c+ 2ur schoo is cassified as o1 performing based on these mathematics scores+
,u!ricC
'C Agree !African American subgroup is ess than 50: proficient+#
/-<C Agree 1ith no e6panation
/C .isagree !e+g+, B5(: of a students are proficient+C#H Agree based on 65: criteria !e+g+, BDoth
groups are beo1 65: proficient#H 2ther agree 1ith incorrect reasoningH 3eacher does not kno1+H
Scorer cannot concude 1hether ans1er is correct or incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
-6 25 !+2(# 7 !+0-# 54 !+6$# +67
8-+- 0hat actions shoud your schoo consider to a'oid being abeed Bo1 performingC in the coming
year4 Sa"ter open responseT 0hich of these statements do you agree 1ith based on these data4
!>6pain your ans1er for each+# SFor each statement, give teacher an inde& card with the
statement on it to keep handy as he'she looks at the data%T
a+ 3his year a eighth graders shoud get more intensi'e instruction in mathematics+
b+ 3his year a African American students shoud get more intensi'e instruction in mathematics+
,u!ric
'C 5i'es same ans1er for 5+2a and 5+2b 1ith appropriate e6panation
/-<C Same ans1er for 5+2a and 5+2b, but no e6panation for one of the itemsH 5i'es the same
ans1er for 5+2a A;. 5+2b 1ith no e6panation for either item
/C 5i'es the same ans1er for a and b 1ith an incorrect rationae !B7 disagree because the tabe
doesn*t sho1 a students in the schoo+CH B7 agree because a of the students are not up to the
proficient e'e+C#H 5i'es different ans1ers for 5+2a and 5+2bH 3eacher does not kno1+H Scorer
cannot concude 1hether ans1er is correct or incorrect+

EC
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
7( 25 !+$2# 20 !+25# $4 !+4$# +56
8-+ 3continued4
c+ 3eachers shoud obtain a detaied breakdo1n of ast year*s test resuts by item or content
standard+
,u!ric
'C Agree !1ith reasonabe e6panation about why ha'ing detai is important#
/-<C Agree 1ith no e6panation
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
77 0 20 !+26# 57 !+74# +-7
d+ 2ur eighthAgrade mathematics program isn*t necessariy Bbroken,C there*s a good chance that 50:
or more of African American students 1i meet the proficiency re?uirement this year+
,u!ricC
'C Agree !e+g+, ackno1edges that sampe size is sma and 1oud be possibe for 1"2 students to
do better thus bumping the percent proficient o'er 50:H states that 4-: is cose to 50: and
1oud be possibe to get to 50: this year#
/-<C Agree 1ith no e6panation !no detai, missing e6panation regarding why African Americans
coud meet 50: e'e#
/C .isagreeH 3eacher does not kno1+H Scorer cannot concude 1hether ans1er is correct or
incorrect+
n Teachers Score %re.uenc$ 3(roportion4 M
/ /-< '
7( 27 !+$4# 10 !+1$# 42 !+5$# +5(
EA
Scenario *
;o1 7*m going to ask you to think about some data from a hypothetica cassroom+
Suppose you*re teaching mathematics and at the end of a unit on measurement, you ga'e a 100Apoint
test on measurement concepts and skis and your students obtained the scores sho1n in this cass ist+
SShow Figure (+T Pou kno1 that students from this schoo ha'e had troube 1ith measurement items on
the state test in pre'ious years, and you*re 1ondering 1hether you need to do more teaching in this area
or can mo'e on to the ne6t topic+ Pou takes these scores into the teachers* ounge and ask coeagues to
take a ook+ 0hen they ask about the test you e6pain that you designed it so that if a student gets a 70:
or better on it, you are reay ?uite confident that he or she understands the concepts+ 0hen a student*s
score is o1er than that, you fee there is something they sti don*t understand+
2ne of your coeagues pus out his cacuator and sho1s that the mean for these scores is 70+5+ B3he
mean score is greater than 70+ Pou*'e done your Iob+ &o'e onX 3here*s ots more math to co'er+C
%igure G
)assroom .ata Set
Student
Number Score
1 95
2 65
3 85
4 55
5 90
6 70
7 100
8 80
9 65
10 80
11 65
12 90
13 65
14 55
15 45
16 15
17 70
18 65
19 75
20 85
21 65
EB
*-'- .o you agree 1ith this coeague4 0hy or 1hy not4
=o ,u!ricC Coded onl$, not scored
*-+- 0hat do you think 1oud happen if you ga'e the same cass of students another test on
measurement the ne6t day4
=o ,u!ricC Coded onl$, not scored
*-7- 7f you 1ere the teacher, 1hat 1oud you do4
=o ,u!ricC Coded onl$, not scored
E=
Appendi C
Scenario De#elopment and ,elia!ilit$ Information
E?
Data Scenario De#elopment
Because of the projects interest in getting at multiple, distinct facets of teachers
use of data (question posing, data location, data comprehension, data interpretation and
data use), an essentially criterion-referenced approach was taken to the development of
the data scenario assessment items. The data scenarios were developed for this study with
the advice and review of experts in data-informed decision making and assessment. The
experts judged that the items represented the intended constructs. In addition, the
scenarios were pilot-tested to make sure that they were comprehensible and tapped the
intended skills and concepts. The mean scores for the individual items are displayed in
Exhibit C-1.
E'
Ehi!it C-'
%re.uenc$ Distri!utions, Mean Scores and ,elia!ilities for Scored Items
Item ID
n
3Teachers4
Score %re.uenc$
M
3SE4
Alpha
/ /-< '
%orm
'
%orm
+ %orm 7
Kuestion Posing !1 item#
A+1 -5 26 2$ $6
+56
!+046#
+46( ;@A +5-(
.ata /ocation !( items#
%+1'1 47 2- 1 1-
+$(
!+071#
+520 ;@A ;@A
%+1'2 4( 12 6 $1
+6(
!+062#
;@A +461 ;@A
)+2 (7 $0 ;@A 67
+6(
!+047#
;@A +450 +566
)+1 (7 25 ;@A 72
+74
!+045#
;@A +4($ +604
)+$ (5 4 10 -1
+(1
!+025#
;@A +4-1 +602
D+2 4( 6 ;@A 4$
+--
!+047#
+47- ;@A +$74
D+1 4( 5 ;@A 44
+-(
!+0$2#
+466 ;@A +175
)+4 (5 10 0 -5
+(0
!+044#
;@A +451 +56-
>+1 (1 4 $ -4
+(4
!+02$#
+471 ;@A +612
.ata )omprehension !16 items#
>+4b (0 6( 15 6
+15
!+0$1#
+475 ;@A +5-(
.+2b (6 1- 47 $1
+57
!+0$6#
;@A +414 +5(5
5+1a -5 16 41 2-
+57
!+0$-#
;@A +45- +610
>+4a -- 5$ 1 $4
+$(
!+052#
+502 ;@A +564
>+$c (2 $- 10 44
+5$
!+04(#
+$7- ;@A +526
>+$d -- $2 - 4-
+5(
!+050#
+46( ;@A +570
%+2 (6 42 ;@A 54
+56
!+051#
+47( +414 ;@A
5+1b -6 $0 - 4-
+60
!+050#
;@A +4$4 +5(5
>+$b (1 $1 $ 57
+64
!+050#
+46( ;@A +564
'&
Ehi!it C-' 3continued4
%re.uenc$ Distri!utions, Mean Scores, and ,elia!ilities for Scored Items
Item ID
n
3Teachers4
Score %re.uenc$
M
3SE4
Alpha
/ /-< '
%orm
'
%orm
+ %orm 7
.ata )omprehension !continued#
5+1c -6 25 7 54
+67
!+04(#
;@A +455 +562
>+$e -( 26 6 57
+67
!+04-#
+4(6 ;@A +5(6
5+1o -2 24 5- -2
+71
!+051#
;@A +455 +605
>+4d (1 10 15 66
+-1
!+0$6#
+4(4 ;@A +5-6
.+2a (7 16 2 7(
+-2
!+0$-#
;@A +456 +5-$
>+2 (1 4 10 77
+(0
!+026#
+4(6 ;@A +5((
>+$a (1 2 - -1
+($
!+021#
+474 ;@A +5(1
.ata 7nterpretation !5 items#
.+2c (6 71 1 24
+26
!+044#
;@A +4$4 +567
5+2ab 7( 25 20 $4
+56
!+04-#
;@A +471 +5-7
.+4a ($ 44 0 4(
+5$
!+052#
;@A +444 +60$
5+2d 7( 27 10 42
+5(
!+052#
;@A +4-0 +5($
.+4b ($ $( 2 52
+57
!+051#
;@A +4-0 +55-
.ata Lse !1 item#
5+2c 77 0 20 57
+-7
!+025#
;@A +471 +57$
5ote. )ere +ere tree forms of te data scenario intervie+ and eac item appeared on t+o of te
tree forms.
'8
,elia!ilit$ and Scores for Data Scenario Euestions
For purposes of analysis, those questions on the data scenarios with right and wrong
answers were considered to be assessment items. In a case such as the present one, in
which the intent is to measure performance relative to a standard rather than performance
relative to peers, professional test publishers would estimate reliability through
techniques associated with item response theory. These procedures require several
hundred examinees for each item, however, and thus were out of reach for a research
project of the present scope.
Classical test theory can be applied with fewer respondents and examines the
relationship of each item to performance on the rest of the items. The complication in the
current instance is the matrix sampling of items done to manage the burden imposed on
individual teachers. There were three forms of the data scenario interview, and each data
scenario appeared on two of the three forms. To provide some indication of reliability,
analysts computed alpha coefficients, examining the relationship between an individual
item and the rest of the items in the data scenario interview, for each item and each of the
two forms of the interview that included it. These coefficients and shown in Exhibit C-1.
However, the purpose of the data scenario interviews was not to produce individual
teacher-level scores. The goal was to provide an estimation of data literacy at the school
level. Different teachers took different assessment forms containing different subsets of
the items, and different teachers total scores are not directly comparable.
Scales were analyzed at the school rather than the individual teacher level. That is,
the scores for all the teachers at a school who took item 1 were averaged and that mean
score was assigned to the school; likewise for item 2, etc. Through this process, the data
set was structured as 27 school-level records with a score for every item.
Reliability was estimated by computing an alpha coefficient for the total score.
Despite the small school sample and the intentional inclusion of distinct abilities in the
assessment, the reliability for the total score was fairly high, alpha = .74.
For all statistical comparisons reported in the body of the report, the school-level
total score was the variable used in the analysis.
'%