Practice: Get quality data into your evaluator’s

hands
Key Action: Use technique to ensure valid and reliable data

VIGNETTE: Challenges of an MSAP Rigorous Evaluation for an Interdistrict Program

Purpose: Conducting a rigorous evaluation poses particular

challenges for interdistrict magnet programs. For example,
communication and data collection must be coordinated
across multiple districts; and appropriate comparison
schools may need to be identified outside the interdistrict
consortium. In this interview, an interdistrict magnet director
describes challenges that emerged throughout the rigorous
evaluation process, and the strategies her evaluation team
used to overcome them. Reflecting on her lessons may be
useful as you consider the feasibility of a rigorous evaluation
design in your own district.

Note: Although this vignette addresses the evaluation

challenges of interdistrict programs, the insights about
communication and comparison school selection may be
useful for all magnet programs.

Source: Interview with Karla Fawbush, Director of Magnet School

Programs, Northwest Suburban Integration School District,
MN, on November 21, 2008.

Questions for Reflection

1. What rigorous evaluation issues do you have in common with the Northwest Suburban
Integration School District?

2. What additional issues may you need to address as you design your plan for rigorous
evaluation?

3. How would you assess your district’s or consortium’s capacity to support a rigorous
evaluation of your magnet programs?

4. Are there rigorous evaluation demands you anticipate your district or consortium cannot
address? How do you anticipate this situation might affect your evaluation design?

1
 Practice: Get quality data into your evaluator’s
hands
Key Action: Use technique to ensure valid and reliable data

Challenges of an MSAP Rigorous Evaluation for an Interdistrict Program

Background: Northwest Suburban Integration School District (NWSISD) staff received federal
Magnet Schools Assistance Program (MSAP) funding to support new magnet schools in both the
2004 and 2007 cycles. After having its 2004 MSAP proposal for conducting rigorous evaluation
denied, the district’s 2007 proposal was accepted. In this interview with Karla Fawbush, Director
of Magnet School Programs for the interdistrict consortium, she describes how the evaluation
team used lessons learned during the 2004 cycle to create the rigorous evaluation design for the
2007 proposal. One aspect of the improved 2007 design was a stronger process for selecting
appropriate comparison schools.

Some history

The NWSISD is a consortium of seven school districts created in 2001 after two of the districts
were declared “racially isolated” by the Minnesota state legislature. In Minnesota, if a district has
20% more students of color than a contiguous district, it is considered racially isolated. The
schools and communities of the NWSISD work together to create solutions to intentional or
unintentional segregation. One of these solutions is the formation of magnet schools. The two
other interdistrict consortiums in the metro area run the operations of their magnet schools;
however, the NWSISD has 15 magnet schools run by the individual districts in its consortium. A
student can start with one of three magnet themes: Science, Technology, Engineering and Math
(STEM); International Baccalaureate (IB); or Visual, Performing, Literary and New Media Arts,
and start elementary school in one district, continue on to middle school or junior high in another
district, and then finish the strand in high school in a third district.

One challenge for our consortium is coordinating the magnet school operations of our seven
member districts, which vary in size, infrastructure and programs. We have small districts with
two-four schools, and we also have the largest district in the state. Members of our consortium
include urban, suburban and rural districts. They differ in how they coordinate curriculum and
staff development, how they transport students, and how they manage student data, for example.
Even before we dealt with the issue of the rigorous evaluation, we had to learn a great deal about
how each district functions and whom to contact for various information. For example, some
districts have a staff person for communications, while others do not. Specific roles and
responsibilities vary greatly from district to district. We had to work out the complexities of being
an interdistrict consortium, so that by the time we were awarded the second MSAP grant, which
included the rigorous evaluation, we had much of that structure in place.

2
 Practice: Get quality data into your evaluator’s
hands
Key Action: Use technique to ensure valid and reliable data

The challenge of finding comparison schools

There are some challenges that are specific to being an interdistrict program in terms of finding
comparison schools for a rigorous evaluation. For example, most of our member districts have
elementary schools, middle schools, and high schools, but one of our bigger districts has
th
elementary schools that run through 6th grade, and junior highs that run through 9 grade. In
preparing to gather data to compare schools, we need to take into account that one high school
may have students in grades 9-12 while a potential comparison school might have grades 10-12.
th
Therefore, since the state writing test takes place in 9 grade, we would either decide not to use
that test as a measure of student achievement, or we would want to find another comparison
school.

A major challenge in our rigorous evaluation design was selecting comparison schools for our
three new MSAP high schools, two of which are urban. The first step to finding appropriate
comparison schools was trying to match schools based on head count and demographic data: the
total number of students; the percentages of students who receive free and reduced-price lunch;
those who have limited English proficiency; and those who are part of a minority ethnic or racial
group. We focus mainly on our black population because that’s the largest racial/ethnic group in
the two urban schools. For our more suburban magnet high school, it was fairly easy to find a
comparable school since there are other similar suburban schools within our consortium. We
simply went to the Minnesota Department of Education (MDE) website and looked at the October
1, 2006, demographic data to find a school within a member district that had similar
subpopulations to those in our magnet high school.

But with the two more urban magnet high schools, it was more difficult. There are no schools of
any size that are similar in demographics within our member districts, so we had to go outside of
the consortium to find comparison schools. This was an additional complication to factor into the
process. We always start with the data that’s downloadable from the MDE website—head counts
and demographic information—but, of course, there’s additional information that we need for the
rigorous evaluation, so we need to obtain that data directly from the individual districts. We
narrowed the search for comparable schools to our local metro area because we thought it would
be easier to contact those districts to get their assistance. Initially we were hoping to find a
Minneapolis school because we had already gotten data from them during the previous grant
cycle. But as it turned out, the Minneapolis school district is starting its own programs with
treatments that are similar to those of the NWSISD magnet schools, so none of those schools
would have made a good comparison school.

After reviewing the data, we discovered that there were about five high schools in the metro area
that were pretty close to our two urban magnet schools in terms of demographics. They weren’t
perfect because school size wasn’t an exact match, but we knew that would be difficult to find:
one of our urban magnet high schools has about 800 students and the other has several
thousand. Then, among the five potential comparison schools that were the closest demographic
match, we also had to verify their programs. It turned out that one of the stronger comparison
school matches was just starting an International Baccalaureate Diploma Programme, which is
too similar to what we are implementing in our own magnet schools. So we went with the next
closest match. From there, we called their assessment and data management people and got
permission to collect and use the data from their district for the rigorous evaluation.

3
 Practice: Get quality data into your evaluator’s
hands
Key Action: Use technique to ensure valid and reliable data

Communicating with comparison schools outside our consortium

The staff members who work with the data have been very gracious about letting us use the data,
but some of the things we need for the rigorous evaluation, like pulling the grade-point averages
and individual test score data, isn’t something they regularly do. We appreciate their collaboration
and agreement to continue this process over the course of three years, even if they are not part
of our consortium.

Interacting with so many administrators throughout the consortium gives me the opportunity to
make our evaluation as thorough as possible. For example, through a very skilled assessment
director in one of our districts, we’ve learned about potential issues with one of state tests.
Originally, we planned to use math, reading, writing, and science tests, but after talking with this
assessment director, we felt that the science test was too new to use this year in the rigorous
evaluation. Since last year’s test was the first time the science test had been administered, some
students taking the test on the computer experienced technical difficulties. In addition, the science
test wasn’t used for AYP (Adequate Yearly Progress) and none of the three new Northwest
Suburban magnet schools has science as its principal theme. We factored those points into the
final evaluation plan, not using the test in the data analysis for the first rigorous evaluation report.