Practice: Evaluate your outcomes to show your

program is making a difference

Key Action: Design the most rigorous evaluation possible

TOOL: Decision Tree: Determining Feasibility for Rigorous Evaluation Design

Purpose: This decision tree guides you through a series of questions to help you determine
which rigorous evaluation design is most appropriate for your program.
Progressing from most to least rigorous designs, the flow chart presents the
criteria that must be met for each type of evaluation. The accompanying list of
evaluation design definitions provides an explanation of the structure and
methods used in each design.

Note: Design 1 or “Experimental” is not the only type of rigorous evaluation

design; designs 2 through 5 in the chart, which are all “Quasi-experimental,”
are also considered to be rigorous.

Instructions: 1. Review the decision tree and accompanying definitions.

2. Get answers to any questions you have about whether your schools meet
the various criteria.

3. Develop an opinion about which evaluation design will be most

appropriate.

1
 Practice: Evaluate your outcomes to show your
program is making a difference
Key Action: Design the most rigorous evaluation possible

Decision Tree: Determining Feasibility for Rigorous Evaluation Design

Are students Is there an Can you ensure that
randomly assigned oversubscription, or more students who aren’t
to magnet students applying than accepted do not enroll in EXPERIMENTAL DESIGN1
school(s) by YES are accepted to the YES a school similar to the YES
lottery? magnet school(s)? magnet school(s)?

NO NO NO

Quasi-Experimental Design2

Are consistent (5- Are consistent (5- Have attendance Are there a large
year) student year) demographic boundaries for non- number of schools
achievement data data available for magnet schools (e.g., 20 magnet Have students
available for YES magnet and non- YES been consistent for YES schools and 40 non- YES
used the same
magnet and non- magnet schools? the past 5 years? magnet comparison test for 5 years?
magnet schools? schools)?
Are consistent (5 Have attendance
NO NO NO NO NO

PRE-POST TEST OR POST-TEST ONLY5 Does the district contain pre-

conversion schools with similar
NO demographics and prior student
achievement? YES
NO
YES

INTERRUPTED
COMPARISON GROUP Are there multi-year COMPARISON GROUPS
student data in both
TIME SERIES3
PROPENSITY SCORING YES PROPENSITY SCORING
magnet and non-
TO MATCH STUDENTS4 magnet comparison TO MATCH SCHOOLS4 INTERRUPTED
schools?

Are there multi-year 2

 Practice: Evaluate your outcomes to show your
program is making a difference
Key Action: Design the most rigorous evaluation possible

Decision Tree for Magnet Program Rigorous Evaluation Design: Definitions

Refer to these key definitions of various rigorous evaluation designs to help you navigate the decision tree on page 2.
1 Experimental This design is also referred to as a “randomized controlled trial.” Experimental design is possible only when more students
Design apply for the magnet school than can be accommodated. When there is oversubscription, students can be randomly assigned
to the magnet school through a lottery system. It is also critical that nonselected students then enroll in school(s) that are not
similar to the magnet school to be able to attribute differences in outcomes to differences in the magnet program.
2 Quasi- All types of quasi-experimental designs share the characteristic of attempting to control for some unknown quality that may
experimental influence an outcome. An important variable is selection bias: whether the very act of choosing to attend a magnet school
design indicates a difference between a magnet school student and a non-magnet school student, even if both share key
characteristics such as demographics and prior achievement. All quasi-experimental designs, then, are efforts to “equalize” as
much as possible the two groups.
3 Interrupted This design requires student-level data from repeated intervals before and after magnet school attendance. Trends in each
Time Series magnet student’s outcomes, such as achievement prior to enrollment in a magnet school, are compared to trends after magnet
school attendance. Thus, the evaluation avoids the issue of selection bias by comparing each “chooser” with him/herself and it
is possible to attribute changes in outcomes to attending the magnet school. Interrupted time series require a fairly long history
of scores and sufficient numbers of participants to ensure that the changes are, in fact, associated with enrollment in magnet
schools.
4 Comparison Many quasi-experimental designs compare groups of magnet school students to groups of non-magnet school students. The
Group key to this type of design is to ensure as much as possible that students in the comparison group are similar to those in the
magnet school. One technique used is propensity score matching, which selects a comparison group using variables that are
most likely to have an effect on the outcome, such as demographics and prior test sores, in the case of magnet school
achievement. Often, the matching occurs in two stages—first, schools are matched on demographics, prior achievement, and
other variables of interest; then students within those schools are matched to form the comparison group.
5 Pre-Post- Depending on the data available, these types of comparison group designs may be the most rigorous design possible for your
Test or Post- evaluation. Both compare outcomes of magnet students with non-magnet students. In pre-post-test comparison designs, the
Test Only pre-test indicates differences in outcomes before magnet school attendance and, additionally, matching is used to make sure
the differences between the two groups of students is as small as possible. With post-test only design, the matching of students
needs to be strong in order to control for differences in post-test results between the magnet students and non-magnet
students.