Problem of Practice Evaluation Plan Sean Murphy North Carolina State University 2011-2012 Dr.

Jennifer Corn

NELA Evaluation Plan Xtreme Reading

WHAT DID YOU DO? EVALUATION OF STRATEGIES

A: Train teachers on the 13 instructional stages used year long as core instructional strategies B. Train teachers on research based literacy strategies through Xtreme Reading Program C. Train teachers to use Possible Selves strategy D. Provide on-going support for Xtreme reading teachers and students Activities Evaluation Questions What do you need to know? Quantity A: How many hours were spent training teachers on 13 stages of instruction? How many hours were spent supporting teachers on the 13 stages of instruction? Quality A: Did the teachers think the session was of high quality? Measures/Data Sources How will you find out? Training agendas Support logs Professional Development Questionnaire Results What were the results? According to training agendas, teachers received approximately 4 hours of training on the 13 stages of instruction at the beginning of the year. According to the support logs, teachers received 7 hours of support specifically regarding teacher practices (13 stages of instruction). Teachers did not receive support for 13 stages of instruction (as a whole model) after initial training. However, there was specific support for differentiation and student collaboration. For the former teaching strategy, I provided 3 hours of support (mainly through directive support and modeling) and curriculum coaches provided .5 hours. Teachers received 1 hour of support on student collaboration from myself and 2.5 hours of support from coaches. 3

 Training agendas Quantity B: On how many strategies were teachers trained? How many professional development training sessions were given? Quality B: To what extent did teachers think the trainings were useful for their instruction? Support logs Professional Development Questionnaire

Teachers were trained on a total of 3 strategies during the first semester of implementation, conducted over 4 training sessions. Teachers received a total of 12 hours of training each during the Fall Semester. Teachers were trained on 2 strategies from January through March in 3 training sessions, totaling 8 hours for each teacher. Generally, teachers found the training increasingly useful as the year progressed. During the first session, 1 out of 3 teachers strongly agreed that training was useful for their instruction, 2/3 disagreed. For the second session, 1/3 strongly agreed and 2/3 agreed. With the third session, all teachers strongly disagreed that it was useful. All agreed the fourth session was useful. Questionnaires showed that 3/3 teachers felt confident and prepared to teach the Possible Selves strategy after training. According to interviews, teachers were introduced and discussed how to use Possible Selves strategies in the classroom. Sharing of Possible Selves lessons was facilitated by administration during weekly PLC sessions. Only 1 lesson was shared, student artifacts were collected but not reviewed. Support logs showed administration provided 26 hours of support either individually or with Xtreme teachers as a PLC, over the course of the school year. The most hours of support 4

Quantity C: To what extent did the session include an introduction to Possible Selves, discussion of how to use Possible Selves in the classroom, and sharing lessons that used Possible Selves? Quality C: How did teachers rate the quality of the training? Quantity D: How many hours of support were provided for the strategies?

Teacher interviews Professional Development Questionnaire

Support logs Likert scale

Quality D: How supported did teachers feel during the process?

Focus groups

occurred during November, while January saw the least hours of support. Support came in the form of coaching, modeling and facilitating planning. KU Coaches: 10, Admin Intern: 16. Support included coaching and facilitating planning. According to self-report likert scales, 3 out of 3 teachers strongly agreed that administrators prepared them for implementing strategies. They also strongly agreed that administration supplied adequate on-going support to effectively teach strategies. However, the feelings about curriculum coaches were less enthusiastic. 2 teachers agreed that the coaches prepared them for implementing strategies and 1 disagreed. Regarding the ongoing support from curriculum coaches, 1 teacher felt neutral about the level of support and 2 felt inadequately supported.

In a 30-minute focus group, teachers voiced negative feelings toward support 18 times and positive feelings toward support 6 times. It was stated that more on-going support and coaching was needed after initial training. Specifically, teachers felt neglected, abandoned, and used by the curriculum coaches after training. They generally agreed that administration provided on-going support (4 of 6 positive comments were directed toward administration). Decisions (Guiding Questions: What do the results mean? What are you going to do now?) Summary of Results (Interpretation of Data): 5

Before I could measure the success of teacher and student objectives, I first had to measure the success of the initial training. Preparation was broken into two distinct parts: The first sizable amount of training was devoted to teacher practices. Named the 13 stages of instruction, Xtreme Reading includes a set format for lesson design. The evidence-based routine requires teachers to devote class time to independent practice, modeling, differentiated practice and collaborative learning. The second block of time was spent training teachers on student practices, where numerous reading strategies are explicitly taught to enhance student literacy. Teachers were trained on the 13 stages of instruction once at the beginning of the year. Training on the Xtreme Reading literacy strategies were dispersed throughout the school year. A teacher perception survey was not collected for the initial training on the 13 stages of instruction. Over the course of the year, teachers were trained on 3 out of 6 literacy strategies, plus the Possible Selves motivational strategy. With the notable exception of the Word ID strategy (which was negatively skewed because of the online format), teachers agreed the training was strong. However, the results for support were not so positive. On the whole, teachers agreed that administration supplied adequate support to implement the Xtreme Reading program. Teacher had mixed perception about the support from Xtreme instructional coaches. This can be partially explained by time. Whereas administration was in the building everyday (and, as an intern, I was able to devote ample time specifically to these teachers), the coaches were only on campus sporadically. Whats more, a variety of unexpected situations kept them from coming to the school during the Fall. During this initial phase of implementation, administration provided the bulk of support. Next Steps (Action Steps): Provide More Spiraled Support for 13 Stages of Instruction: One major lesson learned, in reflection, was the necessity for clarifying the need for instructional change. One of the reasons Xtreme Reading was chosen as an intervention program was that it required differentiated and collaborative work. At the time of initial training it was assumed that this was already a routine, to some extent, in the classrooms. Thus, teachers did not have ample training on effective pedagogy, as presented in the 13 instructional stages routine. Informal and formal observations showed that South Johnston teachers relied heavily on direct instruction and assessment. This was true both before and after the program, proving the need to measure the amount of differentiated instruction and collaborative work. In preparing for next years implementation, much consideration and focused support should be given on instructional change. Offer more support. Three major changes should be made in supporting methods: 1) Frontload support, 2) Offer more ongoing support, 3) Make support more directive. First, after consulting with the KU coaches, it is clear that amount of 6

support should be more frontloaded next year, with consistent visits and updates during the first quarter. Second, while the amount of support might decrease in the second semester, open and regular dialogue should persist throughout the implementation (including analysis of data collected). The third point will be discussed in more detail below. At this point, though, it is worth mentioning that support should be more rooted in the materials and include specific follow-up about how instruction can be improved. Manage the Transition. In hindsight, it is clear that the transition from should have been planned and managed more effectively. Many of the teachers made it clear in support sessions that they were unfamiliar with specific strategies. They voiced apprehension with teaching reading strategies explicitly and had a hard time letting go of traditional instructional practices (predominately lecture and test). This could be expected, but plans should have including pathways forward, including methods for celebrating accomplishments and acknowledging areas for improvement.

WHY DID YOU DO IT? EVALUATION OF OBJECTIVES.

1. Teachers know and use research-based strategies effectively 2. Teachers differentiate instruction 3. Students collaborate during class 4. Students use literacy strategies proficiently Evaluation Questions What do you need to know? 1. How confident do teachers feel using research-based strategies? How often do teachers use researchbased strategies? Measures/Data Sources How will you find out? Professional Development Questionnaire Learning Strategies Walk Through Rubric Teacher Self-Report Survey Retrospective Pre/Post Student Survey 2. How often do teachers differentiate instruction? Learning Strategies Walk Through Rubric Retrospective Pre/Post student Survey Results What were the results? Based on survey data, 66% of teachers they strongly agreed that they were confident teaching three out of four of the strategies they received training. They were least confident teaching the Word ID strategy. In comments, teachers said this training was ineffective because it was conducted virtually through Face Time link. 26 copies of the Learning Strategies Walk Through rubric showed that teachers were using research-based strategies on 12 occasions; 4 in January, 6 in February, 2 in March. According to 26 copies of the Learning Strategies Walk Through Rubric, teachers differentiated instruction 18 times; 2 in January, 6 in February and 10 in March. Differentiation consisted of purposeful organization of group work, leveled texts, task variety and individual support. 8

3. How often do students collaborate during class?

Retrospective Pre/Post Survey Learning Strategies Walk Through Rubric

According to a retrospective pre/post survey, Xtreme Reading students said the number of times they collaborated in the classroom increased from the first days of school to the middle of spring. Specifically, in response to a prompt asking, how often did you collaborate the beginning of the school year, 33% responded about once a week, 29% responded once or twice a week and 14% collaborated everyday. In the same survey, a prompt asked student, how often do you collaborate now? 20% of students marked that they collaborated once a week, 42% reported once or twice a week and 25% said they collaborated everyday.

In 26 copies of the Learning Strategies Walk Through Rubric taken from December-March, students were collaborating 18 times. Collaboration consisted of student pairedreading 8 times and collaboration on projects 10 times. 2 in January, 6 in February, 10 in March. 4. Pre-Post Assessments Pre assessments were given for 3/4 of the How many students used the for Each Strategy strategies; however, post assessments were strategies proficiently? only given for 2/4. Of these two post assessments, teachers did not keep the student answers. Decisions (Guiding Questions: What do the results mean? What are you going to do now?) Summary of Results (Interpretation of Data): Results suggest that the Xtreme Reading program led to changes in both the type of content presented and the way in which it was taught. Because the program required direct instruction of reading strategies (whereas other South Johnston English classes did not), Xtreme Reading constituted new content for teachers. In addition, because each strategy was to 9

be taught using effective pedagogical routines (e.g. differentiation and student collaboration), we expected teacher habits to change as well. Of initial concern; however, was whether or not the training effectively prepared teachers. Survey responses showed that teachers felt confident about teaching the strategies. Because the program was designed to impact both the what (content) and how (method) of teaching, year one of the Xtreme Reading program could be considered a moderate success. According to the evidence collected, it seems that content was more readily adopted, though method had a more sustained impact. After receiving training on a particular strategy, teachers immediately began explicitly teaching it. Analysis of walk through rubrics suggest that after an initial period of implementing the strategies as trained, teachers reverted back to familiar content (such as rote vocabulary). Conversely, the how of their pedagogy showed a more sustained impact. The key indicators for this included differentiated instruction and collaborative group work, as measured by pre/post student perception survey and Walk Through rubric. Initially, teachers used student collaboration and lesson differentiation only when teaching the Xtreme Reading strategies. Over time, though, teachers began using these practices when teaching their traditional content. Ultimately, the amount of meaningful differentiation and student collaboration escalated for all three teachers over the school year. This was noted in both walk-through observation and student perceptional data. It is hard to credit Xtreme Reading training and support with this growth, as numerous other factors could have contributed. However, a central focus of the training and support was on differentiation and collaboration. Next Steps (Action Steps): Utilize the programs resources. This was alluded to above in the offer more directive support bullet. The Xtreme Reading program has a vast amount of resources, including lesson plans, pacing outlines, and formative assessments that can inform instruction. Yet, almost all of the support we offered (especially in the beginning of the year) did not put these materials to use. On a regular basis, administration and/or instructional coaches should sit down with the teachers and the materials. This can be used to ground conversation. Teachers should reflect on how closely their planning and pedagogy follow the guidelines. Of course, teachers should have the autonomy to make modifications; however, they must know what the standard looks like and try adhering to the program as it is designed. Make Goals Clearer and More Prominent: In retrospect, the outcomes for the program shouldve been framed in both teacher and student terms. We were trying to change instructional habits as much as content. However, this is a more delicate process, as former teaching habits were already in place, demanding attention to the transition. Whats more, the entire instructional cycle should have been emphasized, so that teachers were constantly reminded how differentiated teaching and collaborative learning fit into the framework for teaching. Once the goals were stated more clearly, systems should be in place to regularly monitor progress. The walk through form could 10

have been used more consistently with teachers, anchoring coaching conversations.

11