EVALUATION OF DETERMINING INSTRUCTIONAL PURPOSES

A Proposal Submitted to the Far West Laboratory for Educational and Research Development By Matthew Long of Keystone Educational Insight Associates

Introduction On October 1, 2011, the Far West Laboratory for Educational and Research Development (FWL) issued a request for proposal (RFP) for an evaluation of its Determining Instructional Purposes (DIP) training program. FWL is looking towards future growth but must determine if DIP is worthy of capital investment. This evaluation will look to provide FWL with information on whether there is a market for the program. In addition, the programs effectiveness will be examined as well as if one method of administering the program is more effective and/or efficient. This information will result in a recommendation from Keystone Educational Insight Associates (KEIA) of Somerset, PA on whether FWL should move forward in the creation and marketing of the DIP training program. Determining Instructional Purposes FWL has recently produced a training package that is geared towards equipping current and future school administrators with the skills needed to plan effective school programs. These skills are made up of three general processes; setting goals, analyzing problems, and deriving objectives. Each of these processes serves as a separate training unit. Clients may choose to use only the unit associated with their area of concern or any combination of the three units. Units can be administered via concentrated workshops or by individual sessions which are spread out over a longer timeframe. Units contain four-to-six modules with specific instructional objectives. Typical modules contain reading materials, practice activities, and feedback related to the activities. A coordinators handbook which provides information on coordinating the training courses is also part of the training package. All of these materials are produced in print form. Evaluation Method KEIA staff will meet with FWL stakeholders including the DIP designers to learn more about the training programs objectives. This meeting will clarify what the designers believe the training program will accomplish. A goal-based evaluation model will be used so that all stakeholders can be aware of the objectives that the evaluation is focusing on. Electronic surveys will be sent to school administrators in San Francisco school districts to gauge interest and attitudes towards participating in the training program. Two test groups will be selected from the respondents with each group consisting of eight participants, all from the same school district, and one coordinator. One test group will complete the training via a five day workshop while the other group will undergo training in three hour increments per day over a three week period. These time periods will allow each group to cover all of the material in each of the three units. Participants will take pre-participation and post-participation tests to see how their knowledge has changed when presented with situations involving planning effective school programs. These tests will be criterion-referenced and scored according to a rubric. This data will be used as one of the sources to evaluate the effectiveness of the instruction. The two groups test scores will
1

also be analyzed to determine if the method of administration plays a role in the effectiveness of the program. All participants in both groups will also take part in individual, semistructured interviews that will be conducted post-training. The interview questions will involve a branching process to gather feedback and opinions on the program, program materials, and method of administration. Interview responses will help determine whether the participants believed the program was a success and provide ideas for any recommendations of changes to the program materials. Responses will also serve as a data source for discovering if a preference for administration method exists. Project Personnel Matthew Long, a senior evaluation specialist at KEIA, will have the primary responsibility of planning and implementing the DIP evaluation. Mr. Long has served as the principal evaluator in 12 evaluations during his 10 year career, including projects for the Pennsylvania Department of Education, Carnegie Learning, and curriculum audits for several public and private educational institutions in the Mid-Atlantic region. He holds a B.S. degree in Earth-Space Science Education from Shippensburg University and a M. ET from Boise State University. Dr. Bethany Goshorn will design the survey and interview instruments for the evaluation, serve as the interviewer, and summarize and interpret the data. Dr. Goshorn is a Professor of Psychology at Pennsylvania State University, where she teaches graduate courses in Qualitative Research and Educational Statistics. She received her Ph. D. degree in Statistical Science from Duke University. Dr. Goshorn has published several journal articles published in on how technology is changing the way educational research is done. Ethan McNaughton will serve as the coordinator for the DIP test group that is doing individual sessions. Mr. McNaughton served as the Assistant High School Principal at Lowell High School in San Francisco for 17 years before retiring in 2008. While Mr. McNaughton has not previously served as a training coordinator, he has extensive experience taking part in training exercises related to the duties of being a school administrator. He will review, organize, guide, and monitor training activities as described in the coordinator handbook. Jessica Janes will serve as the coordinator for the DIP test group that is doing the five day workshop. Ms. Janes joined KEIA in 2009 after a seven year career as an educator/trainer for Carnegie Mellon University. Her previous work experience makes her a great fit as a coordinator for the DIP program. She will review, organize, guide, and monitor training activities as described in the coordinator handbook.

Task Schedule Task 1. Meet with FWL staff to discuss KEIA proposal, FWL-desired modifications if any, and DIP program objectives. 2. Submit data-collection plan for all instruments and draft copies of electronic surveys, pre- and post-test materials and interview protocols to FWL for review. 3. Provide KEIA with feedback on data-collection plan, electronic surveys, test materials, and interview protocol. 4. Review data collection plan and materials as appropriate. Submit final copies to FWL. 5. Send out electronic surveys to possible participants. 6. Provide program materials to program coordinators for review. 7. Summarize survey data. Teleconference with FWL staff to present and discuss data. Use data to select test groups from survey responders. 8. Conduct test group administered through individual sessions. Pre-participation test will be administered the preceding Friday. 9. Conduct test group administered through five day workshop. Pre-participation test will be administered the preceding Friday. 10. Administer post-participation test to both test groups. 11. Conduct post-participation interviews with participants. 12. Summarize test and interview data. Teleconference with FWL staff to present and discuss data. 13. Write final report and submit to FWL. Agency Responsible KEIA Deadline October 31, 2011

KEIA

November 21, 2011

FWL

December 5, 2011

KEIA KEIA

December 16, 2011 December 21, 2011 December 27, 2012

FWL

KEIA

January 9, 2012

KEIA

January 30, 2012February 17, 2012

KEIA

February 13-17, 2012

KEIA

February 18, 2012

KEIA

February 20, 2012

KEIA KEIA

March 12, 2012 April 16, 2012

Evaluation Budget PERSONNEL Matthew Long Bethany Goshorn Ethan McNaughton Jessica Janes TOTAL PERSONNEL OPERATING Travel 4 round-trip flights to San Francisco Rental car total of 10 days @$35 per day 14 Nights Hotel @ $105 per night 18 Days Per Diem @$75 Two Focus Groups Room Rental for 8 days @ $100 per day School District Meeting Room Rental for 15 nights @ $75 per night Meals (79 total) $10 plate Instructional Materials for all Units- 18 total Coordinators Handbooks- 2 total Communications Internet Mailings Supplies Miscellaneous supplies TOTAL OPERATING TOTAL PROJECT BUDGET $17,500 $14,300 $3,500 $1,550 $36,850

$1,600 $350 $1,470 $1,350

$800 $1,125 $790 $450 $9

$250 $150

$150 $8,494 $45,344