Evaluation and Program Planning 25 (2002) 15±22

www.elsevier.com/locate/evalprogplan

A participatory evaluation of an inner-city science enrichment program

Glenda Quintanilla a, Thomas Packard b,*
a
Department of Children and Family Services, Adoptions Division, Los Angeles County, Los Angeles, CA, USA
b
School of Social Work, San Diego State University, 5500 Campanile Drive, San Diego, CA 92182-4119, USA

Abstract
A participatory evaluation (PE) of an inner-city science enrichment program for elementary school youth was conducted using an
evaluation team consisting of staff, board members, students, parents, and representatives of the agency's major funder. This evaluation
team designed and implemented the entire evaluation with guidance from an evaluator-consultant and researchers from a local university.
Data gathered from surveys of alumni, parents, students, and teachers revealed high satisfaction with the program and a validation of the
hands-on teaching model. Alumni reported that the program impacted their scienti®c and social competence. The highly participative design
process was seen as very successful by all participants, and offers useful guidelines for other PE, including active participation of all
stakeholders, commitment to a shared vision, and a good match between the organization and the evaluator. q 2002 Elsevier Science Ltd.
All rights reserved.
Keywords: Participatory evaluation; Organizational learning; Science enrichment; Appreciative inquiry

1. Introduction organizations and in actually using evaluation ®ndings are

This paper describes a participatory evaluation (PE) of an
inner city science enrichment program in San Diego: The
Elementary Institute of Science (EIS), which for over 30
years has provided after school, weekend, and summer
programs of hands-on science activities for youth who
historically have grown up to be under represented in the
science and technology ®elds. This case study will review
the participative evaluation process used to design and
implement the evaluation, resulting in immediately usable
®ndings and the creation of an internal capacity for ongoing
evaluation and organizational learning for program
improvement.
After providing a conceptual foundation for the evalua-
tion process used (PE and organizational learning) and a
review of similar science enrichment programs, the program
will be described. Next, the evaluation process will be
summarized, using Burke's (1998) participative evaluation
model. After a brief summary of evaluation ®ndings, we
will share some lessons learned.
well-known. The limitations of traditional methods have
been eloquently articulated by Schorr (1997), who criticized
the overemphasis on evaluability in traditional terms. She
supports ªnew approaches to the evaluation of complex
interventions [which] share at least four attributes: They
are built on a strong theoretical and conceptual base, empha-
size shared interests rather than adversarial relationships
between evaluators and program people, employ multiple
methods and perspectives, and offer both rigor and rele-
vance.º (p. 147) PE has emerged as a model which
addresses Schorr's concerns, particularly by emphasizing
the shared interests of stakeholders.
Patton (1997) has suggested that this participation of
stakeholders leads to an increased use of evaluation ®nd-
ings. Also, according to Burke (1998), ªwith low involve-
ment by the key stakeholders, there is a risk that program
bene®ciaries will lose a sense of ownership of the evaluation
®ndings. PE needs to be seen as an investment in the future.º
(p. 48)

3. Participatory evaluation
2. Issues in human services evaluation
Participatory evaluation ªis generally used to describe
Problems in adequately evaluating human service situations where stakeholders are involved in evaluation
decision making as well as share joint responsibility for
* Corresponding author. Tel.: 11-619-594-6723; fax: 11-619-594-5991. the evaluation report with an external evaluatorº (Turnbull,
E-mail address: tpackard@mail.sdsu.edu (T. Packard). 1999, p. 131).
0149-7189/02/$ - see front matter q 2002 Elsevier Science Ltd. All rights reserved.
PII: S 0149-718 9(01) 00045-3
16 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
According to Burke (1998), there are several key
elements of a PE process. First, of course, key stakeholders
must be involved in the decision making. The discussion
below will describe the processes used here to ensure in-
depth involvement of all key stakeholders in the process.
Second, the inequities of power and voice among participat-
ing stakeholders must be acknowledged and addressed.
Power dynamics which could be expected in a case such
as this would include relatively low in¯uence on the part of
parents and students, and high in¯uence of the representa-
tive of the agency's key funder; staff; and `high status' board
members, in this case those with graduate degrees. Perhaps
because of the agency's philosophy of highly valuing young
people and parents, there was no evidence of domination of
decision making by particular individuals or roles. This will
be discussed in more detail in Section 10.
Other elements of a PE process according to Burke
include: (a) ªan expanded role for evaluators that includes
elements of organizational learning and planned changeº
(p. 46), (b) multiple and varied data collection methods,
(c) an action component, (d) the building of the organiza-
tion's capacity for doing its own evaluation in the future,
and (e) an educational focus, on both individual and collec-
tive learning. The ways in which this project incorporated
all of these elements will be discussed below.
While the use of PE is growing (Cousins & Whitmore,
1998), Turnbull (1999) has noted the lack of empirical
research regarding this method. One study of PE, which is
relevant to the project described here concerned a large not-
for-pro®t organization which developed internal PE capa-
bilities (Minnett, 1999). This project parallels the Minnett
study, on a smaller scale, using external evaluators.
4. Organizational learning
Another principle guiding the project discussed here is
organizational learning (Argyris & Schon, 1996). While
historically human service organizations often agreed to
evaluation only under pressure to satisfy the needs of
funders or other outside groups, there has been growing
interest in using evaluations as opportunities to learn
about program operations to enhance effectiveness. Organi-
zational learning has become a popular approach for this.
According to Argyris and Schon (1996, p. xxi) organiza-
tional learning ªrefers broadly to an organization's acqui-
sition of understandings, know-how, techniques, and
practices of any kind and by whatever means.º
According to DiBella and Nevis (1998), ªthere are
three essential criteria of organizational learning:
First, new skills, attitudes, values, and behaviors are
created or acquired over time¼ Second, what is
learned becomes the property of some collective
unit¼ Third, what is learned remains within the
organization or group even if individuals leaveº.
(pp. 25±26)
Due to EIS's philosophy of valuing and accenting posi-
tive perspectives and visions, the organization also used
principles of appreciative inquiry in this evaluation.
According to Cooperrider and Srivastva (1987, p. 130),
appreciative inquiry is ªan innovative theory capable of
inspiring the imagination, commitment, and passionate
dialogue required for the consensual re-ordering of social
conductº (p. 131). This mode of inquiry is ªdistinguished by
the art and the science of asking powerful, positive ques-
tionsº (Cooperrider, 1996, p. 5). Appreciative inquiry seeks
to reveal the positive elements of an organization in order to
help achieve its ideal future (for a current review, including
some critical analysis, see a special issue of the OD Practi-
tioner: Sorensen, Yaeger, & Nicoll, 2000).
5. Inner-city youth and science education
National statistics indicate that as students enter middle
and high school, their participation in advanced courses in
science and math decreases. Students' decreased participa-
tion in advanced courses in science and math may be due to
ªthe reticence of elementary teachers to teach science
[which] has been linked to their perceptions of their ability
to understand and teach science wellº (Worch, Gabel, &
Odell, 1994, p. 401). Also, students' decreased participation
in advanced courses of science and math may be due to lack
of hands-on science instruction (Worch et al., 1994). One
national study reports that the number of students of color
(Hispanic and African±American) and female students
enrolled in science and math courses is lower compared to
other students (Campbell, Voelkl, & Donahue, 1998).
This situation has contributed to the creation of after-
school and weekend hands-on science enrichment pro-
grams geared towards minority and inner-city youth, who
seem to have the least opportunity to experience hands-on
science instruction in school (Brownstein & Destino, 1995).
Brownstein and Destino state that hands-on science instruc-
tion both engages students in the process of inquiry and
in¯uences social engagement in participating students.
Furthermore, they cite ªa considerable amount of evidence
[which] supports the connection made between the class-
room learning environment and cognitive and affective
learning outcomesº (p. 29).
6. Some science enrichment programs
As part of this evaluation, a `best practices' survey was
conducted to learn of the practices of other science enrich-
ment programs which could be adapted by EIS. Three not-
able examples will be summarized here.
Hands-on Science Outreach (HOSO) ªis a once-a-week,
after school, recreational science enrichment program for
children from pre-Kindergarten (age 4) through sixth
gradeº (Goodman & Rylander, 1993, p. 1) located in
 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
Rockville, Maryland. Three eight-week sessions are offered
each year during the fall, winter, and spring.
The Saturday Science QUEST is a science enrichment
program for elementary children and preservice elementary
teachers. The program ªexplore[s] concepts in speci®c areas
of science with an emphasis on the idea that science is all
around us and fun to learnº (Worch et al., 1994, p. 1). Also,
ªchildren spend 2.5±3 h per week exploring one major
[science] theme or topic¼using a hands-on approachº
(Worch et al., 1994, p. 1). It also ªuse[s] university labora-
tories and other science facilities¼[in order to] in¯uence
children to pursue science and science-related careersº
(Worch et al., 1994, p. 2). Additionally, cooperative learn-
ing is encouraged to enhance positive interdependence.
The Saturday Science Academy (SSA) of Clark Atlanta
University is a science enrichment program for African
American third±eighth-grade students located in Atlanta,
Georgia. The SSA was created in 1979 as a way of address-
ing students' limited exposure to science, especially among
African American students (Brownstein & Destino, 1995).
7. Background and setting
Founded in 1964 in an inner-city neighborhood of San
Diego as a `hands-on science enrichment program', the EIS
believes that children need to experience science `up-close
and personal' in order to become acquainted with this ®eld
and discover the vast knowledge and opportunities which
this ®eld generates. As a result, it adheres to the idea of early
intervention as a critical helping tool for children between
the ages of 7 and 13. This hands-on, experiential strategy
emphasizes that children need to experience science and
technology personally in order to understand and appreciate
these subjects. Research has found that students often lose
interest in these subjects because they are typically taught
through lectures and textbooks, with decreasing oppor-
tunities for hands-on projects due to shrinking educational
budgets (Worch et al., 1994, p. 401).
EIS provides a variety of classes geared to actualizing
students' success, including ªa variety of activities and
experiments in electronics, computer science, biology, engi-
neering, chemistry, physics, oceanography, photography,
and ecologyº (Elementary Institute of Science, 1996, p. 1).
EIS uses college students or recent college graduates as
teachers.
All classes, including biology, computers, engineering,
photography, and natural sciences, provide hands-on
instruction to stimulate, encourage, and nurture each
student's natural inquiry about the world. Students are not
tested, ranked, or scored in order to measure the level of
knowledge they have gained. Each instructor is allowed to
devise a unique curriculum relevant to his or her area of
expertise. EIS offers classes in after-school, Saturday, and
summer sessions. The summer session lasts 9 weeks and the
other sessions run from October to May.
17
This program model has been re®ned over the years; and
staff, children, and parents have seemed very satis®ed with
it. However, although EIS has been operating for more than
30 years, it has never undergone a program evaluation in
order to assess its operating strategies and techniques. As a
result, the EIS 1995 strategic plan included implementation
of a program evaluation as a strategic goal.
8. The evaluation design process
This evaluation will be described using the PE process
suggested by Burke (1998), based on her PE principles
which were summarized earlier. The PE steps are: (a) decid-
ing to do it; (b) assembling the team; (c) making a plan; (d)
collecting the data; (e) synthesizing, analyzing, and verify-
ing the data; (f) developing action plans for the future;
and (g) controlling and using outcomes and reports.
While Burke reported some signi®cant problems with one
particular PE, the case here represents successful applica-
tion of PE principles and methods.
8.1. Deciding to do it
As part of an agency strategic planning process, the board
and staff decided that a program evaluation would help the
agency accomplish several goals, including developing and
strengthening peer learning opportunities, establishing a
structure for tracking and involving alumni, and supporting
an ongoing planning process. An evaluation was expected to
provide documentation of program results, seen as impor-
tant to agency fundraising because ªcorporate and founda-
tion funders expressed the need for measurable outcomes
before considering funding supportº (Jacobs Center for
Nonpro®t Innovation, 1997, p. 18). The agency also wanted
to contact EIS alumni in order to `uncover stories' about
their EIS experience to discover commonalties which could
lead to improvement for EIS. Therefore, speci®c goals of
the evaluation were to learn about program outcomes and
accomplishments and to learn how the program could
become even better. The plan intended to include program
`success indicators' such as the percentages of students who
return for subsequent programs and who go on to college or
science-related ®elds, student interest in science, and
students emulating respectful relationships. These success
indicators were later used as a basis for developing evalua-
tion questions. The decision to begin an evaluation process
was made by staff, board members, parents, and the
agency's major funder. Students were not represented in
this decision, but these other stakeholders resolved to
involve them in all subsequent activities.
8.2. Assembling the team
Next, an evaluation design team was formed. Members
included representatives from the agency Board of Direc-
tors; staff; students; parents; and consultants from the
18 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
program's major funder, the Jacobs Center for Nonpro®t
Innovation (JCNI). The team explicitly intended to use
principles of PE and organizational learning to design and
implement an evaluation that was truly driven by program
participantsÐyouth, parents, staff, and board membersÐ
using outside experts to ensure that the process and methods
selected were sound from a research perspective. For the
initial design phase, expertise, primarily in the form of the
design and facilitation of planning workshops, was provided
by a consultant employed by the funder, JCNI. This consultant
was a doctoral student in urban planning who had extensive
experience in community development and group facilitation.
8.3. Making a plan
The original evaluation design team completed the EIS
evaluation design during ®ve workshops (typically 3 h
Saturday sessions) throughout 1997. This resulted in a
shared vision for the organization, including desired
outcomes and proposed evaluation questions, inputs, indi-
cators, and outputs. In order for the evaluation to ®t with the
mission, values, philosophical assumptions, and goals of
EIS, the team and the evaluation consultants recognized
that the evaluation process would need to be a hands-on
process, which would require the participation of all key
EIS stakeholdersÐstudents, staff, parents, and the Board
of Directors. The team intended that evaluation would
become an ongoing process, integrated into the operation
of EIS and owned by its stakeholders because it would be
their work product.
Following are the evaluation questions which the design
team developed.
1. Does EIS increase students' interest and con®dence in
the area of science?
2. Does EIS increase students' scienti®c competence?
3. Does the EIS environment nurture social competence?
4. Does EIS help students see a wider range of life options
(e.g. higher education, choice of science careers)?
5. How involved are families in the EIS program?
6. What visions and suggestions for the future do students,
parents, alumni, and staff have?
After the plan was completed, the group decided to take a
break from the intense process they had just completed
while they searched for an evaluation consultant to assist
with implementation of the evaluation. Eight months later, a
smaller evaluation team composed of a representative
subset of the original design team reconvened to plan imple-
mentation. To create a representative mix from each stake-
holder group, team members chosen included two EIS board
members (a professor of psychology who chaired the team,
and a physician researcher at a local internationally-
renowned medical research center), three parents, three
students who were also members of the agency's Youth
Board, two consultants from the agency's major funder,
and two staff: the executive director and the education direc-
tor. The decision to include students in all aspects of the
evaluation ¯owed naturally from the organization's
mission, vision, and values. The Youth Board had a promin-
ent role in the organization, most recently through their
commitment to raise funds for the capital campaign for a
new facility (Elementary Institute of Science, 2000). To
assist with implementation they hired the authors: a pro-
fessor and a research assistant from a local university. In
addition to possessing teaching and research skills, the
professor had been an organization development consultant
for 15 years, using a participative philosophy for working
with client organizations. We worked with the team as
evaluation consultants, providing methodological expertise
and facilitation of team meetings. This team held three
Saturday sessions of 3 h each over a 4 month period to
develop an implementation plan for the evaluation.
8.4. Collecting the data
The team made decisions regarding the targets for the
evaluation, data collection instruments, and evaluation
methods. They decided to gather data from alumni, parents,
students, and staff, using multiple data collection methods,
including structured interviews of alumni, pre- and post-
survey questionnaires for parents whose children attended
the program, questionnaires for teachers at the end of each
program series (fall to spring, summer), a short question-
naire for students older than age 10, and an analysis of
program documentation. The team determined that the
evaluation would be simple and unobtrusive, and would
be linked with the agency's strategic plan. An additional
aspect of the evaluation was a review of best practices in
hands-on elementary science education, in which one of the
authors looked for similar programs from across the country
to enable the evaluation team to assess other practices for
possible adoption to improve their program.
There were spirited discussions regarding several key issues,
such as whether or not scienti®c competence or attitudes
towards science should be explicitly measured and whether
or not changes in self esteem should be measured using stan-
dardized tests. The group ultimately decided not to use such
measures, because their obtrusiveness outweighed the value,
which they might have provided. This discussion also led the
team to conclude that the aspect of scienti®c competence,
which the program attempted to develop, was focused on
process skills such as the scienti®c method rather than on scien-
ti®c knowledge, which was seen as a responsibility of the
schools. The team concluded that questions regarding social
competence would best be answered by asking alumni directly
rather than through standardized self-esteem measures. These
discussions had a side effect of refocusing and clarifying for the
group the true intentions of the program, which emphasized
increasing life options for youth.
Another bene®t to this participative approach to evalua-
tion emerged through the discussion regarding who should
 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
conduct the interviews of alumni. One of the programs
discovered through the best practices analysis was a vision-
ing process for the City of Chicago (Cooperrider, 1996),
which involved having young people conduct interviews
with community leaders. Based on the Chicago experience
and EIS's own philosophy, the team decided to train some
of the Youth Board members to conduct alumni interviews.
Two Saturday training sessions were held for the training.
This process also served as a pilot test of the instrument and
a way to re®ne the interview questions and protocol. Triads,
consisting of an interviewer, and interviewee (alumni),
and an observer, role-played the interview process. The
agency's philosophy of involving and respecting youth
was demonstrated at one session during which the evalua-
tion team chair, a professor of psychology, role-played an
interview with a youth. The youth provided very useful
feedback on the wording of questions and the way the
professor asked them, with the professor responding appre-
ciatively and adjusting his approach. Another example of
the team's working to cultivate equitable relationships
among all stakeholders has been the use of a youth member
as a meeting chair. For one meeting at which the Chair was
absent, a youth member was asked to chair the meeting and
functioned effectively in that role.
Another component of the Evaluation Team meetings
was a ritual in which the team reviewed the evaluation
goals at the beginning of each meeting. Because the major-
ity of the participants had never participated in the planning
and implementation of an evaluation process, the review of
the evaluation goals provided them with a reminder of their
role in the evaluation. In addition, it helped keep the team
focused during meetings. A second ritual was an oral
critique at the end of each session, providing participants
with an opportunity to recognize their own and others' parti-
cipation in the evaluation process and to learn how to make
future meetings even more effective.
The team decided to collect data in the following phases:
I: alumni interviews;
II: questionnaires ®lled out by parents before and after
their children's participation in the program;
III: questionnaires ®lled out by teachers at the end of each
program;
IV: questionnaires ®lled out by students at the end of each
program;
V: collection and analysis of program documentation and
student demographics.
The team decided to continue interviewing alumni until at
least 50 had been contacted, and to make all other aspects of
the evaluation ongoing activities.
8.5. Synthesizing, analyzing, and verifying the data
Consistent with principles articulated by Burke (1998),
data from alumni, students, parents, and teachers were
19
collated by the researchers and presented at workshops to
the evaluation team and to the board of directors at two of
their regular meetings. While this paper is not intended to
report evaluation ®ndings, some selected summary informa-
tion may be of interest to a reader curious about the
outcomes of the process. Fifty three alumni were inter-
viewed, with over 70% reporting that EIS de®nitely or prob-
ably impacted their con®dence regarding both science and
social skills. Eighty-seven percent reported that EIS
de®nitely or probably impacted their increased interest in
science. The parent surveys began with the 1999 summer
program, and the student surveys began with the summer
2000 program. Parents and students reported very high
levels of satisfaction with the program. Parents reported
the achievement of stated program outcomes including
students' increased interest in science, enjoyment of an
intellectual experience, and learning how to do experiments.
The ®nal aspect of the evaluation, gathering of program
documentation, was seen as less urgent but very important
in the long term. As part of the development of the evalua-
tion, agency staff recognized that they needed to develop a
better system for tracking ongoing program activities such
as demographic data on students and teachers, program data
including students enrolled in each program and the length
of their participation, and parent involvement. Staff and
evaluators, with support of the rest of the Evaluation
Team, identi®ed program and demographic data to be
collected on a regular basis. These data will be entered
into a comprehensive database for ongoing monitoring
and analysis of student demographics and trends in enroll-
ment and program activities.
The Evaluation Team and the agency's Board of Direc-
tors were pleased with the ®ndings, since results were posi-
tive and highlighted opportunities for improvement that
were already planned (e.g. a larger facility) or were feasible
for implementation (e.g. year-round photography classes).
One board member noted that provisions should be made in
the new building for enough room for the photography
program. The board also encouraged staff to pay more
attention to orientation of new teachers, which staff had
already planned, based on their earlier review of the results.
Other suggestions from parents and alumni, such as the
interest in a program for older teenagers, will be used in
agency planning.
8.6. Developing action plans
The team made a commitment to use surveys of parents,
teachers, and students on an ongoing basis, with results fed
back to staff to help improve program offerings and opera-
tions. Data will also be used in reports to the board and
funders, and for proposals submitted to foundations for
additional funding. Notably, several months ago a prospec-
tive funder who had a particular interest in enhancing the
involvement of women in science careers asked the agency
for feedback data from female alumni. The researchers were
20 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
able to extract this information from the alumni survey data
and summarize this for use by the agency. Such requests
will be easy to accommodate in the future because the
evaluation system will be institutionalized as an ongoing
aspect of agency operations.
8.7. Controlling and using outcomes and reports
Consistent with one of Burke's principles, the data
remain in the hands of the stakeholders, to use to address
the original evaluation purposes: learning about program
accomplishments and identifying opportunities for program
improvements. After each academic year program and each
summer program, evaluators will collate data and the
Evaluation Team will meet to review results and develop
action plans. These results and action plans will also be
presented to the Board.
9. Lessons learnt
The alumni survey administration went well and met
expectations. The only change from the original plan was
that students were trained as interviewers but were not able
to actually conduct any of the interviews. Most alumni did
not want to come into the agency where the students could
interview them, so the interviews were conducted by the
researchers, almost all by phone. The convenience of a
phone interview made it easier to get alumni cooperation,
which outweighed not having students do the interviews.
We would only suggest to researchers considering using
this procedure that they facilitate explicit discussions
regarding the costs and bene®ts of the use of stakeholders
as interviewers and encourage the stakeholders to make a
thoughtful decision based on their particular circumstances.
The interview training process was useful nevertheless in
helping re®ne the questions and protocol as well as in giving
the team members an experiential feel for the data collection
process.
When staff and the researchers told the Evaluation Team
about the incompleteness of the agency's database of alumni,
all realized the importance of keeping more thorough and up
to date records for future use. In the author's experience,
many community-based organizations do not place a high
priority on maintaining data, which could facilitate contact
with former clients for follow up and evaluation. The
experiences here may help demonstrate the value of such
data management to administrators who in the future may
want to contact former clients for evaluation purposes. Of
course, such arrangements would need to be explained in
advance to clients, and client consent obtained for follow
up.
After the data were fed back to the Evaluation Team and
the board, members of the team were interviewed or
received a questionnaire to provide their observations on
the evaluation process and its results. A research assistant
who had not previously been af®liated with the project
conducted interviews. A total of eight stakeholders
responded. Interviews were held with two board members,
the original evaluation consultant (she was involved as an
evaluation team member but did not conduct the evaluation
or analysis of data), one parent, and one youth board
member. Surveys were sent to two parents, two youth, and
two staff, all of who had been members of the Evaluation
Team, and three of these surveys were returned.
Respondents were asked for their observations on what
factors had helped make the evaluation successful, what
could have been done better, and what recommendations
they would offer to others doing such evaluations. Key
success factors noted included active participation of all
stakeholders, the evaluation consultants' leadership and
guidance, and a good match between the client and evalu-
ators. An important implication here is that the client
organization should clearly articulate not only technical
requirements of evaluation consultants but also the agency's
philosophy and values and any expectations regarding how
the client and consultant will work together. Before a
consultant is chosen, there should be an open and frank
discussion of client and prospective consultant philos-
ophies, roles, and decision-making expectations to ensure
alignment.
Evaluation Team respondents also concluded that the
process could have been shortened, especially in the design
phase, perhaps by moving more quickly to a smaller group
so that decisions could be made more quickly, and by
having shorter intervals between meetings. A more explicit
and disciplined use of a timeline would help keep the
process moving so that momentum is maintained. If consult-
ants are sensitive to the time availabilities and pacing
preferences of the stakeholders (who in this case were
mainly volunteers, including board members and parents
with day jobs and students with time commitments includ-
ing athletics and other extracurricular time demands), they
can regularly have the group review progress to ensure that
events are on schedule or dates are adjusted. The stake-
holders suggested that such pushing from the consultants
on this might have helped. Because of our participative
philosophy, we may have held back from being overly
directive; but it seems that if a truly collaborative climate
has been established, consultants need not hold back in
making directive suggestions, as long as decision-making
remains with the stakeholders.
Based on the successes here, both the stakeholders and
the consultants strongly encourage others to heavily involve
key stakeholders throughout the process while using the
expertise of consultant/evaluators to manage project details.
We also learned the value of building the evaluation process
into regular program development. In this case, the evalua-
tion was linked with the agency's strategic plan and even its
capital fundraising campaign, enhancing its value and rele-
vance. Finally, the stakeholders would recommend to others
that they trust their instincts. As one respondent put it: ªmost
people close to an organization have a pretty good idea of
 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
what information they need.º A consultant's role in this then
becomes providing expertise on helping the stakeholders get
program models and objectives into evaluable forms and
suggesting valid and reliable data collection options.
One ®nal lesson, perhaps obvious but worth emphasizing,
is that the development of a comprehensive evaluation
which involves building ownership and stakeholder partici-
pation takes time. This is particularly true in the case of
volunteer stakeholders with other responsibilities. Meetings
may need to be scheduled on evenings and weekends, and
spread out over time. Consultants and stakeholders should
be patient as well as focused, and should keep a long-term
time perspective. This will be easier if the evaluation is seen
as the development of a new and valuable element in the life
of an organization rather than as a one-shot endeavor.
10. Final thoughts
In a recent discussion of PE, Cousins and Whitmore
(1998) identi®ed several issues for consideration in future
evaluations. The experiences on this project suggest
responses to some of their questions. First, who is in control
of the evaluation is often an issue. A related ethical issue
concerns who owns the ®ndings and can dictate the use of
them. There are also more technical questions: Who selects
participants? How is technical quality de®ned and by
whom? How will participants be trained? Finally, they
ask, what conditions enable PE to ¯ourish? The evaluation
process described here was successful due in part to the
power sharing among all key stakeholders. The evaluation
was funded by the agency's main funding source, and one
could expect that the funder may dominate decision-
making. In fact, agency staff, board members, students,
and parents participated fully in all decisions, with the
representative of the funder acting as an equal partner.
The data were seen as owned by the Evaluation Team and
the Board of Directors. The evaluation team, with guidance
and recommendations from the outside evaluator, made
technical decisions.
This process also con®rmed learnings articulated by King
(1998). High levels of interpersonal and organizational trust
were developed through the collaborative vision-setting
process, in which participants created shared meanings of
their experiences over time. Perhaps because of the organi-
zation's long-standing participative and parent- and student-
centered philosophy, its history of community involvement,
and stable, long-term ®nancial support from a local founda-
tion, power issues which often emerge in formal organiza-
tions were rarely relevant here; and when they did emerge
they were explicitly addressed through discussions of the
goals and processes of the evaluation and the roles of each
participant. In fact, the only substantive issues concerned
consultant suggestions to consider making the evaluation
more rigorous in traditional ways (e.g. the use of a control
group and standardized measurement instruments, sampling
21
techniques) and these were easily resolved after discussions
of the pros and cons of various options in this setting. The
consultants provided technical suggestions and guidance,
and facilitated discussions regarding issues such as ques-
tionnaire construction and the limitations of methods
which did not use traditional standards such as randomiza-
tion and control groups. The team took time to thoroughly
address such issues, even when this slowed down the
process. Doing this in a collaborative spirit seemed to
work well, with the team making key decisions and any
resultant methodological limitations noted.
Since its creation in 1964, the EIS has valued children's
inquiry about their world. It seems that the principles of
appreciative inquiry and organizational learning worked
here largely because they were consistent with the agency's
philosophy of creating a positive and exciting learning
environment for students and nurturing their inherently
inquisitive nature. These principles were implemented in
both explicit and subtle ways. The structure of the initial
evaluation design was known as the the `Learning Lab.'
Each meeting of the evaluation team began with a review
of the goals of the evaluation: to learn about program
outcomes and accomplishments and to learn how the
program can be improved. Each meeting ended with a
critique of the meeting's process and the results to date,
with re®nements and new actions planned to ensure that
the group built upon what it was learning and improved
its operations. Appreciative inquiry principles in particular
were re¯ected in the framing of the evaluation's purposes
and questions in positive ways rather than focusing on
problems. The initial vision of the Learning Lab was
based on positive principles such as the empowered child,
strong families, and beloved community; and emphasized
activities including `constant af®rmation and encourage-
ment' (Elementary Institute of Science, 1997). Meeting
critiques focused on not only opportunities for improvement
but also on a recognition of what was going well, and appre-
ciation for those involved.
EIS was able to design an evaluation process consistent
with its mission, vision, values, philosophical assumptions
and goals. Support and involvement from its stakeholders,
including staff, students, parents, volunteers, board members,
and funders helped EIS throughout the design and imple-
mentation of its evaluation process. Their participation
ensured that the variables being studied were those con-
sidered to be important. Despite differences in age and
education for the common purpose of implementing the
evaluation, members of the evaluation team came together
easily. The organization has a long-established culture of
valuing diversity, particularly related to ethnicity and age,
and this expedited team building on the evaluation to the
extent that little speci®c attention needed to be paid to rela-
tionship building.
Finally, this project supports conclusions by Whitmore
(1998) regarding essential ingredients for a successful parti-
cipative evaluation. There was a receptive context, with
22 G. Quintanilla, T. Packard / Evaluation and Program Planning 25 (2002) 15±22
staff, board, youth, and parents as well as the agency's main
funder being interested in the evaluation; the evaluators
were committed to a participatory process and possessed
strong `people skills' in areas such as meeting facilitation;
and enough time was allowed, over a period of three years,
for the vision of the evaluation to be developed and then
implemented. All these factors contributed to the building of
agency capacity regarding organizational learning and
evaluation, so that the process has been institutionalized
and become part of the culture of the organization.
Acknowledgements
Thanks to Ms Doris Anderson, Dr Billy Vaughn, Ms
Teresa Lingafelter, and the rest of the Evaluation Team at
EIS; and to the Jacobs Center for Nonpro®t Innovation for
their support.
References
Argyris, C., & Schon, D. (1996). Organizational learning II: Theory,
method, and practice, Reading, MA: Addison-Wesley.
Brownstein, E., & Destino, T. (1995). Science enrichment outreach.
Science Teacher, 62 (2), 28±31.
Burke, B (1998). Evaluation for a change: Re¯ections on participatory
methodology. In E Whitmore, Understanding and practicing participa-
tory evaluation. New directions for evaluation (pp. 43±56). Vol. 80.
San Francisco: Jossey-Bass.
Campbell, J., Voelkl, K., & Donahue, P. (1998). NAEP 1996 trends in
academic progress. US department of educationÐNational center for
education statistics. (On-line). Available: http://nces.ed.gov/NAEP/
96report/97986.shtml (October 1998).
Cooperrider, D. (1996). The child as agent of inquiry. OD Practitioner, 28
(2), 5±11.
Cooperrider, D., & Srivastva, S. (1987). Appreciative inquiry in organiza-
tional life. In W. Pasmore & R. Woodman, Research in organizational
change and development (pp. 29±169). Greenwich: JAI Press.
Cousins, J., & Whitmore, E. (1998). Framing participatory evaluation. In E.
Whitmore, Understanding and practicing participatory evaluation. New
directions for evaluation (pp. 5±24). Vol. 80. San Francisco: Jossey-Bass.
DiBella, A., & Nevis, E. (1998). How organizations learn, San Francisco:
Jossey-Bass.
Elementary Institute of Science (1996M). 31 years of hands-on science
enrichment. Science News, 1, 1.
Elementary Institute of Science. (1997, May 31). Learning lab. Available
from the Elementary Institute of Science, 588 Euclid Ave. San Diego,
CA 92114.
Elementary Institute of Science. (2000). A campaign for young minds.
Available from the Elementary Institute of Science, 588 Euclid Ave
Euclid Ave. San Diego, CA 92114.
Goodman, I., & Rylander, K. (1993). An evaluation of childrens' participa-
tion in the hands on science outreach program, Cambridge: Sierra
Research Associates.
Jacobs Center for Nonpro®t Innovation (1997). Elementary institute of
science: Campaign planning study, San Diego: Jacobs Center for Non-
pro®t Innovation.
King, J. (1998). Making sense of participatory evaluation practice. In E.
Whitmore, Understanding and practicing participatory evaluation.
New directions for evaluation (pp. 57±68). Vol. 80. San Francisco:
Jossey-Bass.
Minnett, A. (1999). Internal evaluation in a self-re¯ective organization:
One nonpro®t agency's model. Evaluation and Program Planning,
22, 353±362.
Patton, M. (1997). Utilization-focused evaluation, Thousand Oaks, CA:
Sage Publications.
Schorr, L. (1997). Common purpose, New York: Anchor Books Doubleday.
Sorensen, P., Yaeger, T. & Nicoll, D. (2000). OD Practitioner 32 (1).
Turnbull, B. (1999). The mediating effects of participation ef®cacy on
evaluation use. Evaluation and Program Planning, 22, 131±140.
Whitmore, E. (1998). In E. Whitmore, Understanding and practicing parti-
cipatory evaluation. New directions for evaluation (pp. 95±100). Vol.
80. San Francisco: Jossey-Bass.
Worch, E., Gabel, D., & Odell, M. (1994). Saturday science QUEST: A
science enrichment program for elementary children and preservice
elementary teachers. School Science and Mathematics, 94 (8), 401±
406.