Forum

American Journal of Evaluation

31(3) 418-420
The Author(s) 2010
Advances in Evaluating Reprints and permission:
sagepub.com/journalsPermissions.nav
Evaluation Theory DOI: 10.1177/1098214010372250
http://aje.sagepub.com

Amy A. Germuth1

Keywords
evaluation, theory, research, culture

There is increasing recognition that evaluation theories and models should be more empirically
based (Mark, 2008; Shadish, Cook, & Leviton, 1991; Smith, 1993) and that prescriptions about how
to conduct evaluations using these theories should be based on empirical knowledge about practice.
However, as Mark (2008) notes, whereas approaches for conducting research and evaluation on eva-
luation use have become increasingly sophisticated, approaches for conducting research and evalua-
tion on evaluation theory are less well articulated and understood. Although research and evaluation
of evaluation theories are neither new ideas (Mark, 2001, 2003; Shadish et al., 1991; Smith, 1993;
Worthen, 2001) nor new practices (Amo and Cousins, 2007; Miller & Campbell, 2006), the three
preceding articles present new and different perspectives for guiding evaluation and research of eva-
luation theories.
Millers article argues that lack of frameworks and guidance may be factors related to the lack
of research that has been conducted on the role of evaluation theory in practice. In Developing stan-
dards for empirical examinations of evaluation theory, Miller outlines a taxonomy for empirically
studying critical aspects of the value of theory to practice that includes (a) operational specificity, (b)
range of application, (c) feasibility in practice, (d) discernible impact, and (e) reproducibility. She
notes that the knowledge gained by such study of evaluation theories should result in evaluations
that are markedly different on multiple dimensions, including the consequences of doing evalua-
tion in a particular fashion.
In Examining theory in cultural context, Kirkhart contends that because evaluation theory
guides epistemological, methodological, and practical choices, it is critical to question how well eva-
luation theory addresses dimensions of cultural context while conversely reflecting on how cultu-
rally biased assumptions may enter evaluation theory. She provides nine recommendations for
examining evaluation theories through a cultural lens that taken together are meant to clearly reveal
the cultural location of a theory while assessing and weighing the relevance and fidelity of a theorys
cultural representation. Importantly, she notes that careful consideration must also be given to what
is not addressed by the particular theory being assessed.

1
EvalWorks, LLC, Durham, NC, USA

Corresponding Author:
Amy A. Germuth, EvalWorks, LLC, 150 Solterra Way, Durham, NC 27705, USA
Email: agermuth@mindspring

418
Germuth 419

Smiths article asserts that how one assesses evaluation theories depends on how one views such
theories or models, whether as a type of intervention, an ideology, or as a set of principled proce-
dures or strategies. He makes a case that most theories can be viewed as a combination of the above
and that by viewing theories as part ideology, strategy, and intervention, this conception produces
tensions between whether one adopts standardized procedures versus adaptive practices and
makes difficult how to test specific models. He concludes that empirical studies of practice as Miller
proposes will tell us whether the model produces the promised results, but moral analysis (more
along the lines of what Kirkhart suggests) is needed to assess whether the espoused values are com-
patible with relevant professional, social, and cultural values.
As a practitioner who uses multiple theories to guide my evaluation practice, Dr. Millers article
appeals to me because it nicely outlines critical considerations I seek to clarify about evaluation the-
ories and their application. If one views evaluation theories as procedures or practices, then the
assessment of their operationability, applicability, feasibility, utility, and reproducibility flows natu-
rally within an evaluative paradigm. However, Kirkharts article reminds me that no evaluation
stands in a vacuum, everything must be evaluated within its cultural context, including the actual
theory guiding the evaluation. To do such is not easy, and is a task one may wish to avoid, but failing
to locate the cultural context of the evaluation and its guiding theory may have significant and dire
consequences for all aspects of the evaluation, not least of which includes stakeholder identification,
treatment, and findings. For me, this means that a more prescriptive evaluation of evaluation theory
may not suffice and that cultural assumptions and impacts cannot be overlooked or even worse,
ignored, when assessing the value of an evaluation theory. Smiths article nicely brings these points
together, asserting that how one assesses evaluation theories depends on how one views such the-
ories, whether as a type of intervention, an ideology, or as a set of principled procedures of practice.
He notes that most theories can and should be viewed as a combination of the above and that locating
theories within these characterizations makes it difficult to assess them using either Millers or
Kirkharts frameworks in isolation from each other. I wondered if meshing the two frameworks
together would produce a methodology that would allow a more robust examination of evaluation
theory as Smith suggests.
Where does this leave us? Although evaluation theories provide practitioners with multiple
views, guidance, methods, reasons, and political stances from which to conduct evaluation, these
articles purport that research on evaluation theories is itself beneficial in that research will further
clarify for practitioners these above aspects. Although I agree with this, my first question is what
happens if research into evaluation theories reveals conflicting benefits? For example, what if a the-
ory, when viewed as a procedure or practice is of high utility and reproducibility but is based on or
promotes cultural biases? What do we do with the theory? Do we take some aspects of it and leave
other aspects behind? If so, which parts do we do what with? In addition, who decides when the bias
is so great that such an evaluation theory or model should no longer be used or espoused as is?
Second, is it enough to caution that we must view and assess evaluation theories from multiple
standpoints (e.g., procedure, intervention, and ideology) or is a more prescriptive framework needed
to assure we assess evaluation theory more holistically? For example, might we need to review eva-
luation theories from multiple perspectives (not just cultural ones) such as those from various sta-
keholders to more fully assess the overall value and worth of the actual theory. This goes back to
the idea of whether there is a way to mesh the frameworks of Miller and Kirkhart to produce a more
robust measure of an evaluation theory.
My third and last question has two parts: where do we go from here and where do we hope to end
up? These articles have justified the need for assessing evaluation theories but they themselves pro-
pose methods, ideologies, and interventions for doing so. Do these need to be evaluated as well, and
if so, before or after assessing evaluation theories? In addition, who determines whether these are the
right or best approaches to take when assessing evaluation theories. Although these questions

419
420 American Journal of Evaluation 31(3)

in particular seem the most difficult to answer, I believe it is exactly the questions persons will ask
once they read the preceding articles. It is such questions that will continue to promote thinking
around how to assess evaluation theories and push evaluation of evaluation theory into the forefront
of evaluation.

Declaration of Conflicting Interests

The author(s) declared no conflicts of interest with respect to the authorship and/or publication of
this article.

Funding
The author(s) received no financial support for the research and/or authorship of this article.

References
Amo, C., & Cousins, J. B. (2007). Going through the process: An examination of the operazionalization of
process use in empirical research on evaluation. New Directions for Evaluation, 116, 5-26.
Mark, M. M. (2001). Evaluations future: Furor, futile, or fertile? American Journal of Evaluation, 22, 457-479.
Mark, M. M. (2003). Toward an integrative view of the theory and practice of program and policy evaluation. In
S. I. Donaldson & M. Scriven (Eds.), Evaluating social program and problems: Visions for the new millen-
nium (pp. 183-204). Mahwah, NJ: Lawrence Erlbaum.
Mark, M. M. (2008). Building a better evidence for evaluation theory: Beyond general calls to a framework of
types of research on evaluation. In N. L. Smith & P. Brandon (Eds.), Fundamental issues in evaluation
(pp. 111-134). New York, NY: Guilford.
Miller, R. L., & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review.
American Journal of Evaluation, 27, 296-319.
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice.
Newbury Park, CA: SAGE.
Smith, N. (1993). Improving evaluation theory through the empirical study of evaluation practice. Evaluation
Practice, 14, 237-242.
Worthen, B. R. (2001). Whither evaluation? That all depends. American Journal of Evaluation, 22, 409-418.

420