Reinventing Evaluation Author(s): Rodney K. Hopson Source: Anthropology & Education Quarterly, Vol. 36, No. 3 (Sep.

, 2005), pp. 289-295 Published by: Wiley on behalf of the American Anthropological Association Stable URL: http://www.jstor.org/stable/3651397 . Accessed: 20/03/2014 03:26
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Wiley and American Anthropological Association are collaborating with JSTOR to digitize, preserve and extend access to Anthropology &Education Quarterly.

http://www.jstor.org

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

Commentary Reinventing Evaluation

RODNEYK. HOPSON

Duquesne University This commentary reviews "Negotiating Researcher Roles in Ethnographic Program deliberations Evaluation"and discusses the changingfieldof evaluation.It situatespostmodern in evaluation anthropologyand ethnoevaluation,two concepts that explore the interdisciplinary merger in evaluation, ethnography,and anthropology.Reflecting on Hymes's seminal work, the commentary invites reinvention of evaluation and consideration of its value, purpose,and use in scholarlyanalyses of schooling, human learning, problemsof practice,the teaching of anthropology,and larger theoretical questions within the field. [educational evaluation, evaluation anthropology, ethnoevaluation, postmodernism] The article by Linda Harklau and Rachel Norwood provides opportunity to contribute to discussions about the changing field of educational and program evaluation, and to rethink evaluator roles or subject positions. The article is especially timely in light of postmodern deliberations in evaluation and other disciplines and the increasing need to redefine and revisit disciplinary relevance in a world challenged by shifting cultural and political orientations, ideologies, and conflict. I offer further evaluation extrapolations to the already messy topic of postmodern evaluation, its implications, and ways of viewing important questions in the field. Rediscovering Ethnographic Program Evaluators: A Withering Species or an Opportunity Whose Time Has Come? The problem of ethnographic program evaluation posited by Harklau and Norwood suggests that the evaluator who practices ethnographic techniques is either a species on the verge of extinction or a marginalized group. They write at the outset that despite the increasing need for evaluators and evaluation, and in light of standardsbased education reform, accountability mandates, and performance-based initiatives (Fitzpatrick, Worthen, and Sanders 2004), "ethnographic program evaluation seems to be marginalized both within evaluation research and the anthropology of education." Furthermore, the authors' recognition of the dearth of articles on this topic in this journal is significant considering the journal's mission as well as that of Committee 4 of the Council of Anthropology and Education: Ethnographic Approaches to Evaluation in Education. Still, this seeming disciplinary marginalization offers opportunity, even promise, for providing value-added positions that make clear how evaluation and ethnography converge. Elsewhere (Hopson 2002), I too suggest that too little attention has been paid to the role of the ethnographer in evaluation and to the connections between evaluation and ethnography. Using Fetterman's conceptual underpinnings

Anthropology and Education Quarterly, Vol. 36, No. 3, pp. 289-295, ISSN 0161-7761, online ISSN 1548-1492. ? 2005 by the American Anthropological Association. All rights reserved. Direct all requests for permission to photocopy or reproduce article content through the University of California Press's Rights and Permissions website, www.ucpress.edu/journals/rights.htm.

289

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

290

& Education Quarterly Anthropology

Volume 36, 2005

of ethnographic evaluation (1984, 1989) and Stake's (1991) appeal to fourth-generation evaluation, I suggest that the historical development of ethnography and evaluation contribute to timely conceptual, methodological, and epistemological interdisciplinary possibilities for ethnoevaluation. The social constructivist perspective of evaluation from which the ethnoevaluator operates promotes a blending of traditions that values the cultural interpretation of ethnography with the judgment-framing and description-forming of evaluation. Merging multiple roles, the ethnoevaluator helps uncover complex social realities and contexts of programs and policies through formative or process evaluative decisions and by promoting holism and cultural interpretation of events and activities experienced by program beneficiaries and other stakeholders. Another disciplinary quid pro quo is emerging through the blending of topics such as practicing anthropology, evaluation theory, applied anthropology, cross-cultural evaluation, context in evaluation, and history of evaluation with applications in environment, health care, technology, community development, and philanthropy (Butler and Copeland-Carson in press). Evaluation anthropology does more than bring together anthropologists who earn their livelihood through evaluation contracts; it is a way of reinventing evaluation. Copeland-Carson (in press), for instance, both wonders about and defines a specific role of the anthropological evaluator, one that crosses boundaries of theory and practice and recognizes the opportunities and barriers that pervade evaluation anthropology scholarship: After all, what else is evaluation if not the systematicstudy of a culturalsystem to tell the story of its development,efficacyand impact,and translatethis story for various audiences? The socioculturalsystem may be a school, a corporation,or a health care clinic. Regardless of the focus, much of our role is to describeand analyze the socioculturalsystem that grows up around a planned interventioncalled a project,a program,or an initiative.So, one way of seeing our work in anthropologicalevaluation is that we are practicingethnographers who specialize in particularcultural institutions or contexts (public/private partnerships, environmentalprojectsin Africa,microfinancein ruralAmerica,etc.). Like all ethnographic activities, anthropological evaluation is implicitly based on theories that underlie the methods of the entire discipline. Butler and Copeland's volume raises as many important questions as answers, echoed nearly a decade before by Linda Camino (1997) in another bulletin of the National Association for the Practice of Anthropology (Reed 1997a). Distinguishing anthropological and non-anthropological ethnography, and blending similarities between practitioners of classic ethnography and qualitative evaluation researchers, Camino contemplates how ethnography is construed, its place in evaluation research, the value of specific anthropological training for purposes of evaluation, and valueadded applications of evaluation and ethnography. She suggests that anthropologists have several "preadaptive traits" that are advantageous to the ethnographic evaluation, including * the initial approach to the field, which includes encouraging stakeholders to grasp the importance of notions of "ethnography" or "culture" or preparing for field visits through literature reviews that reveal multiple cultural contexts within which programs and projects operate; * use of ethnographic which discerns when to use what set of methods or methodology, how to use the spectrum of ethnographic inquiry and fieldwork methodologies;

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

Hopson

Evaluation Reinventing

291

* knowing how to handle data, both as a dialectical, iterative, and investigative process and how to balance and distinguish description, analysis, and interpretation in the evaluative process; * presenting results through creative reporting media and in understanding and presenting stakeholder views, and especially in advocating for marginalized or "hidden" groups; and * working with stakeholdersthrough multiple role areas such as in communication, negotiation/collaboration, and facilitation/teaching. The spirit of the article by Harklau and Norwood, as well as this commentary, recognize the opportunities for evaluation, ethnography, and anthropology not as some fugacious or shoddy approach (what Rist [1980] once described as "blitzkrieg ethnography") without rigorous and credible display or purpose. The challenge in today's postmodern scenario is to resist static and inflexible turf spaces, institutional structures, and assumptions (see Reed 1997b for his discussion of these "blindspots") that pigeonhole the evaluator, the ethnographer, and the anthropologist and instead to accentuate how a multivocal and contested project terrain may benefit from the confluence and self-reflexivity of interdisciplinary perspectives. Relocating Postmoder Evaluation

The very nature of postmodernism implies a changing state of knowledge and reaction to claims about truth, ideology, representation, authority, power, and identity. The postmodern condition revisits and rethinks cultural phenomena and assumptions typical of Western European universalisms and rationalism, with an eye toward new cultural and political orientations (Peters 1997). Bain's (1997) use of Lyotard's theories typifies how metanarratives are embedded in the notion of history, progress, and emancipation. That "progress" of the West is also rooted in a history of subjugation, exploitation, and enslavement is a discourse that reimagines and resituates human reality. Suggesting that Lyotard's question of the development of knowledge is unrelated to ideas of freedom and emancipation, Bain (1997:4) reminds us that We recognizenow that we have "progressed" so far, at least in the United States,that we a of our than imprison greaterpercentage countryand speany other"developed" population cializein imprisoning poorpeople and people of color;thatthe disparitybetweenrichand poor is growing;thatxenophobiais on the rise in a nationof immigrants; thatwe have "progressed" so far thatour primaryvisions of the "endof history"arenuclearor ecologicalannihilation. The emphasis on critique, questioning, multiplicity, and proliferation of forms of reason are important characteristics of the postmodern condition, all applicable and timely notions in current and future discussions in evaluation. Postmodern evaluation deliberates on these same notions of shifting power, ideology, truth, identity, and representation (Mabry 1997; Mabry et al. n.d.; Ryan and Schwandt 2002). For example, Greene and colleagues (2004) challenge neutral, value-free claims toward evaluative knowledge. Noting the dominant social reality that reifies privilege and disadvantage through social order, social policies, and socially constructed definitions of who is deserving, educated, and valued, we suggest that evaluation has a liberatingrole to play in extricatingsociety from this particularsocial construction of privilege and advantage. Given that evaluation inevitably advances certain values and

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

292

Anthropology& EducationQuarterly

Volume 36, 2005

interests and not others, it can either maintain and reinforcethe existing system, or it can serve to challenge,disrupt,and strive to change the existing social orderto one that is more equitable,and just more democratic.And one of the most importantways that evaluation can serve democratizing values is by generating and legitimizing multiple alternative constructionsof reality,for instance, through diverse stories about people's daily experiences in differentsocial contextsand throughvaried judgments,arisingfrom these different experientialcontexts, about the quality and effectiveness of particularsocial policies and programs.[Greene,Millet, and Hopson 2004:102-103] As Harklau and Norwood illustrate with their discussion of the Promise Program, and as Reed (1997b) and LeCompte et al. (1999) also point out, the ambiguity, conflict, and resistance around roles are important though hardly new to ethnographers. The roles reified by the Promise ethnoevaluators also imply assumptions about how evaluation is valued and represented in wider educational and societal arenas. Here, let me reflect on the role of the lab-coated (evaluator) clinician. This role is reflective of claims about "scientific" evaluation embedded in current federal and state education programs. In contrast, Denzin's (2005) "mandate" for the first International Congress of Qualitative Inquiry rejects regulatory and prescribed ways of thinking, doing, and believing about our educational arenas and the larger society, as incorporated in current funding proposals, program operations, and institutional and academic structures. Denzin hints at resisting governmental proclamations on what counts as "science" and appropriate educational research methods-including evaluation methods-critiquing the implications of neoconservative discourse, methodological conservatism, and fundamentalism that proclaim "gold standards," "best practices," and "guiding principles" in the name of scientifically based education reform. Hence, assuming the evaluator role is not like trying on clothesthat is, finding an appropriate suit that fits from an array of predetermined roles and positions. These evaluator roles have implications beyond program development, micro examples, and applications of power and discourse at agency or project levels; they reify dominant discourses and power dynamics shaped by larger societal triggers. Pushing the Postmodern Envelope in Evaluation Just as Hymes (1999 [1969]) posited the reinvention of anthropology more than 30 years ago, the postmoder push in evaluation seems a ripe opportunity for rethinking and reinventing the field of evaluation. On the cusp of the Civil Rights Movement and changing global and sociopolitical landscapes (including the birth of what we call the modem era of evaluation), Hymes's reexamination of the field of anthropology was timely. His choice of topics was as salient to the field then as it is now, a precursor of things to come and testament to anthropology's relevance to a changing world. Hymes questioned aloud the "ethics of research, the relations between anthropologists and those who are or have been studied, [and] what is legitimate as a topic" (1999:xiv). This reinvention of anthropology as a field was inextricably linked with a host of factors worth brief explanation as they relate to evaluation. One of the initial questions posed by Hymes was identity: Who can be an anthropologist? He suggested an epiphany for the field characterized by the realization that "our own cultural backgrounds and unexamined assumptions" inform conceptions of anthropologists and fieldwork. There are two implications Hymes makes clear, as

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

Hopson

Reinventing Evaluation

293

have others in the evaluation field (Hood 1999a, 1996, 2001; Hopson 1999). First, there is a need for "increased participation in anthropology by people of diverse cultural backgrounds" (Hymes 1999:xv). This is an obvious statement to those of us committed to diversifying our professional and disciplinary ranks. I contend that the continued and increased significance of evaluators of color is not about numbers but about ensuring a melange of perspectives that help unseat the societal prejudice and stereotypes that influence disciplines, ideologies, and methodologies. Second, Hymes suggested a need "for those who study to be studied" (1999:xv). This appeal, taken up more recently by Nader (1999), stirs rethinking of the adequacy (accuracy or validity in evaluation standards terms) and relevance of anthropology. Instead of "studying down" (i.e., focusing research questions on marginalized are you poor? Why do your peoples that relay self-fulfilling prophecies-Why children not function well at school on standardized tests?), Nader suggests that anthropologists also make a habit of "studying up" (i.e., focusing questions on the most powerful strata of society-Why are you so affluent? How do your children outperform other students in schools?). (See also Hamann 2003.) The implications for studying up involve more than a framing of evaluation questions; studying up is a deliberate way of exposing potentially transformative, democratic (Greene, Millet, and Hopson 2004; Mertens 2005), and emancipatory lenses in the evaluation enterprise for the purposes of exposing power implications of seeking truth, contributing to knowledge, generating theory, and using appropriate methods. Another tacit implication of Hymes's question involves distinguishing the field of anthropological evaluation in the same way the postmodernist wonders: What is evaluation? Who is an evaluator? What roles and identities must an evaluator play? Denzin (2002) pushes the postmoder envelope by situating a place for evaluator performances and a vocabulary for performance in evaluation. The performance "turn" in evaluation is at its core participatory and experiential, weaving anthropological and performance studies alongside critical pedagogy and cultural studies in the evaluation exercise. These identities go beyond recognizing evaluation as the systematic assessment and procedure of judging worth or merit but also highlight and seek new ways of understanding oneself and presenting in ways understood by diverse audiences. In Denzin's words, the "liminal evaluator engages in evaluation to discover meanings for herself. She works the hyphens, the spaces between her professional identity and the worlds of experiences that are embedded in the evaluation site" (2002:147-148). There are additional questions Hymes poses-What can be studied? What can be known? But I hope the point is clear that the questions in anthropology 30 years ago not only remain relevant for anthropology today but also apply to research methodologies, techniques, and interdisciplines such as those used in evaluation anthropology and ethnoevaluation. Reinventing evaluation is a postmodern act, in which one does not settle for the static descriptions and representations of the field but helps to reshape it, posing questions that situate evaluation and anthropology in sensitive, political, and epistemological terrains.

Rodney K. Hopson is associate professor of evaluation, educational ethnography, and sociolinguistics,and chair of the Departmentof EducationalFoundationsand Leadershipat Duquesne University (hopson@duq.edu).

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

294

Anthropology& EducationQuarterly

Volume 36, 2005

References Cited
Bain, William 1997 The Loss of Innocence: Lyotard, Foucault, and the Challenge of Postmodern Education. In Education and the Postmoder Condition. Michael Peters, ed. Pp. 1-20. Westport, CT: Bergin and Garvey. Butler, Mary 0., and Jackie Copeland-Carson, eds. In press Creating Evaluation Anthropology: Introducing an Emerging Subfield. National Association for the Practice of Anthropology Bulletin No. 24. Berkeley: University of California Press. Camino, Linda A. 1997 What Can Anthropologists Offer Ethnographic Program Evaluation? In Practicing Anthropology in a Postmodern World: Lessons and Insights from Federal Contract Research. National Association for the Practice of Anthropology Bulletin No. 17. Michael C. Reed, ed. Pp. 41-57. Washington, DC: American Anthropological Association. Copeland-Carson, Jackie In press "Theory-Building" Evaluation Anthropology: Bridging the Scholarship Versus Practice Divide. In Creating Evaluation Anthropology: Introducing an Emerging Subfield. National Association for the Practice of Anthropology Bulletin, No. 24. Mary O. Butler and Jackie Copeland-Carson, eds. Berkeley: University of California Press. Denzin, Norman 2002 Performing Evaluation. In Exploring Evaluator Role and Identity. Katherine Ryan and Thomas Schwandt, eds. Pp. 139-165. Greenwich, CT: Information Age. 2005 The First International Congress of Qualitative Inquiry. Electronic document, accessed March 21, 2005. http://www.qi2005.org/DenzinICQI.pdf, Fetterman, David M., ed. 1984 Ethnography in Educational Evaluation. Beverly Hills: Sage. 1989 Ethnography: Step by Step. Newbury Park, CA: Sage. Fitzpatrick, Jody L., Blaine R. Worthen, and James R. Sanders. 2004 Program Evaluation: Alternative Approaches and Practical Guidelines. 3rd edition. New York: Longman. Greene, Jennifer C., Ricardo A. Millet, and Rodney K. Hopson 2004 Evaluation as a Democratizing Practice. In Foundations and Evaluation: Contexts and Practices for Effective Philanthropy. Marc T. Braverman, Norman A. Constantine, and Jana Kay Slater, eds. Pp. 96-118. San Francisco: Jossey Bass. Hamann, Edmund T. 2003 Imagining the Future of the Anthropology of Education if We Take Laura Nader Seriously. Anthropology and Education Quarterly 34(4):438-449. Hood, Stafford 1999a Creating Evaluation Strategies Appropriate for African Education. Paper presented at the First Annual Meeting of the African Evaluation Association, Nairobi, Kenya, September. 1999b Assessment in the Context of Culture and Pedagogy: A Collaborative Effort. Journal of Negro Education 67(3):184-186. 2001 Nobody Knows My Name: In Praise of African American Evaluators Who Were Responsive. New Directions for Evaluation 92(Spring):31-43. Hopson, Rodney K. 1999 Minority Issues in Evaluation Revisited: Reconceptualizing and Creating for Institutional of Evaluation American Journal 20(3):445-451. Opportunities Change. 2002 Making (More) Room at the Evaluation Table for Ethnography: Contributions to the Responsive Constructivist Generation. In Exploring Evaluator Role and Identity. Katherine E. Ryan and Thomas A. Schwandt, eds. Pp. 37-56. Greenwich, CT:Information Age. Hymes, Dell, ed. 1999 [1969] Reinventing Anthropology. Ann Arbor: University of Michigan Press. LeCompte, Margaret, Jean J. Schensul, Margaret Weeks, and Merrill Singer 1999 Researcher Roles and Research Partnerships. Ethnographer's Toolkit, Vol. 6. Walnut Creek, CA: AltaMira Press.

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions

Hopson

Reinventing Evaluation

295

Mabry, Linda, ed. 1997 Evaluation and the Postmodern Dilemma. Advances in Program Evaluation, Vol. 3. Greenwich, CT: JAI Press. Mabry, Linda, Leslie Goodyear, Hazel Symonette, and Rodney Hopson N.d. Postmodern Evaluation: Reflections on Multiplicity. Unpublished MS. Mertens, Donna 2005 Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods. 2nd edition. Thousand Oaks, CA: Sage. Nader, Laura 1999 Up the Anthropologist-Perspectives Gained from Studying up. In Reinventing Anthropology. Dell Hymes, ed. Pp. 284-311. Ann Arbor: University of Michigan Press. Peters, Michael, ed. 1997 Education and the Postmodern Condition. Westport, CT: Bergin and Garvey. Reed, Michael C. 1997a Practicing Anthropology in a Postmodern World: Lessons and Insights from Federal Contract Research. National Association for the Practice of Anthropology Bulletin No. 17. Washington, DC: American Anthropological Association. 1997b Introduction. In Practicing Anthropology in a Postmodern World: Lessons and Insights from Federal Contract Research. National Association for the Practice of Anthropology Bulletin No. 17. Michael C. Reed, ed. Pp. 1-10. Washington, DC: American Anthropological Association. Rist, Ray C. 1980 Blitzkrieg Ethnography: On the Transformation of a Method into a Movement. Educational Researcher 9(2):8-10. Ryan, Katherine E., and Thomas A. Schwandt, eds. 2002 Exploring Evaluator Role and Identity. Greenwich, CT: Information Age. Stake, Robert E. 1991 Retrospective on "The Countenance of Educational Evaluation." In Evaluation and Education: At Quarter Century. Ninetieth Yearbook of the National Society for the Study of Education. Milbrey W. McLaughlin and Denis C. Phillips, eds. Pp. 67-88. Chicago: University of Chicago Press.

This content downloaded from 152.118.24.10 on Thu, 20 Mar 2014 03:26:35 AM All use subject to JSTOR Terms and Conditions