Review: Alternative Assessments I

 Describe the two epistemologies in ch. 3 (o/s)

 Compare the two principles for assigning value

(util/int-pl)

 Identify pros/cons of the two evaluation

approaches we discussed last week
Alternative Approaches to
Evaluation II
Dr. Suzan Ayers
Western Michigan University
(courtesy of Dr. Mary Schutten)
 Consumer-Oriented Approach
 Typically a summative evaluation approach

 This approach advocates consumer education

and independent reviews of products

 Scriven’s contributions based on groundswell of

federally funded educational programs in 1960s
 Differentiation between formative/summative eval.
 Consumer-Oriented Checklist
(Scriven, 1974, p. 102)
 Need
 Market
 Performance
 True field trials [tests in a “real” setting]
 True consumer tests [tests with real users]
 Critical comparisons [comparative data]
 Long term [effects over the long term]
 Side effects [unintended outcomes]
 Process [product use fits its descriptions]
 Causation [experimental study]
 Statistical significance [supports product effectiveness]
 Educational significance
 Cost effectiveness
 Extended support [in service training]
Producer’s efforts to meet these standards
improve product effectiveness
 Key Evaluation Checklist developed to evaluate
program evaluations
 Educational Products Information Exchange
(EPIE): Independent product-reviewer service
 Curriculum Materials Analysis System (CMAS)
checklist: Describe product, analyze rationale,
consider: antecedent conditions, content,
instructional theory & teaching strategies, form
overall judgments
 Uses of Consumer-Oriented
Evaluation Approach
 Typically used by gov’t. agencies and
consumer advocates (i.e., EPIE)

 What does one need to know about a product

before deciding whether to adopt or install it?
 Process information
 Content information
 Transportability information
 Effectiveness information
 Consumer-Oriented Pros/Cons
 Strengths: valuable info given to those who
don’t have time to study, advance consumers’
knowledge of appropriate criteria for selection
of programs/products

 Weaknesses: can increase product cost,

stringent testing may “crimp” creativity, local
initiative lessened b/c of dependency on
outside consumer services
 Consumer-Oriented Qs
 What educational products do you use?

 How are purchasing decisions made?

 What criteria seem to most important in the

selection process?

 What other criteria for selection does this

approach suggest to you?
 Expertise-Oriented Approach
 Depends primarily upon professional expertise
to judge an institution, program, product, or
activity

 This is the first view that relies heavily on

subjective expertise as the key evaluation tool

 Examples: doctoral exams, board reviews,

accreditation, reappointment/tenure reviews
etc…
 Expertise-Oriented Types
 Formal Review Systems (accreditation)
 Existing structure, standards exist, set review
schedule, experts, status usually affected by results
 Informal Review systems (grad S committee)
 Existing structure, no standards, infrequent
schedule, experts, status usually affected
 Ad hoc panel review (journal reviews)
 Multiple opinions, status sometimes affected
 Ad hoc individual review (consultant)
 Status sometimes affected
 Expertise-Oriented Pros/Cons
 Strengths: those well-versed make decisions,
standards are set, encourage improvement
through self-study

 Weaknesses: whose standards? (personal

bias), expertise credentials, can this approach
be used with issues of classroom life, texts, and
other evaluation objects or only with the bigger
institutional questions?
 Expertise-Oriented Qs
 What outsiders review your program or
organization?

 How expert are they in your program’s context,

process, and outcomes?

 What are characteristics of the most/least

helpful reviewers? (list brainstorms on board)
 Participant-Oriented Approach
 Heretofore, the human element was missing
from program evaluation

 This approach involves all relevant interests in

the evaluation

 This approach encourages support for

representation of marginalized, oppressed
and/or powerless parties
Participant-Oriented Characteristics
 Depend in inductive reasoning [observe, discover,
understand]

 Use multiple data sources [subjective, objective,

quant, qual]

 Do not follow a standard plan [process evolves as

participants gain experience in the activity]

 Record multiple rather than single realities [e.g.,

focus groups]
 Participant-Oriented Examples
 Stake’s Countenance Framework
 Description and judgment
 Responsive Evaluation
 Addressing stakeholders’ concerns/issues
 Case studies describe participants’ behaviors
 Naturalistic Evaluation
 Extensive observations, interviews, documents and
unobtrusive measures serve as both data and
reporting techniques
 Credibility vs. internal validity (x-checking, triangulation)
 Applicability vs. external validity (thick descriptions)
 Auditability vs. reliability (consistency of results)
 Confirmability vs. objectivity (neutrality of evaluation)
 Participatory Evaluation
 Collaboration between evaluators & key organiz-
ational personnel for practical problem solving
 Utilization-Focused Evaluation
 Base all decisions on how everything will affect use
 Empowerment Evaluation
 Advocates for societies’ disenfranchised, voiceless
minorities
 Advantages: training, facilitation, advocacy,
illumination, liberation
 Unclear how this approach is a unique participant-
oriented approach
 Argued in evaluation that it is not even ‘evaluation’
 Participant-Oriented Pros/Cons
 Strengths: emphasizes human element, gain
new insights and theories, flexibility, attention to
contextual variables, encourages multiple data
collection methods, provides rich, persuasive
information, establishes dialogue with and
empowers quiet, powerless stakeholders

 Weaknesses: too complex for practitioners

(more for theorists), political element,
subjective, “loose” evaluations, labor intensive
which limits number of cases studied, cost,
potential for evaluators to lose objectivity
 Participant-Oriented Qs
 What current program are you involved in that
could benefit from this type of evaluation?

 Who are the stakeholders?

 Alternative Approaches Summary
Five cautions about collective evaluation
conceptions presented so far
1) Writings in evaluation are not models/theories
 Evaluation is a transdiscipline (not yet a distinct discipline)

 “Theoretical” underpinnings in evaluation lack

important characteristics of most theories

 Information shared is: sets of categories, lists of

things to think about, descriptions, etc.
2) “Discipleship” to a single ‘model’ is dangerous
 Use of different approaches as heuristic tools, each
appropriate for the situation, recommended

3) Calls to consolidate evaluation approaches into

a single model are unwise
 These efforts based in attempts to simplify
evaluation
 Approaches are based on widely divergent
philosophical assumptions
 Development of a single omnibus model would
prematurely close a divergent phase in the field
 Just because we can does not mean we should;
would evaluation be enriched by synthesizing the
multitude of approaches into a few guidelines?
4) The choice of an evaluation approach is not
empirically based
 Single most important impediment to development
of more adequate theory and models in evaluation

5) Negative metaphors underlying some

approaches can cause negative side effects
 Metaphors shared in ch. 3 are predicated on
negative assumptions in two categories:
 Tacitly assume something is wrong in system
being evaluated (short-sighted indictment)
 Based on assumptions that people will lie, evade
Qs or withhold information as a matter of course
 Alternative Approaches’ Contributions
Approaches shared in ch. 4-8 influence evaluation
practices in important ways
 Help evaluators think diversely
 Present & provoke new ideas/techniques
 Serve as mental checklists of things to consider,
remember, or worry about
 Alternative approaches’ heuristic value is very high,
but their prescriptive value is less so
 Avoid mixing evaluation’s philosophically incompatible
‘oil/water’ approaches; eclectic use of alternative
approaches can be advantageous to high-quality
evaluation practices

Table 9.1
 Exercise
 Clearly identify your evaluand
 Is it a program, policy, product, service, other?
 Who does it (or should it) serve?
 Who is in charge of it?

 Find a partner and explain what you have written

 Does it make sense?
 Does it match what you wrote?
 Does it avoid specifying criteria?
 Is it simple enough?
 Did you avoid commenting on the merits of the
evaluand?