Gysen. 2006. The Modus Narrandi

Evaluation
http://evi.sagepub.com
The Modus Narrandi: A Methodology for Evaluating Effects of Environmental

Policy
Joos Gysen, Hans Bruyninckx and Kris Bachus
Evaluation 2006; 12; 95
DOI: 10.1177/1356389006064176
The online version of this article can be found at:
http://evi.sagepub.com/cgi/content/abstract/12/1/95
Published by:
http://www.sagepublications.com
On behalf of:
The Tavistock Institute
Additional services and information for Evaluation can be found at:

Email Alerts: http://evi.sagepub.com/cgi/alerts
Subscriptions: http://evi.sagepub.com/subscriptions
Reprints: http://www.sagepub.com/journalsReprints.nav
Permissions: http://www.sagepub.com/journalsPermissions.nav
Downloaded from http://evi.sagepub.com at UNIVERSITAT AUTNOMA DE BARCELONA on March 14, 2007

2006 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
Evaluation
Copyright 2006
SAGE Publications (London,
Thousand Oaks and New Delhi)
DOI: 10.1177/1356389006064176
Vol 12(1): 95118
The Modus Narrandi

A Methodology for Evaluating Effects of
Environmental Policy
JOOS GYSEN
Catholic University of Leuven, Belgium
HANS BRUYNINCKX
Catholic University of Leuven, Belgium and Wageningen University, The Netherlands
K R I S B AC H U S
Catholic University of Leuven, Belgium
Are we being effective? has become an increasingly central question in

policy evaluation research. This article defends the position that this
question can only be answered by employing a methodology that is adapted
and can stand scrutiny in the policy field in which the evaluation occurs. In
our experience the evaluation of environmental policy requires a renewed
effort to develop methodologies that take into account the key
characteristics of that policy area. The modus narrandi, based on causal
narrative story reconstruction, is our contribution to these efforts. It is a
methodology for effect evaluation that looks at effectiveness and side effects
within contexts characterized by causal uncertainty.
K E Y WO R D S : causality; effectiveness; environmental policy evaluation;
modus narrandi methodology; side effects
Introduction
This article aims to address some central issues in the evaluation of the effects of
environmental policy. Implementation and effectiveness of policy have recently
been subject to inquiry at different policy levels and in different policy fields. At
the European level, the European Environmental Agency (EEA) has been
particularly active in trying to answer the question are we being effective? (EEA,
2001) and aims to develop appropriate methodologies in order to do this. Others
including scholars and practitioners (e.g. administrations) have engaged in
answering the effectiveness question. Most initiatives have been designed around
95
Evaluation 12(1)
specific environmental subfields and concrete evaluation exercises, e.g. the evaluation of specific water treatment policies, air pollution abatement policies. In most
instances, a case study is at the core of the research and evaluation methodology.
This article aims to contribute to a number of approaches that provide
elements of more universally applicable methodologies to measure the effectiveness and side effects of environmental policy (Bressers and Klok, 1991;
Mickwitz, 2000). In addition, it presents effect evaluation, a methodology that
takes into account both effectiveness and side effects. Effect evaluation
combines both approaches into one evaluative statement. We emphasize some
conceptual and methodological issues within the field of environmental policy
evaluation, not only because of our personal interest in specific methodological
and conceptual challenges in this subfield, but also because environmental policy
evaluation is of increasing importance. Environmental issues have achieved a
sense of urgency and seriousness. This has been accompanied by an increase in
both public and political interest. In addition, the amount of financial and other
resources allocated to both environmental policy-making and environmental
policy evaluation has increased significantly. Yet environmental policy is a relatively young policy field. The first serious policy initiatives date from only a few
decades ago. Their evaluation and, more specifically, the link between policymaking and environmental outcomes have only relatively recently become
serious issues for evaluation scholars and professionals.
This article first addresses the central concepts of effectiveness and causality
and their role in environmental policy evaluation. Second, some of the key
characteristics of environmental policy and their influence on subsequent evaluative efforts are discussed. Third, a methodology is designed fitted to the reality
of environmental policy.
Conceptualizing Effects and Effectiveness

Our central goal is the development of a methodology that allows claims to be
made about the effectiveness of environmental policies. This is an inherently
normative question because it involves an appreciation of whether and to what
extent a policy works. In order to answer this type of evaluation question, at
least three different evaluative subquestions have to be taken into account.
(1) Questions can be descriptive. These questions try to address the what has
happened?. question.
(2) Questions can refer to causal linkages. Causal questions do not simply ask
what has happened? They try to link changes with the policy.
(3) Questions can be normative. These questions evolve around the central
question, Are the results satisfactory? According to the EEA, this
question can be answered from the perspectives of effectiveness, relevance,
efficiency or utility. We are especially interested in effectiveness. Although
we recognize the importance of relevance, efficiency and utility, the scope
of this article is limited to effectiveness, and more precisely, in relation to
environmental policy.
96
Gysen et al.: The Modus Narrandi

The effectiveness question cannot be separated from the two former types of
evaluation questions (the descriptive and causality questions), if it is to be
answered in a realistic and satisfactory way. In fact, the methodology proposed
strongly emphasizes the fact that effectiveness evaluation builds on welldocumented and developed answers to the descriptive and causal questions.
First, to further refine the conceptual framework, an important but often ignored
distinction is made between main effects (intended) and side effects (unintended).
Main Effects
The main effects are those intended by a policy/intervention. They were anticipated and wanted and can be divided into three categories: output effects,
outcome effects and impact effects.
The output is defined as the tangible results of a measure (EEA, 2001). Output
has a rather short-term dimension. Output becomes evident shortly after the
implementation process of the policy instrument or even during the process, e.g.
when a new type of permit becomes compulsory, the amount of permits granted
from the first day will be an output effect. These output effects are not necessarily part of the actual policy objectives. They do not necessarily have an automatic or direct relationship with policy performance. For example, the number
of environmental permits granted, or increasing the number of field inspectors,
can be policy objectives, but they are not a direct indication that the environment
is improving. They can only give an indirect indication of the intended policy
results. However, in practice, output effects are often the only effects that are
monitored.
The outcome of the policy, defined as the response of the target groups to the
output (EEA, 2001), corresponds in principle with the policy objectives. Where
the output effects usually occur in the short term, outcome effects are most likely
to occur in the mid to long term. A change in environmental infringements, for
example, is a possible outcome of the number of extra field inspectors. Other
examples of environmental policy outcome effects are reductions in emissions,
increased recycling rates or shifts in the use of different modes of transport.1
Any effect that influences the state of the policy issue is defined here as an
impact effect. Mickwitz (2000) would call this effect the ultimate outcome. We
choose not to use this term because of the significant distinction we make
between outcome and impact. The impact effects or impact (EEA, 2001) of a
policy are often only visible in the long term. In the example of environmental
policy, impact effects are related to actual changes in the state of the environment. Often these effects are expressed in terms of quality. Better air quality
could be a possible impact effect if the policy was aimed at reducing the unauthorized emission of air pollutants.
Side Effects
Side effects are defined here as the unintended effects of a policy or intervention.
The difference between side effects and main effects is based on the criterion of
intention. Intention was chosen because it was felt that other criteria fail in
97
Evaluation 12(1)
terms of clarity and consistency. Vedung (1997) uses target area as a distinct
criterion. In his reasoning all effects within the target area are main effects. Those
which fall outside of the target area of the policy are side effects. This conceptualization is not necessarily contrary to our idea of intention, as long as target area
is defined as the part of reality that the actors try to shape (Van de Graaf and
Hoppe, 2000).2 Following from this definition we argue that trying to shape
involves at least a passive intention. To avoid vagueness we suggest using intention as the distinguishing criterion. It is important here to stress the difference
between the criteria intention and anticipation (see Box 1). An effect can be
anticipated without being intended: therefore these concepts should not be used
interchangeably. Since intention is the defining feature, side effects can be anticipated or unanticipated. As will be demonstrated, whether they are anticipated
can make a significant difference in many ways, but they remain side effects.
Further distinctions are drawn here between direct, indirect and derived side
effects. First, direct side effects are visible and measurable first-instance effects.
These effects result directly from the policy, even though they are not intended.
For example, the Flemish Law on Soil Pollution and Sanitation (1994) had a
direct side effect, i.e. the establishment of an economic industry of professionals
in this field that did not previously exist. This led to the creation of at least 1000
jobs in the sector (Bachus and Bruyninckx, 1998). Second, indirect side effects
result from direct side effects. They are mostly visible and measurable. They do
not result directly from the policy but from another side effect. Drawing on the
same example, the creation of this economic sector has meant that additional
training and educational programmes are required and several of these now exist.
They are an indirect side effect. Third, derived side effects are difficult to separate
from indirect effects. The difference between the two is that a derived side effect
requires an extra effect or catalyst to occur. It doesnt automatically follow from
a direct or indirect effect. A good example of an extra effect is the multiplier
effect, a chain reaction of additional income and purchases that results in a final
increase in total purchases that is greater than the initial increase in purchases.3
For example, because of the new market for soil protection and sanitation, strategic choices within the environmental consulting and services sector can be
considered as derived effects.
Box 1. Example of a Side Effect of a Policy to Reduce Pollution
The Council Directive 91/676/EEC concerning the Protection of Waters against Pollution caused by Nitrates from Agricultural Sources was implemented in Flanders with a
strong emphasis on changes in the intensive pig-farming sector. The two main instruments
were a reduction in the number of pigs being farmed, and a collection of dung in specialized installations to prevent dumping in the environment. One side effect of this policy
i.e. unintended is a reduction in employment in the pig-farming sector. Clearly this
side effect was anticipated, demonstrated by the fact that measures are taken in the policy
to soften this negative side effect. Additionally it is clearly not a main effect because the
policy aims to reduce the level of nitrates not the employment rate.
98

Based on these conceptual distinctions, five types of effectiveness can be
identified: (1) institutional effectiveness, (2) target group effectiveness, (3) impact
effectiveness, (4) societal effectiveness and (5) side effectiveness. Table 1 provides
an overview.
Institutional Effectiveness
Institutional effectiveness is the match of the output of a given policy or program
to the output objectives of that policy or program. We have chosen to attribute
this particular term to this phenomenon because the term institutional clearly
incorporates the link to the performance of those institutions or organizations
that are expected to initiate policy actions. This can include government bureaucracies at different policy levels, compliance agencies, etc. It is the performance
Table 1.
Overview of Effects and Effectiveness
Effect
Definition of effect
Type of
Effectiveness
Definition of Effectiveness
Output E
The tangible results

of a measure
Institutional
The match of the output of

a given policy or programme
with the output objective
of that policy/programme
Outcome E
The response of the

target groups to the
output
Target group
The degree to which the

target group responds to
the policy, due to the policy,
as aimed for by that policy
Impact
The degree to which the

change of the policy issue,
caused by the policy, is in
line with the policy goals
Societal
The extent to which the

effect is in line with broader
societal objectives
Impact E
Direct SE
The influences on the

policy issue
Visible and measurable

first instance efects.
These effects follow
directly from the policy,
even though they are
not intended
Indirect SE
Does not directly follow

from the policy but from
another side effect
Derived SE
The result of the

interaction of another
SE and an external
factor
Side
The extent to which a policy

has produced unintended
effects which are in line with
related policy objectives and
public interest
99
Evaluation 12(1)
of the institutions or organizations that is the focus of the evaluation, not what
follows from their actions.
Target Group Effectiveness

Target group effectiveness means the degree to which the target group responds
to the policy, due to the policy and as intended by that policy (Gysen et al., 2002).
It clearly leaves the institutional boundaries of the policy, and instead focuses on
changes to actor behaviour. For example, an evaluation would evaluate changes
in emissions from a certain company, not changes in the water quality of the river
in which the emissions are deposited. It is important to acknowledge that target
group effectiveness measures outcome effects with reference to outcome objectives, not as proxies to impact effects.
Impact Effectiveness
A third type of effectiveness is impact effectiveness. This is defined as the degree
to which the change of the policy issue, caused by the policy, is in line with the
policy goals (Gysen et al., 2002). Bruyninckx and Cioppa (2000) define this effectiveness (based on applying it to international environmental regimes) as an
environmentally effective institution eradicates or alleviates anthropogenic
deductions from and/or deposits to an ecological system or systems in balance
with the systems natural regenerative processes.
This definition provides a basis for evaluation, independent from the immanent
objectives of the institution or policy. However, we do not go as far as Bruyninckx and Cioppa. In our view the intended impact or environmental objectives as formulated in the policy remains central to setting standards for the
evaluation. Impact effectiveness is not goal-free. For example, environmental
quality cannot be judged apart from specific policy goals on environmental
quality. It is, to the contrary, a strictly goal-based evaluation.
Societal Effectiveness
Within the EEA framework (EEA, 2001) societal effectiveness corresponds with
both the relevance and utility questions. Societal effectiveness addresses the
question whether or not the impact (or impact effects) satisfies societal needs: Is
the effect a contribution to broader societal objectives? This is the point where
we switch from our dominant goal-based design to a goal-free design (Scriven,
1991).4 Societal effectiveness is not measured by the direct objectives of the
policy. It is goal-free in the sense that the evaluator will define goals on which
the policy will be judged in terms of societal effectiveness. The evaluator will have
to choose the goals and the set of criteria based on a set of widely accepted
societal principles and goals. Sustainable development is such a concept and
national sustainable development strategies, which exist in almost every country
nowadays, are an expression of such broader principles and goals. This type of
effectiveness measurement necessarily broadens the scope of evaluation. It
allows the evaluator to look further than what may be somewhat narrow policy
goals. However, it is not a substitute for one of the other effectiveness evaluations
based on clear policy objectives.
100

These types of effectiveness (institutional, target group, impact and societal)
all remain part of what we have called main types of effectiveness. The common
denominator is that they are all based on main i.e. intended effects of a
particular policy.
Side Effectiveness
Side effectiveness is the fifth type of effectiveness within an integral approach to
effect evaluation. Side effectiveness is partly based on a goal-free approach. The
criteria of merit and the standards of performance are based on the goals and
objectives of the stakeholders and the objectives of other, related policy fields.
For example, if the focus is on employment side effects, employment policies will
be analysed to find relevant policy objectives. Side effectiveness expresses the
extent to which a policy has produced unintended effects which are, or are not,
in line with related policy objectives and stakeholder interests. A set of merit
criteria and standards of performance are constructed and the side effects are
measured against these. These effects can be direct, indirect or derived effects.
Contrary to our approach on main effects and effectiveness, a further distinction
is not drawn between various types of side effectiveness. Essentially, the distinction between direct, indirect and derived side effects is only valuable for bridging
the causality gap between policy and effect. In the case of main effects the distinction is important for the same reason. In addition it is essential to be able to
distinguish between the various types of effectiveness because of possible
confusion of users of policy evaluation results, for example, to avoid a situation
where findings of institutional effectiveness are mistaken for results of impact
effectiveness. Obviously, the findings of these two do not have the same status.
In the case of side effects, by definition they are not linked with policy objectives,
and hence, further distinctions based on stated policy goals are irrelevant.
The result will be a statement on the side effects of a policy, which adds value
to the more mainstream effectiveness evaluation, because it is broader than only
looking at the policy goals. Yet, it must be emphasized that the evaluation of side
effects is also valuable if carried out by itself, as for example with environmental
and social impact assessments.
Particular Difficulties Associated with the Evaluation of

Environmental Policies
Evaluating the effectiveness of any field of policy-making cannot be disconnected
from the typical features of the issue at stake and the characteristics of the policy
field. The following paragraphs elaborate this argument in relation to environmental policies and their effects.
Characteristics of Environmental Phenomena

Environmental phenomena are characterized by some special features, which
taken together distinguish them to a certain extent from purely social phenomena. The term environmental phenomena is used in this article to include both
natural phenomena, with no direct human intervention, and socialenvironmental
101
Evaluation 12(1)
interactions. First these characteristics will be listed and grouped. Subsequently
the consequences of these features for environmental policy-making and evaluation are discussed.
Complexity Environmental phenomena combine complex patterns of impact
and causation. Often there are serious knowledge deficits, uncertainty and a lack
of data and to a great extent the phenomena rely on expert knowledge and scientific rationality.
(a) Complex patterns of impact and causation. The diffuse character of
environmental issues adds to the difficulty in establishing clear-cut causal
relationships. A typical example is air quality. It is influenced by many factors.
Even if we would focus on CO2 emissions, they are still manifold, including traffic,
household consumption, industrial emissions, power plants. This makes any
evaluation based on the links between policy objectives and effects inherently
difficult.5 In addition, environmental phenomena typically cross sovereign and
administrative boundaries. Events in one place can lead to effects elsewhere. A
well-known example is the acid rain in Scandinavia that is partly caused by industrial emissions in Central and Western Europe. Phenomena also tend not to be
limited by policy-makers time-frames. Lafferty and Meadowcroft (1996) refer to
the combination of these characteristics as the time-scale effect. As todays
events can cause effects many years from now, this adds to the difficulty in determining clear-cut causal relationships.
(b) Knowledge deficit. Lafferty and Meadowcroft (1996) consider knowledge
deficits in this field as closely related to the uncertainty, complexity and technical difficulty of the environmental phenomena as they are very difficult to
comprehend and therefore knowledge about them is lacking. The complex interaction of physical, social and biological factors requires advanced scientific
refinement. Global warming is a good example. We still lack consensus on its
effects and, without consensus, uncertainty dominates and complicates policy
discussions.
(c) Lack of data. Even if the detailed physical, social and biological relationships and interactions were understood, an important problem would still remain:
monitoring and gathering data. Because of other features related to complexity,
such as technical complexity and diffuse sources, monitoring and data gathering
can be very difficult and therefore very expensive. Next to these physical
obstacles, environmental policy is a relatively young policy field which means that
good longitudinal data are often scarce.6
(d) Scientific rationality. Environmental phenomena depend heavily on
science to be explained. Most such phenomena cannot be understood by nonexperts. It is clear that because of the complexity of the environmental phenomena that are increasingly important, such as global warming, biodiversity or
genetic modification, the reliance on science has intensified.7
Irreversibility and thresholds Irreversibility is defined here as when recovery,
whether natural or man-made, is no longer feasible even within the longest known
time-frame. Irreversibility also means that there is a sharp and abrupt cut-off
102

point beyond which effective environmental policy-making is impossible.
Examples of irreversibility include degenerated topsoil, for instance, in the desert
area of Africa; species becoming extinct; coral reefs dying.
Closely linked to irreversibility is the existence of thresholds. Thresholds refer
to certain points at which processes or situations change dramatically. This
change is often preceded by a period marked by the absence of any clearly visible
effect. The more a threshold is passed, the more difficult it becomes in general
to reverse the process. Passing the final threshold leads to irreversibility.
The environment as a public good The natural environment is in principle
an example of a public good. Public goods and externalities both pose specific
difficulties for policy-making and for evaluation. For example, the costs of maintaining public goods are too high for any individual and nobody can be excluded
from the advantages (Mueller, 1979).
Closely related is the concept of externalities. An externality occurs when an
individual of a group imposes high costs to society (STOHO, 1996). Weale (1992),
while discussing pollution, observed that the distribution of costs and benefits
that are typically present when actors behave and organize around public goods
and cause externalities will not necessarily be equal. Some will suffer more from
some externalities while others will benefit more from the effects of the public
good.8
Both the character of public goods and the distributional aspects of externalities further complicate environmental issues as policy objects. Figure 1 summarizes the different features of environmental phenomena.
Influence on Environmental Policy-Making and Evaluation

Environmental policy is partly determined by the features of the socioenvironmental phenomenon it aims to deal with. The more these characteristics are
evident in the subject of the policy, the more difficult it is to formulate and
Complexity
Complex patterns of impact and causation

Lack of data
Knowledge deficit
Science rationality
Irreversibility and threshold

The environment as a public good
Figure 1. Features of Environmental Phenomena
103
Evaluation 12(1)
implement good policy. The same reasoning can be applied to the evaluation of
environmental policies.
Policy development and implementation is a process of searching for acceptable compromise and feasibility given specific contexts. This is based on social,
political and economic dynamics more than on the purely objective scientific
features of the problem at hand. As long as parties involved do not agree on the
nature and the effects of a certain natural phenomenon, controversy will persist.
This means that, in addition to the characteristics described above, a process of
social mediation determines the specific characteristics of the policy outcome.
The results of this process are often below the expectations of the rational,
scientific advice formulated regarding the policy. This is important to bear in
mind when evaluating the effectiveness of any given environmental policy.
Uncertainty, for example, is (in principle) an enemy of the policy-maker (and
the evaluator for that matter). Uncertainty brings guestimation and risk taking
to the policy arena. The uncertainty in policy-making is a function of the uncertainty in science. Limited knowledge influences the possibilities of the policymaker. Sometimes the policy-maker will have no other option than to make
policy on a subject that is not fully understood. In such a case the policy has blind
spots. The problem of lacking data aggravates the uncertainty problem. Incorrect
policy theories can be the result of policy formulated on the basis of incomplete
information. This problem finds its way into evaluation: without data, evaluations
are typified by guesses and estimates.
Thresholds are also subject to different interpretations within the political
arena. Whereas scientists regard them as a matter of determining non-linear
processes and their mostly environmental consequences, thresholds are less
definite and are subject to social debate and construction in policy processes.
Other factors such as the cost of the process, social considerations like the
variable effects of a threshold in different locations or distributional aspects
come into play. This will result in political thresholds that can also be the subject
of an evaluation.
The previous two points also have an effect on two of the crucial political
elements of policy-making. Environmental policies have to be responsive to
concerns about accountability and liability. An event today can result in effects
many years from now, and in another place. If the other place is in another
country, national legislation is no longer sufficient. Supranational structures are
required to deal with this challenge but these structures (or regimes) are characterized by great complexity and inertia (Keohane et al., 1993; Kutting, 1998;
Levy and Young, 1995; Peterson, 1997a, 1997b; Young, 1997).
Policy-making and evaluation are also affected by time-lag effects. Policymakers and evaluators are sometimes confronted with a policy the effects of
which are not all noticeable yet. Second, as Weale (1992) stated, future generations are not well represented in relation to contemporary decisions. This results
in problems with the value system on which the evaluation criteria are based, as
well as problems of uncertainty.
To conclude, the inherent or specific characteristics of environmental problems
are accentuated and take on additional meaning in a policy-making process. This
104

process is bounded by additional limits because of social and political bargaining. The result is often a policy that is only indirectly linked to presumed ecological objectives. These elements put stress on the evaluation of environmental
policies. The next section presents a methodology that aims to answer relevant
evaluation questions in these kinds of environmental policy setting.
Introducing the Modus Narrandi as an Effect Evaluation

Method for Environmental Policy
The modus narrandi, as we have named this methodology, responds to two
fundamental evaluative needs: 1) assessing the legitimacy of a policy (the legitimacy argument) and 2) addressing the features of environmental policy when
aiming to prove the causal relationship between the policy and effects (the
causality argument).
Before exploring these arguments more deeply it is necessary to clarify what
is meant by effect evaluation: an effect evaluation brings together the findings
of an effectiveness evaluation and a side effects evaluation.
Assessing Legitimacy
Effectiveness evaluation aims to evaluate the intended effects of a policy intervention. These effects are usually evaluated in light of the goals of the intervention. Four types of effectiveness evaluation have been distinguished: output
effectiveness, target group effectiveness, impact effectiveness and societal effectiveness. These all have in common that the criteria used to make evaluative
statements are the goals or objectives of the intervention. Therefore there is no
need, within an effectiveness evaluation, to weight these effects in relation to
other elements. Scriven (1991) refers to these as autonomous criteria as opposed
to compensatory criteria. The former stand alone, while the latter are traded off:
the performance of one evaluand is compared or traded off against the performance of another evaluand.
In addition to these types of effectiveness, a fifth type of effectiveness was
defined: side effectiveness. Intuitively one might think that the criterion used in
side effect evaluation is more of a compensatory criterion, as Scriven suggests,
but this is not the case. Side effects are evaluated using criteria derived from the
context of the evaluand, without trade-offs. This methodology uses goals and
objectives of other related policy fields and also stakeholders to assess side
effects. Each of these is considered, in the first instance, as a stand-alone criterion.
The study of all effects is important because we strongly believe that the legitimacy of an intervention is based on its effectiveness and on its side effects, but
most of all, on the synthesis of the assessments of both effectiveness and side
effects. This synthesis is more than the mere combination of the findings of both
evaluations. It also comprises trading off the effects and the side effects. This
synthesis is the result of bringing the effects and side effects together into a
compensatory evaluative frame.9
The best way to assess the legitimacy of a policy is by carrying out a comprehensive assessment of all effects. An effectiveness evaluation runs the risk of
105
Evaluation 12(1)
missing important side effects that could be so detrimental that a highly effective intervention does more harm than good or is perceived as such. Examples
include employment or economic side effects of environmental policies, or effects
on social inequality. A side effect evaluation, on the other hand, ignores the main
aim or intended effects of the intervention. Indeed, positive side effects might
even be achieved more efficiently via interventions in other policy fields or via
other instruments in the same policy field. Either way, both an effectiveness
evaluation and a side effect evaluation, conducted independently of one another,
offer only a partial view on the intervention and therefore contribute only
partially to an understanding of the impacts and the legitimacy of an intervention. The best argument for legitimizing the intervention can therefore be found
in a combined, compensatory assessment. Evidence for the importance of a
synthesis approach is provided by authors within the impact assessment school
(Barrow, 1997; Burdge, 1994). Both social and environmental impact assessments
aim to involve effects other than the intended effects on policy decision. The
methodological aspect of this will be discussed under the heading The Steps of
the Modus Narrandi.
Proving Causality
The key challenge for effect evaluation is proving the causal relationship
between the effect and the policy. In theory, an experimental design offers the
best solution to this challenge. However, the practice of policy-making in general,
and environmental policy-making in particular, is very different from the clinical
approach assumed in experiments. In addition, the characteristics of environmental phenomena and their impact on environmental policy-making and evaluation, are significant and create a difficult set of challenges for effect evaluation.
The essence of this challenge lies to a great extent in the issue of causality.
Whether side effects or effectiveness are studied, the problem of causality has to
be tackled by linking interventions to (side) effects in order to make causal
claims, rather than claims about observed correlations. This approach opposes
the Humean model of constant conjunction. Mahoney (2004) discusses this
approach as follows:
According to Hume (1748/1788), we infer causation when we repeatedly observe
putative causes followed effects; that is, two events are constantly conjoined in our
experience. However, we can never actually confirm the existence of causation because
the imagined necessary connections linking the two events cannot themselves be
observed or known. In contrast to this Humean model, scholars concerned with causal
mechanisms argue that exploration of the black box connecting independent and
dependent variables is essential to good causal research. On this view, we are not satisfied with merely establishing systematic covariation between variables or events; a
satisfactory explanation requires that we are also able to specify the social cogs and
wheels that have brought the relationship into existence.
Mahoney addresses the problem of the black box within the framework of
social science. He claims that, unless causal mechanisms are proposed, the
problem of the black box will not be resolved. Causal mechanisms are empirically underspecified properties of particular causal agents. Related to causality
106

in evaluations, this means that, in order to explain the causal links between an
intervention and an effect, one must investigate the black box that will contain
explanations as to why the effect exists or what influenced its observable features.
Uncovering the black box provides insights that will allow us to move from correlation relations to causal relations. In our experience, opening one black box will
often, certainly in complex settings, lead to a new black box. Uncovering these
layers of black boxes is fundamental for a more solid understanding of the causality behind the correlation. Causal proof lies in uncovering the causal mechanisms. Causality, in this approach, can be seen as the sum of two elements: one
is causal proximity (Homer-Dixon, 1995), the other is causal contribution. Causal
proximity expresses both the strength of the link between intervention and effect
and variable and hence, second, the validity of the established relationship.
Strong proximity signifies a direct link between intervention and effect. Weak
proximity means that several other factors in a causal chain link the intervention
to the effect. Causal contribution (Gysen and Bachus, 2003) indicates the importance of the correlation (or in the case of strong proximity, causality) of one
variable as opposed to the other variables in a multi-correlation relationship with
an event. This is an addition to Homer-Dixons causal proximity. This refinement is necessary in order to fully grasp the relationships and especially to frame
the causal relationship within a context characterized by many influencing factors
and multiple black boxes.
This approach to causality will be central to the methodology for effect evaluation. Two groups of evaluation methodologies are commonly used in the literature, each based on a different conceptualization of causality. The first group,
referred to here as the rationalistic methods, is based on a causality concept that
leans in the direction of the Popperian falsification approach (Popper, 1968).
Policy-making is evaluated as a social activity based on certain theoretical
assumptions that can be falsified by setting up policy experiments or by searching for failed or falsified policies. The methodological translation of this
approach is found in evaluations that are based on purely quantitative
approaches, modelling and on experiments. It is clear to us that, in terms of the
social sciences in general and evaluation methodologies for environmental policy
in particular, a rationalistic or Popperian view is untenable. First, propositions
underlying policy-making are generally not falsifiable. Second, for the overwhelming majority of social theory and policy-making, finding the case that falsifies the theory or the policy case that doesnt work, is rather easy as universal
social scientific laws are almost non-existent.
There is a set of alternative methodologies that has the potential to get
around this problem. These could form the foundation for a more realistic and
workable environmental policy evaluation methodology. This second group of
methodologies is referred to here as the hermeneutic methods. First, one could
base the evaluation methodology on the causal claims embedded in a variety
of grand policy theory. However, this option is confronted with the problem of
a lack of an accepted general theory: no single grand theory can explain all
causal links in social science, and hence in policy contexts. A second option is
working in an inductive way. The researcher starts with observation of reality
107
Evaluation 12(1)
and uses observation to explore the issue, generate hypotheses about the policy
process, and the possible effects. The third option focuses on middle-level policy
theories. With this option a shift towards a lower level of abstraction (middlelevel) occurs and the policy is approached from a more context-specific point of
view. These middle-level theories can address the general characteristics of political or bureaucratic policy-making, theoretical insights regarding the environmental policy-making dynamics, or other more specific and partial approaches to
the policy-making context. The fourth and last option implies a change in
thinking about the causality question. This option is to be used where no direct
link or relationship can be established or tested. The goal is to reconstruct the
policy process with regard to the effects based on a probable causal explanatory
modus. It uses middle-level theories, inductive claims and where possible quantitative data or numbers to explain and illustrate a causal story. By doing this
in a thorough and comprehensive way, the burden of proof shifts from the evaluator to his critics. Others will have to demonstrate that the causal story of how
a policy is (or is not) linked to certain outputs, outcomes or impacts is incoherent, incomplete or false by demonstrating a more probable causal narrative.
Figure 2 illustrates the two frameworks presented here.
Boskma and Herweijer (1988) argued that qualitative case-study designs
focused on causal analyses can be fitted into the rationalist explanatory model,
i.e. hermeneutic methods can lead to results similar to those of rationalist results.
Conceptualization of Causality
Methods
Prove
Hard theory
cf. Falsifaction (Popper)
Rationalistic methods
Focused on proving causality
through experimental research
designs
Causal narrative story

Deductive analysis
Inductive analysis
Midlevel analysis
Soft theory
Focused on understanding
causality through qualitative
research
Hermeneutic methods
Understanding
Conceptualization of Causality
Methods
Figure 2. ConceptualizationMethod Relationships
108

Robert Yin (2003) adds to this discussion that if a chain of evidence is established, the use of case studies can lead to general explanations. The principle is
to allow an external observer to follow the derivation of any evidence (Yin, 2003).
Even the methods that are traditionally linked to explaining can support claims
about effectiveness.
The EEA (2001) has followed similar reasoning in an attempt to design a path
of choice based on evaluative situations and various methodological options.
First, where causal links are few and predictable, standard models may be derived
by examining a small number of case studies. Second, where the application of a
measure is differentiated in space or by target sector, comparative case studies
can be used to help identify causal relationships. Third, where the target group
is relatively small, in-depth interviews can be used. Fourth (and this will be our
methodological starting point), when the link between a policy and its impact on
the environment is too diffuse or extended and therefore the effectiveness of the
impact effects not directly measurable, the focus of the evaluation can shift to
immediate outcomes and outputs as a rough proxy for the impact effects. All
these methods largely adhere to the hermeneutic group of methods, but we would
argue they should be acceptable as methods for identifying a causal relationship.
In an effort to tackle these problems, we suggest a methodology based on the
distinctions between the different effects. Because of the difficulty that lies in
establishing the effectiveness of a policy, we have chosen to focus on causal argumentation and assessment, rather than focusing on establishing hard proof. We
refer to this approach as the modus narrandi10 because this term emphasizes both
the method (modus and the grammatical construction which implies the way it
should be told) and the causal story (narrare). It is based on a careful exploration
of the policy and its various components (e.g. policy process, policy implementation, organization, actors, means). These policy and contextual elements will help
us to gain an understanding of the causal relationships and this understanding will
be the basis of a reasonable explanation of the effectiveness. Finally, a reasonable
causal chain which is contextually constructed (Homer-Dixon, 1995) will be the
result. The chain brings together the various elements which make up the path
from policy through context elements to output, outcome and impact effects.
The Steps of the Modus Narrandi

The point of departure for the application of the modus narrandi is the concept
of formal evaluation as introduced by Dunn (1994) and further elaborated by
Bressers (1998). Formal evaluation distinguishes four phases: explaining the
policy theory, estimating the goal attainment, establishing the effectiveness and
explaining the effectiveness. Based on the previous sections, we make a number
of amendments. First, we rename the first step reconstruction of the effects
spectrum. This step consists of the reconstruction of the policy theory supplemented with the reconstruction of any blind spots. Next, we change estimating
the goal attainment to valuing of effects. Finally, we combine the last two steps
and call this establishing causality. In this step we make use of the two measurements, causal proximity and causal contribution. Figure 3 illustrates the different
steps, which are then elaborated.
109
Evaluation 12(1)
Reconstruction
of the effects
spectrum
Establishing
the goals
Establishing the
underlying principles
to achieve the goal
Reconstruction
of the causal and
impact model
Judging the merit

of the policy theory
Valuing the
effects
Reconstruction
of the causal
narrative story
Empirical data
Triangulation
Establishing
causality
Policy theory
Illustrate causal
links
Causal
proximity
Causal
contribution
Effect
assessment
Figure 3. The Modus Narrandi
Step 1: Reconstruction of the effects spectrum Step 1 contains two phases.

Phase one is the reconstruction of the policy theory which Blommestein et al.
(1984) define as all the assumptions that form the basis of the policy,11 i.e. the
policy-makers interpretation of reality. Van de Graaf and Hoppe add to this the
notion of the arguments and elements of knowledge on which the policy theory
is based. They summarize the concept as follows:
The policy theory indicates why, in the judgment of policymakers, in a specific policy
field they have acted in a specific way. The policy theory is a reflection of the rationality of the policy maker, behind the policy. (2000: 70)
This rationale is at the core of the choice of policy instruments and other
elements of implementation. When reconstructing the policy theory, the aim is
110

to identify the relationships that make up the fundamentals of this rationality. In
Anglo-Saxon literature this approach is called policy theory based evaluation
(e.g. Posavec and Carey, 2003; Rossi et al., 2004). This first step provides an insight
into the main intended effects and the anticipated side effects.12 Logically it does
not cover the unanticipated side effects.
Therefore phase two is necessary: the reconstruction of the blind spot. This
blind spot refers directly to the policy theory and hence to the assumptions of
the policy-maker. It is the part of the policy that is not covered by the policy
theory and therefore leaves room for the existence of unanticipated side effects.
Van de Graaf and Hoppe (2000) suggest reconstructing (the policy theory) by a
close reading of policy text(s). We suggest supplementing this with several qualitative interviews with those involved in writing the policy text(s) and those with
relevant high-level knowledge about the choices that were made. These interviews can add important information, in addition to what are sometimes rather
concise policy text(s), especially on implicit policy goals and assumptions. Using
a combination of methods or at least checking information based on different
sources (triangulation) is of great value when constructing an effect spectrum.13
At this point in the process, a draft causal story can be designed by the evaluator in terms of the expectations embedded in the policy. However, it must be
emphasized that at this stage it is a draft causal story only.
Step 2: Valuing the effects The second step in formal evaluation is measuring
the effects and attributing a value to them. At this stage, separate approaches are
required for main and side effects.
(a) Main effects. The aim is to find out to what extent the policy goals have
been met. Therefore measurements14 of the effect are required that allow their
comparison with stated policy goals. After the comparison, a certain value can
be attributed to the effect in terms of goal attainment. We emphasize the importance of making a clear distinction between goal attainment valued at this stage
and effectiveness, which will be judged at the end of the evaluation. In contrast
to what other scholars such as Scriven (1991) and Mickwitz (2003) claim, we
believe it is important to draw a distinction between those two concepts.15 To
emphasize this difference we explicitly use the term goal attainment as opposed
to the more widely used and differently interpreted goal achievement. In our
view, goal attainment as such does not say anything about the causal relationship
between intervention and effects. It looks only at the effects and compares them
to the policy goals, which is exactly what we intend to do at this stage. For
instance, if a policy on water quality aims at a concentration x1 of a substance X
in the water then the concentration level of X (x2) is compared to x1. If the actual
state does not meet the objective then goal attainment is negative but if the
objective is met then goal attainment is positive. So this phase of the evaluation
only measures the effects and compares them to the policy objectives.
(b) Side effects. The methodology for side effects is more complicated. The
initial step is to define the set of side effects that will be included in the evaluation. For anticipated side effects we can use the same method as for main effects.
For unanticipated side effects we need to use several other methods, including:
111
Evaluation 12(1)
analysis of environmental policy literature on a specific issue; results of previous
evaluations; ex ante estimations of possible costs and benefits; and finally, and
crucial in our opinion, expert interviews with a variety of actors, who have been
close to implementation of the policy. After a short list of side effects has been
created, they need to measured. This is no different from measuring main effects.
The main difference lies in the valuing of side effects. In practice the two
approaches will be used simultaneously. First, the policy objectives of the relevant
policy fields to which the side effects could potentially belong will be looked at.
These are the policy fields in which policy objectives with regard to these effects
are formulated. For instance, if the side effect is a given concentration of particles of nitrogen oxide (NOx) then the air quality policy objectives for NOx may
be investigated. The second approach places the stakeholder perspective
centrally in order to (1) reflect the public interest and (2) emphasize the central
position of those who are unintentionally affected by the policy. It implies a focus
on those who are affected by the policy, while using their perspective as a reference point for valuing the effect. The challenge is to move beyond the
positivenegative dichotomy and look for gradations for the valuation of side
effects. For example, if employment in the pig-farming sector has decreased
because of nitrate reduction policies, this can be considered as a negative side
effect. However, this becomes interesting information only if we can place it
within the further policy objectives of the agricultural sector as a whole, or
possibly in light of stated employment policy goals. It also depends on the situation of those who have lost their jobs. In other words, valuing side effects is a
difficult task which requires clear distinctions that are known to (and accepted
by) all those involved in the exercise.
The reconstruction of the effect spectrum provides us with a list of effects.
Measuring the effects is important in terms of attributing influencing factors to
the effects. Policy evaluators mostly have to rely on existing datasets about the
different types of effects in order to measure them. Gathering extra or new data
normally falls outside the policy evaluators tasks. We visualize goal attainment
by using indicators like initial situation, actual measurement and distance to
target. We do not want to limit our appraisal to binary outcomes such as goal
attainment yes or no.
Step 3: Establishing causality Step 3 is to establish and explain the causal
relationships between the policy and the effects. We draw no distinction between
the two (establish/explain), because the aim is to establish the causality through
a methodology that is based on uncovering and explaining causeeffect relationships. It is at this stage of the methodology that we need to offer a solution to
counter the inherent features of environmental policy evaluation. Assessing
causality and related concepts relevant for evaluation purposes have already been
discussed. This third step in the methodology bridges the gap between the difficult and complex causal setting of environmental policy evaluation and the ambition to make strong and causal evaluative statements.
This third step is also necessary to link the measured effects (the result of the
goal attainment analysis) to the policy. Only by doing this can we distinguish the
112

effects of the policy from the potential effects of other explanatory factors. Only
the effects caused by the policy or intervention can be labelled as policy effects,
whether they are main or side effects.
The first two steps provide the material on which the causal chains can be
constructed. Crucially important for evaluation purposes are the linkages
between the elements of the chain. These linkages are the expression of causality. Causal proximity and causal contribution are used to refine these expressions
to the level where scientific reliability and real life utility and credibility meet.
At this point the meaning or value of the elements that make up the links in the
chain should to be clear.
The core activity at this stage is to build a causal narrative story. This story is
presented as a chain of events/elements. Using causal contribution and causal
proximity, the real life linkages between the elements are reconstructed. This
should be done using as many information sources as possible. Data from experiments are preferred, but given the characteristics of environmental policy this is
likely to be an ideal rather than reality. We suggest using surveys, focusing on
both qualitative and quantitative data, expert interviews and document analyses.
These sources should provide correlations or confirmations on which to build
using qualitative data. The crux of the methodology is that all these data are
organized in a structure (the chain) that offers a clear view of the causal relationships between the first (policy) and the last elements (effect) in the chain. The
aim is not to establish statistically significant results, but to explain why and how
certain effects are present or not. If the causal story line is carefully constructed,
it will provide far more information about why, how and with what sorts of
expectations and results policies have been made and therefore how they can be
improved.
Effect assessment a combined appraisal of the effects Once the causal narrative stories have been constructed in terms of both effectiveness and side effects,
these separate stories need to be combined into one effects evaluation statement.
This statement brings together the results of the effectiveness evaluation, whether
it was an institutional, target group or impact effectiveness evaluation, and the
outcome of the side effects evaluation. This exercise should be done in the most
open and transparent way. At this stage, the evaluator has to weight the different
elements that is results of the evaluation exercise. The weighting of the different elements is the core of the effect evaluations final statement and can be
approached from a descriptive or a prescriptive angle. Although we are generally
not very keen on using the prescriptive approach, we believe that in this situation
only a prescriptive statement can provide additional value. A descriptive approach
at this point in the process will be limited to an echo of the outcome of step 3. In
other words, the evaluator has to make some kind of bad/good/better/ best statement at this stage. The problem with the prescriptive approach namely what
yardstick to use to make such claims can be partially solved by a high level of
transparency. This transparency invites critics to scrutinize this statement. As long
as a yardstick has been explicitly applied, it should be made clear that judgement
statements are the view of the evaluator.
113
Evaluation 12(1)
The core of the statement lies in valuing the effects in a compensatory way.
The outcome of the effectiveness and side effects evaluations will be compared
with each other and a value statement will be made on the whole, which considers
various positive and negative effects. Only by balancing out the intended and
unintended effects can one make a statement on the legitimacy of the given
intervention.
This modus narrandi provides a statement on effectiveness, on side effects and
on the overall impact of the intervention. In practice all identified (side) effects
are listed and fed into a valuation matrix. The compensatory valuation uses the
elements of the matrix and builds a comprehensive story. This story is the evaluative effect statement based on which the legitimacy of the policy is supported
or criticized and also its effects both intended and unintended are explained.
Conclusion
Policy evaluators are confronted with a dilemma. First they acknowledge the
need for good and solid evaluations, preferably based on hard proof. However,
they are also aware of the context in which they try to accomplish this objective.
In environmental policy this context is inherently not hard-proof friendly. Next
to this methodological discussion, evaluations are often used to make statements
on the legitimacy of a policy. This article claimed that a side effect evaluation
and/or an effectiveness evaluation, conducted independently of each other, do
not provide a sufficient basis upon which to draw conclusions about the legitimacy of a policy. The legitimacy of a policy can only be found in the synthesis of
the two.
For effect evaluation the article proposed a methodology that takes into
account the specific conditions and characteristics of environmental policy,
without compromising on the expectations attached to causality-based evaluations. In fact, the modus narrandi is a methodology that builds explicitly on a
causal narrative story that looks into the relationships between the various
components of a policy. To do this, it makes a distinction between main (output,
outcome and impact) effects and side (direct, indirect and derived) effects.
Furthermore it looks into the different components of the policy and the immediate context. These elements are the fundaments of a causal chain. The connection between the elements of the chain is indicated through two measurements:
causal proximity and causal contribution. These connections are the heart of the
answer on causality.
In sum, we hope that this methodology opens the door for effect evaluations
in policy fields such as environmental policy where context and inherent characteristics have been important obstacles. By levelling the playing field for effect
evaluations we hope to make a modest contribution to better evaluations and
hence better policies.
114
Notes
1. European Commission, Towards a methodology for evaluating the effects of
measures taken to implement EU environmental legislation, first draft.
2. The original, Dutch definition of beleidsveld is: het beleidsveld is dit deel van de
werkelijkheid dat de actoren naar hun hand proberen te zetten.
3. Thomson elearning: http://lms.thomsonelearning.com/hbcp/glossary/glossary.taf?gid=
39&start=m
4. The importance of his plea is underlined here but we are also sensitive to Pattons
(1997) remark as to the dangers of this approach. Patton warns about the risk of
unuseful evaluation findings when goal-free evaluation is used. In Pattons view
Scriven leaves out the evaluation needs of the local programme staff. Additionally,
Patton claims that goal free evaluation carries the danger of substituting the evaluators goals for those of the project (1997).
5. In most cases in the literature (e.g. Weale, 1992) the diffuse character is referred to
within the framework of pollution, but it also holds for non-pollution related fields of
environmental policy-making such as biodiversity or soil degradation.
6. For example, in Flanders we have only had a qualitative and quantitative development environmental policy since the 1980s and this in turn affects the state of the art
of monitoring. Lack of data still tends to be a serious problem, not only in Flanders
but in general. Indicator exercises such as the Environmental sustainability index
(ESI, Yale and Columbia) encounter major data problems. These affect the quality of
the exercise.
7. We rely on scientists not only to explain these phenomena but also to discover and
name them. If it wasnt for scientists the hole in the ozone layer would never have
been discovered (Hannigan, 1995: 236).
8. Mining natural resources in third-world countries is a good example. In most cases
the cost of the mining is borne by local people. Not only environmental (e.g. a
damaged environment) but also economic (e.g. low wages and expensive final
products) and social (poor general living conditions) costs are often involved. The
benefits of these mining operations go to the owners of the mining companies, often
multinational companies. Closer to home there are numerous examples from heavy
industries. Often, the members of the boards of large polluting companies do not live
in the vicinity of the plants. The poorer, often uneducated workers are the ones that
inhabit the areas around the plants and suffer as a result of the pollution. These people
are not empowered to change the balance between costs and benefits. Even if they
were well informed about the situation, they would still experience difficulties ascertaining the best way to address these issues.
9. To evaluate the effect of a policy we use the compensatory approach. The final assessment of the effect of a policy involves a trade-off between the different (side) effects.
This is in contrast to the separate evaluation of effectiveness and side effects, which
is done using stand-alone criteria.
10. We deliberately chose this name to indicate the relationship to Scrivens modus
operandi method (Scriven, 1991), a procedure for linking cause and effect by the
reconstruction of a chain of events preceding the effect and the ambient conditions.
Even though some similarities seem to exist at first, there is an essential difference
between the modus narrandi and the modus operandi method. This difference lies in
the focus of the modus narrandi on the narrative reconstruction of the causal story.
The modus operandi offers a way of dealing with situations where it is impossible to
use a control group. The modus narrandi tells us more precisely how.
115
Evaluation 12(1)
11. Translation by the authors.
12. These anticipated side effects can be identified through methods such as environmental impact assessment, social impact assessment, costbenefit analysis, etc.
13. The next logical phase, after identifying the effects and therefore also reconstructing
the policy theory, is to evaluate the policy theory. Although this is an interesting evaluation question, it is beyond the scope of this article.
14. These measurements can come from official statistics, other existing sources, or sometimes have to be gathered by the evaluator.
15. Here, goal achievement is synonymous with goal attainment.
References
Bachus, K. and H. Bruyninckx (1998) Tewerkstellingseffecten en economische impact van
het bodemsaneringsdecreet. Leuven-Mechelen: HIVA-OVAM.
Barrow, C. (1997) Environmental and Societal Impact Assessment: An Introduction.
London: Arnold.
Blommestein, H., J. Bressers and A. Hoogerwerf (1984) Handboek beleidsevaluatie: Een
multi-disciplinaire benadering. Alphen aan den Rijn: Samsom.
Boskma, A. F. and M. Herweijer (1988) Beleidseffectiviteit en case-studies, Beleidswetenschap 2(1): 5269.
Bressers, J. (1998) De evaluatie van beleid, in A. Hoogerwerf and M. Herweijer (eds)
Overheidsbeleid: Een inleiding in de beleidswetenschap, pp. 11941. Alphen aan de Rijn:
Samsom.
Bressers, J. and P. Klok (1991) Hoe valt effectiviteit van beleid te verklaren? Deel 2:
Instrumententheorie, in J. Bressers and A. Hoogerwerf (eds) Beleidsevaluatie,
pp. 13654. Alphen aan de Rijn: Samsom.
Bruyninckx, H. and T. Cioppa (2000) The Effectiveness of International Environmental
Regimes: What about the Environment?, paper presented at the 41st annual convention of the International Studies Association, Los Angeles, CA, 1418 March.
Burdge, R. (1994) A Conceptual Approach to Social Impact Assessment. Middleton, WI:
Social Ecology Press.
Dunn, W. (1994) Public Policy Analysis: An Introduction. Englewood Cliffs, NJ: PrenticeHall.
EEA (2001) Reporting on Environmental Measures: Are we being Effective? Copenhagen:
EEA.
Gysen, J. and K. Bachus (2003) Sectorale uitvoeringsplannen in het Vlaamse afvalbeleid:
Effectiviteitsanalyse van een planningsinstrument. Leuven-Antwerpen: HIVA-Steunpunt
Milieubeleidswetenschappen.
Gysen, J., K. Bachus and H. Bruyninckx (2002) Evaluating the Effectiveness of Environmental Policy: An Analysis of Conceptual and Methodological Issues. Leuven-Antwerpen:
HIVA-Steunpunt Milieubeleidswetenschappen.
Hannigan, J. (1995) Environmental Sociology: A Social Constructionist Perspective.
London and New York: Routledge.
Homer-Dixon, T. (1995) Strategies for Studying Causation in Complex Ecological Political
Systems. Washington, DC: American Association for the Advancement of Science and
the University of Toronto. Found on http://www.library.utoronto.ca/pcs/eps/method/
methods1htm (accessed 28 Sept. 2005).
Keohane, R., P. Haas and M. Levy (1993) The Effectiveness of International Environmental Institutions, in P. Haas, R. Keohane and M. Levy (eds) Institutions for the Earth:
Sources of Effective Environmental Protection, pp. 324. Cambridge, MA: MIT Press.
116

Kutting, G. (1998) Assessing the Effectiveness of International Environmental Agreements: New Dimensions of Analysis, paper presented at the 39th International Studies
Association Convention, Minneapolis, 1721 March.
Lafferty, W. and J. Meadowcroft, eds (1996) Democracy and the Environment: Congruence and Conflict Preliminary Reflections, in W. Lafferty and J. Meadowcroft (eds)
Democracy and the Environment: Problems and Prospects, pp. 117. Cheltenham:
Edward Elgar.
Levy, M. and O. Young, eds (1995) The Effectiveness of International Regimes, unpublished typescript.
Mahoney, J. (2004) Tentative Answers to Questions about Causal Mechanisms, unpublished typescript.
Mickwitz, P. (2000) Evaluating Environmental Policy Instruments, paper presented at the
American Evaluation society conference, Waikiki, USA, 14 Nov.
Mickwitz, P. (2003) A Framework for Evaluating Environmental Policy Instruments:
Context and Key Concepts, Evaluation 9(4): 41536.
Mueller, D. (1979) Public Choice. Cambridge: Cambridge University Press.
Patton, M. (1997) Utilization-Focus Evaluation: The New Century Text. Thousand Oaks,
CA: SAGE.
Peterson, M. (1997a) Regimes as Governance Systems, in O. Young (ed.) Global Governance: Drawing Insights from the Environmental Experience, pp. 2763. Cambridge, MA:
MIT Press.
Peterson, M. (1997b) International Organizations and the Implementation of Environmental Regimes, in O. Young (ed.) Global Governance: Drawing Insights from the
Environmental Experience, pp. 11551. Cambridge, MA: MIT Press.
Popper, K. (1968) The Logic of Scientific Discovery. New York: Harper & Row.
Posavec, E. and R. Carey (2003) Program Evaluation: Methods and Case Studies. Upper
Saddle River, NJ: Prentice-Hall.
Rossi, P., M. Lipsey and H. Freeman (2004) Evaluation: A Systematic Approach. Thousand
Oaks, CA: SAGE.
Scriven, M. (1991) Evaluation Thesaurus. London: SAGE.
STOHO (1996) Inleiding tot de bestuurskunde. Brussels: STOHO.
Van de Graaf, H. and R. Hoppe (2000) Beleid en politiek: Een inleiding tot de beleidswetenschap en beleidskunde. Bussum: Coutinho.
Vedung, E. (1997) Public Policy and Program Evaluation. New Brunswick, NJ: Transaction Publishers.
Weale, A. (1992) The New Politics of Pollution: Issues in Environmental Politics. Manchester: Manchester University Press.
Yin, R. (2003) Case Study Research: Design and Method. Thousand Oaks, CA: SAGE.
Young, O. (1997) Rights, Rules, and Resources in World Affairs, in O. Young (ed.) Global
Governance: Drawing Insights from the Environmental Experience, pp. 123. Cambridge,
MA: MIT Press.
J O O S G Y S E N , senior researcher at the environmental policy research group,

Catholic University of Leuven and the Flemish Centre of Expertise for
Environmental Policy Sciences. Please address correspondence to: Parkstraat 47,
B-3000 Leuven, Belgium. [email: joos.gysen@hiva.kuleuven.be]
117
Evaluation 12(1)
H A N S B RU Y N I N C K X , Ph. D. environmental politics, Colorado State
University. Professor of environmental politics, Catholic University of Leuven,
Belgium and Wageningen University, The Netherlands. Co-promotor Flemish
Centre of Expertise for Environmental Policy Sciences. Please address
correspondence to: E. Van Evenstraat 2, B-3000 Leuven, Belgium. [email:
hans.bruyninckx@soc.kuleuven.be]
K R I S B AC H U S , research co-ordinator at the environmental policy research

group, Catholic University of Leuven and the Flemish Centre of Expertise for
Environmental Policy Sciences. Please address correspondence to: Parkstraat 47,
B-3000 Leuven, Belgium. [email: kris.bachus@hiva.kuleuven.be]
118

Gysen. 2006. The Modus Narrandi

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Gysen. 2006. The Modus Narrandi

Enviado por

Direitos autorais:

Formatos disponíveis

Evaluation

The Modus Narrandi: A Methodology for Evaluating Effects of Environmental

The Tavistock Institute

Additional services and information for Evaluation can be found at:

Downloaded from http://evi.sagepub.com at UNIVERSITAT AUTNOMA DE BARCELONA on March 14, 2007

The Modus Narrandi

Are we being effective? has become an increasingly central question in

Conceptualizing Effects and Effectiveness

Gysen et al.: The Modus Narrandi

Gysen et al.: The Modus Narrandi

Overview of Effects and Effectiveness

The tangible results

The match of the output of

The response of the

The degree to which the

The degree to which the

The extent to which the

The influences on the

Visible and measurable

Does not directly follow

The result of the

The extent to which a policy

Target Group Effectiveness

Gysen et al.: The Modus Narrandi

Particular Difficulties Associated with the Evaluation of

Characteristics of Environmental Phenomena

Gysen et al.: The Modus Narrandi

Influence on Environmental Policy-Making and Evaluation

Complex patterns of impact and causation

Irreversibility and threshold

Figure 1. Features of Environmental Phenomena

Gysen et al.: The Modus Narrandi

Introducing the Modus Narrandi as an Effect Evaluation

Gysen et al.: The Modus Narrandi

Causal narrative story

Figure 2. ConceptualizationMethod Relationships

Gysen et al.: The Modus Narrandi

The Steps of the Modus Narrandi

Judging the merit

Figure 3. The Modus Narrandi

Step 1: Reconstruction of the effects spectrum Step 1 contains two phases.

Gysen et al.: The Modus Narrandi

Gysen et al.: The Modus Narrandi

Gysen et al.: The Modus Narrandi

Gysen et al.: The Modus Narrandi

J O O S G Y S E N , senior researcher at the environmental policy research group,

K R I S B AC H U S , research co-ordinator at the environmental policy research

Você também pode gostar