A Rigorous Definition of Robustness Analysis

Author(s): H.-Y. Wong and J. Rosenhead

Source: The Journal of the Operational Research Society, Vol. 51, No. 2 (Feb., 2000), pp. 176182
Published by: Palgrave Macmillan Journals on behalf of the Operational Research Society
Stable URL: http://www.jstor.org/stable/254258 .
Accessed: 14/01/2015 04:13
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

Palgrave Macmillan Journals and Operational Research Society are collaborating with JSTOR to digitize,
preserve and extend access to The Journal of the Operational Research Society.

http://www.jstor.org

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

journal

of the Operational

Research

Society

(2000)

51, 176-182

y(?!2000 Operational Research Society Ltd. All rights reserved. 0160-5682/00

$15.00

http://www.stockton-press.co.ukljors

A rigorous definition of Robustness Analysis

H-Y Wong* and J Rosenhead
London School of Economics
RobustnessAnalysis (RA) is a problemstructuringmethodfor flexible planning.This paperpresentsa revised definition
of RA which is based on earlierpresentationsof RA as well as new results derivedfrom empiricalresearch.This paper
first identifies the basic concepts that are necessary for a rigorous definition of RA and relationshipsbetween these
concepts. On this foundation,this paper constructsnew definitionsof robustnessscore and discusses their uses.
Keywords: methodology;planning;problemstructuring;soft OR

Introduction
Robustness Analysis (RA), is an approachfor the evaluation of alternativeinitial strategicdecisions, which was first
proposed by Gupta and Rosenhead4.Furtherdevelopment
in the theory and methodology of RA was presented in
Rosenhead.1-3 Reported applications in the past included
the location of production plants,4 sequential development

of a chemical plant,5the location of medical facilities for a

new town,6 personal educationalcareer decisions,7 and the
regional planning of health care provision decision in
Ottawa.8 More recent methodological advances have
included the amalgamation of RA and scenario planning
for the planning of HIV/AIDS health provision.9
The applicationsof RA, and the literatureon it, are less
extensive than those of several of its counterpartsin the
field of problem structuringmethods. One consequence is
that its conceptual foundationshave not been subjected to
the same degree of critical scrutiny.The result is that areas
of ambiguity have persisted, which have also impeded the
furtherdevelopment of the approach.10The principal aim
of this paperis to presenta reformulationof the foundations
of RA on the basis of precise definitionsof all the relevant
terms. An advantage of these more secure foundations is
that their explicitness clarifies the opportunitiesfor further
developments, and some resulting extensions of the methodology are also presented in this paper.
It will be helpful at this stage to clarify the status of the
mathematicalformulationswhich will be presentedin later
sections. The starting point is to recognise that RA falls
under the heading of a decision-aiding rather than a
decision-analytic approach, with both terms taken in a

Correspondence: Dr H- Y Wong, Department of Operational Research,

London School of Economics, Hot ghton Street, London WC24 2AE, UK.
E-mail: hywacompuserve.com

broadsense. By declining the task of identifyingan optimal

decision, decision-aidingside-steps the practicaldifficulties
of identifying individual preferences, aggregating utilities
and estimating probabilities. This simplification of the
analytic task is accompanied,however, by a corresponding
loss of analyticpower. Decision-aidingthereforethrowsthe
burdenof forming these judgements and trade-offsback to
the decision makers. It can only be effective as an assistance to decision making if the elicited informationis so
structured that the problem, otherwise insurmountable
through complexity and uncertainty,is renderedtractable.
As decision aiding approachesdo not attemptto represent
or substitute for the act of decision, the explicit assumptions or axioms of rationalchoice requiredfor quantitative
approaches such as decision analysis are not needed.""2
Nor is there any scope for algebraic formulations of
performancemeasures designed for algorithmic optimisation. The employment of mathematicalformalisms in this
paper has the quite different and limited functions, firstly
by ensuring that the representationsof the problem situation to be employed within RA are unambiguouslydefined;
and secondly, by enabling developments of and operations
on these concepts to be explained in a compact manner.
The first part of this paper identifies basic concepts that
are necessaryto lay down a more secure foundationfor RA.
The second part specifies the relationshipsbetween these
concepts. The following sections articulate an extended
view of the key relationshipamong these concepts, namely
the attainability of future configurations from initial
commitment sets, and derives alternativeexpressions for
computing robustness scores. These extend the range of
aspects of option maintenanceand closure which the methodology can address. Finally there is a discussion of the
significance of adoptingthis revised view.
The researchreportedin this paper is complementedby
work on the development of a new structuredand partici-

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

HKY
Wong
andJ Rosenhead-A
definition
ofrobustness
rigorous
analysis 177

pative methodology for RA, termed Robustness Analysis

Methodology (RAM), which is presentedelsewhere.'0

graphical representationof a hypothetical situation with

three decision and implementationperiods.

Defining the basic concepts in Robustness Analysis

Commitments

In this section, basic concepts of RA are defined. These

concepts, which are necessarily simplified abstractionsof
featuresof real life situations,provide a frameworkfor RA
and may also offer a clearer understandingof the problem
situation being confronted.

A commitmentis defined as a single unique, indivisible

action that will cause specific changes to the system. It is
assumed that the implementationof the same actions in
different periods can have quite different consequences.
Therefore, similar actions that can be implemented in
different periods are treated as different commitments. In
most cases, the changes introducedby the implementation
of a commitmentwill be irreversible(though the changes
made by some commitmentsmay be cancelled by adopting
specific furthercommitments).We use

Systems and the environment

RA deals with decision problems where the results of

decisions taken will lead to changes to the system within
its environment.We define a system as a group of interacting, interrelated, or interdependent elements forming a
complex whole. Environmentis defined as the circumstances or conditions under which the system operatesand
cannot be controlled. Because only some of these circumstances or conditions will affect the existence and development of the system within the environment,all aspectsof the
environmentneed not to be considered.The relativelyloose
definition of the terms proposed here will enable us to
capturea wide range of situations.

Decision and implementation periods

A decision period is a period of time during which a

decision needs to be made from a number of available
commitments (to be defined later). An implementation
period is a period of time duringwhich the chosen commitments are implemented.Once these commitmentsare implemented,which usually takes place towardsthe beginning of
the implementation period, no further changes will be
introduced to the system until the next implementation
period. Each decision period is succeeded by an implementation period. The first decision period is termed the initial
decision period. The planning horizon lies beyond the last

implementationperiod. The durationsof decision periods

and implementationperiods are variable, with the former
typically short comparedwith the latter.Figure 1 providesa

I
t

I15
I

z,.t
to representa commitmenti that is implementablein period
t. For convenience, we put all the commitments that are
implementablein period t in the set

Zt -

In general, a plan consists of commitments to be implemented at each period up to the planning horizon. The
actual decision on the implementationof period t commitments will be made in decision period t; decisions on the
implementationof commitmentsin subsequentperiods are
not required at this stage. In practice, decisions are only
made on the commitments that need to be implemented
immediately.
Commitment sets

A commitmentset is defined as a unique feasible combination of commitments.A commitment set i that is implementablein period t is denoted by
Si, t I

A period t commitmentset only contains commitmentsthat

can be implementedduringthat implementationperiod, so
St t C Zt.

".
Planning

Horizon
1

Pecision period

Imrplermentation
period

Figure 1

Example of a situation with three decision and implementation periods.

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

Vol.51,No.2
oftheOperational
Research
178 Journal
Society

It is assumed that if a commitment set Sit is chosen for

implementationat decision period t, all of its constituent
commitmentsmust be implementedwithin implementation
period t.

Although we can consider commitmentsets that may be

implementedat different implementationperiods, in RA, a
significant emphasis is placed on the possible commitment
sets at the first decision period. These are termed initial
commitment sets, denoted by

Si,.
There are potentially a very large number of possible
combinations of commitments at each decision period.
The natureand content of the problem often makes certain
combinations of these commitments of particularinterest.
Ways of identifying and designing combinations of
commitmentsthat are likely to be of interest are suggested
elsewhere.'n

The defining characteristicof a future scenario is that its

propertiescannot be controlled.In general, a configuration
will behave differentlyunder differentfuturescenarios. We
use the subscriptf to denote a particularfuturescenario.
A perspective is defined as the set of views or value
judgements of a distinctive group of people who will be
affected by the performance of the configuration in the
environment.In general, there will be more than one such
group, and each group will have its own perspective.The
desirabilityof the performanceof a configurationwill vary
betweenthese perspectives.Therefore,it is necessaryto take
into account the groups' perspectives when choosing an
initial commitment set for implementation. We use the
subscriptp to denote a particularperspective.
Methods for the identification of perspectives and the
generationof future scenarios have been suggested.10
Desirable configurations

Configurations

A configurationis defined as a form, shape or patternthat

the system under considerationmay exhibit at the planning
horizon as a result of the implementationof a combination
of commitments.We use
CJ
to refer to configurationj. Potentially there are many
possible configurations at the planning horizon. Issues
regardingthe identificationof configurationsare addressed
elsewhere.l ?
More thanone combinationof commitmentscan resultin
the same configuration(this is because otherwise identical
actions occurring in different periods are designated as
different commitments). Each such combination of
commitments is defined as a configuration composition.
That is, a configuration composition is the union of a
particularsequence of commitmentsets which results in a
given configuration.
We use the notation
Cijk

where
Cj,k C_ZI U .* U Zt

to refer to the kth configurationcomposition of configuration].

Future scenarios and perspectives
A future scenario

is defined as a description

of the environ-

ment at the planning horizon after a hypothesised or

predicted chain of events. Since there can be many such
projections and the fultureoccurrence of these events is
unknowable,it is possible to identifNmany fulturescenarios.

Different configurationscan be expected to performdifferently in the same environmentat the planning horizon. A
range of relevantvaluation criteriacan be used for valuing
the anticipatedperformanceof these configurationsat the
planninghorizon. These criteria,which can be quantitative
and qualitative, will be scalar measures and may include
monetary values, throughputmeasures, perceived image,
and political acceptability,etc.
We define
VALi(C1)
as the performance valuation of configuration C1 using
valuation criterion i. A satisficing level, li, for criterion i
determines those configurations that are considered as
desirable on this criterion.If there are n valuation criteria,
we use the vector
L=[11

12

l]

to indicate the satisficing levels for the n valuation criteria.

For simplicity, we will first consider situations with a
single futurescenario and a single perspective.A desirable
configurationis defined as a configurationwith anticipated
performancevaluation at or above all the satisficing levels
L, that is,
VALi(Cj)

-> li

Vi.

This definition assumes implicitly that higher values for

each configuration valuation are preferred; equivalent
inequalities may be substitutedto handle situations where
lower valuationsare desired.Althoughthe use of satisficing
levels should have broad application, alternativemechanism for identifying desirable configurations may be
employed.
We define WL as the set of desirable configurationsthat
satisfy L. This is in keeping with the fact that RA is

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

H-Y
Wong
andJ Rosenhead-A
rigorous
definition
ofrobustness
analysis 179

founded on the identification of acceptable, rather than

optimal, solutions. 1"2

We now extend this definition to cover situations with

multiple perspectives and multiple future scenarios. We
define
VALq1'p(C1)
as the performancevaluation of the configurationC1 using
valuation criterioni under the conditions of futurescenario
f from perspectivep. The satisficing level on criterioni for
future scenariof and perspectivep is denoted by

and the corresponding set of satisficing levels (across

criteriai) by

We furtherdefine
WL,P
as the set of desirableconfigurationsthat satisfiesLf P. That
is
Cj E WLy.,

remainstrue if there exist other configurationcompositions

of CJ that also satisfy the above condition. We define
A(Si 1. WJP
as the set of desirable configurations that are exactly
attainableby Sij.
Stuperfluousattainability
Previous work on RA has implicitly used this concept of
exact attainability.2> The assumption has been that the
implementationof a commitmentthat is not an integralpart
of a particularconfigurationcomposition will result in the
non-attainability of the corresponding configuration.
However, there exists an alternative view of attainability
that may sometimes be appropriate:a configuration is
attainable by an initial commitment set that includes all
the commitments necessary to be made as well as additional ones that are not required by the configuration
concerned. In this case, we say that the configurationis
superfluouslyattainableby the commitmentset. Therefore,
if the commitmentset Si, contains more period 1 commitments than those required by configuration composition
Cjk,

that is,
z,Q
Si,I D cj,k n

if and only if

configurationC1 is said to be superfluouslyattainable by

'VALi, 'p(Cj) >--ii7f.p

Vi-

Procedures for the elicitation of valuation criteria, valuations and satisficing levels under multiple future scenarios
and perspectives have been developed.'0
Attainability
The preceding section has defined the fundamental
concepts in Robustness Analysis. This section will define
the relationshipof attainabilitybetween initial commitment
sets and configurations.In general, a configurationis said to
be attainableby an initialcommitmentset if, togetherwith a
set of future commitments, it can result in achieving this
configurationat the planning horizon. We distinguish two
types of attainability, exact and supe-rfuous, using the
definitions of commitment sets and configurationsalready
established.
Exact attainability

If the period I commitmentsin a configurationcomposition

Ci k are identical to those in an initial commitmentset S, I,
that is,
Si,C= Cj,knZ,
the configurationCj is said to be exactly attainable by Si,,.
This exact attainabilityrelationship between Si and C1

Si,I

It should be noted that in this paper, the symbols C and

D will be used to define proper subsets which exclude the

condition of equality.
We define A+(Si1, WLf ) as the set of desirableconfigurations that are superfluouslyattainable by the commitment set Si I
The desirability of a configurationmay be taken to be
unaffected by whether or not it is attained exactly or
superfluously,since a superfluouscommitment is, by definition, an independentand unrelatedelement.
Superfluousattainabilitywill occur where early investments are made which are redundant for the particular
configuration under consideration. However, the formal
definition can produce counter-intuitive effects in cases
where commitments which are commutative in effect are
interchangedbetween the initial and a later time period.
Here an apparentsuperfluousattainabilitymay be signalled
for one of the arrangements,where in the event there will
be no superfluousresources. Such cases should be disregarded-though to eliminate them formally from the definitions would requirea quite disproportionateextension of
the notation.
Characteristicof exact and superfluousattainability
The distinctive characteristicof exact attainabilitybetween
a particularinitial commitmentset and configurationis that

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

Research
180 Journal
oftheOperational
Vol.51,No.2
Society

each of the commitmentsin the commitmentset must form

an element of the configuration. Conversely, superfluous
attainability is characterised by the existence in the
commitment set of 'redundant' commitments, which do
not feature in the specific configuration.Where a configuration can only be constructedby a single configuration
composition, the conditions for exact and superfluous
attainability are mutually exclusive: a configuration that
is exactly attainableby an initial commitmentset cannot be
superfluouslyattainableby the same commitment set.
Where a configurationcan be achieved by implementing
different configuration compositions, it is in principle
possible that a configurationcan be attained both exactly
and superfluouslyby the same commitment set. Suppose
that configuration C1 can be achieved by implementing
either configurationcomposition Cijk or Cj k. Then if both
Si, I

jZkn

and

Si

commitmentset may attaindifferentconfigurationsexactly

and superfluously. The number of links of both kinds
between a given initial commitment set and desirable
configurations will vary depending on the number of
desirable configurations as determined by the satisficing
levels set for each future scenario and/or perspective. As
these levels are reduced, the number of attainableconfigurationstend to rise.
The concept of superfluousattainabilityis a new conceptual development. It draws attention to the possibility of
situations where initial commitment sets can increase the
number of attainable configurations, at the expense of
committing potentially redundant commitments. The
types of robustness scores defined in the next section are
based on the two kinds of attainabilitypresented in this
subsection.

DC, knZl

were to hold then C, could be attained exactly and superfluously by Si',. In this rather remote eventuality, which
could only obtain if the effect of implementationof time
dependent commitments is not commutative, it would be
necessary to assess any distorting effect on robustness
calculations.
The general characteristicsof the attainabilityrelationships may be illustratedgraphically. Figure 2 illustrates a
hypotheticalproblem with a numberof initial commitment
sets (left hand side of diagram), desirable configurations
(right hand side of diagram) and the possible attainability
relationships between them (a solid line indicates exact
attainability while a dashed-line indicates superfluous
attainability). This figure illustrates that a configuration
may be attained exactly by one group of commitment
sets and superfluously by another. Similarly, a single

Robustness score
This section presents two alternativedefinitions of robustness score: exact and superfluous. These definitions are
based on the definitions of attainabilitypresented in the
previous section. A further concept that of aggregate
robustnessscore will also be defined.

Exact robtustnessscore
The exact robustnessscore of an initial commitmentset is
defined as the ratio of the number of desirable configurations that are exactly attainableby a commitmentset to the
total number of desirable configurations. We define

C2

S~~~~~~~~~~

C5

CG

Cg

Co
CFio

Figure 2

Attainabilityrelationshipsbetween commitmentsets and desirableconfigurationsof a hypotheticalproblem.

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

H-Y
Wong
andJ Rosenhead-A
rigorous
definition
ofrobustness
analysis 181

r(Si 1, WL ), the exact robustnessscore of the commitment

set Sl 1, as
r(Si 1, WL)

= jA(Sl,

]~P)-

WIf)I

IWL11I

This definition of robustness score will yield the same

results as the earlier definition of RA.
Superfluousrobustnessscore
The superfluousrobustnessscore of an initial commitment
set is defined analogously as the ratio of the number of
desirableconfigurationsthat are superfluouslyattainableby
a commitmentset to the total number of desirable configurations.We define r?(Si 1, WLf ), the superfluousrobustness score of the commitment set Si, 1, as

rJA(S1,I,
r+(Si,

WL,)I

~fp
WL p)
1?

IWLf
jW I

Aggregate robustnessscore
We define the aggregate robustness score of an initial
commitment set as the ratio of the number of desirable
configurationsthat are eitherexactly or superfluouslyattainable by this commitmentset to the total numberof desirable
configurations.We denote
r* (Si, 1, WLf

as the aggregate robustness score of the commitment set

Si,1, which is given by
r*(Si,1

WL)

1,
=A(S WiLf5)
)UA(S,1,
Lf'p

)I

WL,1I
~I

= r(Si 1, WLf

) +

r (Si,1, WLf)

The definition of aggregate robustness score provides a

combined measure of the attainability relationships
between initial commitment sets and configurations.

In these respects, as in others,both exact and superfluous

robustnessscores share the broad characteristicsof robustness identifiedin previous publications.Eitheror both may
be employed, according to circumstance, to assess the
extent of useful flexibility which will remain following
alternative initial decisions. In accordance with the decision-aiding status of RA, these flexibility assessments are
intended as contributionsto the informationalbackground
to choice, not as sole determinantsof proposed choice. In
virtuallyevery case, there will be other factorsrelevantto a
decision-makinggroup or process, not least the shorter-run
effects of alternativecommitments.However, in general, it
is to be expected that the robustnessscores should be more
influentialin situationswhere uncertaintylevels are high.
The discrimination between exact, superfluous and
aggregate robustness which the definitions in this paper
introduce provide additional information content. Exact
robustnesscapturesthe essence of robustnessas previously
described,though on a sounder conceptual footing. These
definitionsmake it clear that this essence was to providean
indicatorof the degree of useful flexibility left open, in a
particularsense. This is that options (configurations)are to
be consideredas 'left open' only if all the initialprerequisite
actions are taken, and if there are no components of the
initial commitmentsmade which would be irrelevantto that
option. Aggregate robustness by contrast adopts a more
relaxed posture on 'openness' (or attainabilityas we have
termedit). An option is regardedas maintainedopen by an
initial commitmentset so long as none of its prerequisites
has been omitted, regardless of the extent to which the
option may fail to put all of the initial resourceinvestments
to work. Superfluousrobustnessmay be seen as the difference between these two indicators.It thereforeprovides a
measure of the degree of extra flexibility which the legitimisation of possibly redundantinvestmentscan provide. It
can therefore contributeto an assessment of the trade-off
betweenpossibly redundantinitialresourceinvestments,and
the extraflexibilitywhich they mightbuy. The incorporation
of such redundancyinto the robustnessframeworkallows it
to representan importantsource of flexibility that enables
the system to respondto futureuncertainties.'3-15

Characteristicsand interpretationof robustnessscores

The exact, superfluousor aggregate robustnessscore of an
initial commitment set can range from zero to one. If the
initial commitment set Si cannot exactly attain any of the
desirable configurationsthen the exact robustness score is
zero. Should no desirable configurationsexist at satisficing
levels Lfp, the exact robustness score is undefined. The
same logic applies to both superfluous and aggregate
robustness scores. As the number of desirable configurations that are exactly or superfluously attainable by a
commitment set approachesthe total number of desirable
configurationsat the planninghorizon, its aggregaterobustness score approachesone.

Discussion
The new definitions of Robustness Analysis, through the
use of set notations, describe clearly and explicitly the
relationshipsamong the concepts of commitments,commitment sets, configurations,attainabilityand robustness. In
particular,the notion of attainability,using commitmentas
the common element, permits the explicit and logical
definition of robustness. Collectively, the ideas presented
in this paper widen the vocabularyavailable to capturethe
prevailing complexity.
The introductionof a commitment-centricapproachto
problem structuringin RA helps to clarify the relationship

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions

182 Journal
oftheOperational
Research
Vol.51,No.2
Society

between initial commitment sets and permits the development of the new concept of attainability.An added advantage of this approachis that the process of interrogatingthe
situationand eliciting informationfrom decision-makersin
terms of elementarycommitmentsin itself is a contribution
to achieving greater flexibility.16
Previous definitions of RA offered no explicit provision
for handling problems with multiple criteria, even though
the idea has been suggested previously.3 This imposed a
considerable limitation on its applicability to problems
where configurationsneed be assessed from a variety of
angles. The new definitions presented in this paper permit
the incorporationof multiple criteria,both quantitativeand
qualitative, to evaluate performance of configurations.
Desirable configurationsare then identified based on the
satisficing levels set for each criterion.
Another significant development of the theory of RA is
the differentiation made between the mutually exclusive
exact and superfluous attainability relationships. Exact
attainability only considers combinations of possible
commitments which are compatible with a given configuration, with combinations containing elements beyond
those required by that configurationexcluded. The introduction of the concept of superfluousattainabilityrelaxes
this assumption. It identifies the increment in flexibility
provided by the additional configurations which can be
attained as a result. This criterion is therefore appropriate
where the decision-making environmentdoes not rule out
the possible redundancyof elements of future options.
Although greater differentiation of robustness, in its
exact, superfluousand aggregate forms, is now available
to indicate the flexibility offered by each decision in
practice, the use of these criteriawill need to take account
of the decision making climate. The comparison of exact
and aggregate robustness scores can be expected to be
useful only under situations where the additional investment of potentially redundantcommitmentsis likely to be
acceptable, for instance, when resource constraintsare not
tight, or when the turbulenceof the environmentis such as
to make committing extra resources to 'buy' additional
flexibility seems an insurance policy worth considering.
Superfluousand aggregate robustnessmay be unsuitableif
converse conditions apply, and the decision makers consequently respond negatively to any form of redundancy.

Conclusions
Complex problems often involve a considerablenumberof
decision makerswho do not necessarily share similarviews
and objectives. By neglecting such diversity, RA in its
earlierformulationoffered no guidance for handlinguncertainties aboutguiding values, 7 an effective disqualification
from an importantclass of decision problems. The extensions to RA presented in this paper formalise the calcula-

tion of robustness scores based on the alternative

perspectives held by different decision making groups.
The process of eliciting and analysing these contrasting
perspectives can itself be expected to improve mutual
understanding, so making negotiated accommodations
more likely to merge, particularlyin political situations or
inter-organisationalproblems.'8

References
1 Rosenhead JV (1980). Planning under uncertainty: I. the
inflexibilityof methodologies.J Opl Res Soc 31: 203-216.
2 RosenheadJV (1980). Planningunderuncertainty:II. a methodology for robustnessanalysis.J Opl Res Soc 31: 331-341.
3 RosenheadJV (ed) (1989). Robustnessanalysis:keeping your
options open. In: Rational Analvsisfor a Problematic World.
ProblemStructuringMethodsfor Coomplexity,
Uncertaintvand
Conflict.Wiley: Chichester.
4 GuptaSK and RosenheadJV (1968). Robustnessin sequential
investmentdecisions. Mgmt Sci 15: B 18-B29.
5 Caplin DA and KombluthJSH (1975). Multiobjectiveinvestment planningunderuncertainty.OMEGA3: 423-441.
6 Llewelyn Davies Kinhill Pty Ltd (1976). Health Services for a
New City,Albury-WodongaDevelopmentCorporation,Australia.
7 RosenheadJV (1978). An education in robustness.J Opl Res
Soc 29: 105-111.
8 Best G, ParstonG and Rosenhead JV (1986). Robustness in
practice the regionalplanningof health services. J Opl Res
Soc 37: 463-478.
9 Rizakou E (1995). Scenario-RobustnessMethodology: An
Approach to Flexible Planning under Uncertainty with an
Application to AIDS-Related Resource Allocation, PhD
Thesis, Departmentof OperationalResearch, London School
of Economics.
10 Wong HY (1998). Making Flexible Planning Decisions: A
Clarificationand Elaborationof the Theory and Methodology
of Robustness Analysis, PhD Thesis, Departmentof Operational Research,London School of Economics.
11 Watson SR and Buede DM (1987). Decision Synthesis: The
Principles and Practice of Decision Analvsis. UniversityPress:
Cambridge.
12 Bell DE, Raiffa H and Tversky A (eds) (1988). Descriptive,
normativeand prescriptiveinteractionsin decision making. In:
Decision Making: Descriptive, Normative and Prescriptive
Interactions.UniversityPress: Cambridge.
13 Stigler G (1939). Productionand distributionin the shortrun.
J of Political Econ 47: 305-327.
14 EppinkDJ (1978). Planningfor strategicflexibility.Long Range
Planning 11:9-15.
15 CollingridgeD andJamesP (1991). Inflexibleenergypolicies in
a rapidly-changingmarket.Long RangePlanning 24: 101-107.
16 MandelbaumM (1978). Flexibility and Decision Making: an
Exploration and Unification. PhD Thesis. University of
Toronto.
17 FriendJ and Hickling A (1987). Planning linderPressulre:the
Strategic Choice Approach.Pergamon:Oxford.
18 ChurchillJ (1990). Complexity and strategicdecision-making.
In: Eden C and RadfordJ (eds). TacklingStrategicProblems.
Sage PublicationsLtd: London.
Received March 1996;
accepted September 1999 after two revisions

This content downloaded from 202.94.83.200 on Wed, 14 Jan 2015 04:13:42 AM

All use subject to JSTOR Terms and Conditions