Data Envelopment Analysis, Operational Research and Uncertainty

Author(s): R. G. Dyson and E. A. Shale

Source: The Journal of the Operational Research Society, Vol. 61, No. 1, Part Special Issue
for 60th Anniversary of the Journal: Challenges for OR (Jan., 2010), pp. 25-34
Published by: Palgrave Macmillan Journals on behalf of the Operational Research Society
Stable URL: http://www.jstor.org/stable/40540225
Accessed: 06-07-2016 17:35 UTC

REFERENCES
Linked references are available on JSTOR for this article:
http://www.jstor.org/stable/40540225?seq=1&cid=pdf-reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact support@jstor.org.

Palgrave Macmillan Journals, Operational Research Society are collaborating with JSTOR to
digitize, preserve and extend access to The Journal of the Operational Research Society

This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
Journal of the Operational Research Society (2010) 61, 25-34 2010 Operational Research Society Ltd. All rights reserved. 0160-5682/10 ^O

www.palgrave-journals.com/jors/

Data envelopment analysis, operational research and

uncertainty
RG Dyson* and EA Shale
University of Warwick, Coventry, UK

This paper discusses a number of applications of data envelopment analysis and the nature of uncertainty in
those applications. It then reviews the key approaches to handling uncertainty in data envelopment analysis
(DEA) (imprecise DEA, bootstrapping, Monte Carlo simulation and chance constrained DEA) and considers
their suitability for modelling the applications. The paper concludes with suggestions about the challenges
facing an operational research analyst in applying DEA in real- world situations.
Journal of the Operational Research Society (2010) 61, 25-34. doi:10.1057/jors.2009.145

Keywords: data envelopment analysis; uncertainty; Monte Carlo simulation

Introduction multiple inputs and outputs using linear programming and

called their approach data envelopment analysis - as the fron-
Data envelopment analysis (DEA) is a method for analysing
tier, formed by the efficient units, envelopes the inefficient
a set of organisational units in order to identify the efficient
units. In our work, we follow those approaches in the sense
units so that they can become benchmarks or peers for the
that we consider the situation of interest to be a set of existing
inefficient units in the set and in a cooperative system can
units and our concern is to identify the efficient ones, which
facilitate the spread of best practice. Examples of the units are
can then provide targets and benchmarks for the inefficient
schools, bank branches and retail outlets and they typically
units and can be sources of best practice. It is important to note
involve multiple inputs being converted into multiple outputs.
that not all efficient units will necessarily be selected as bench-
The DEA model was initially formulated as a deterministic
marks, that is sources of best practice. This approach is exem-
model, but there have been numerous developments that take
plified by the first application that the first author encountered
into account uncertainty including chance-constrained (CC)
working with Emmanuel Thanassoulis and John Foster who
DEA, imprecise DEA (IDEA), Monte Carlo simulation and
was then seconded to the (UK) Audit Commission. The appli-
bootstrapping. The authors have used DEA to analyse various
cation of concern was local government rates departments in
real-world situations involving, for example, local government
England (Thanassoulis et al, 1987; Dyson and Thanassoulis,
departments, bank branches, universities and public houses.
1988). Rates were at that time the local property tax. The
In these and other applications, DEA has been used determin-
efficiency of local government was the concern of the Audit
istically and any uncertainty in the situation has been handled
Commission and the then Deputy Director Ross Tristem, who
only implicitly or by sensitivity analysis. In this paper, we
return to several of the situations and examine the data and was in frequent contact with the Operational Research group
at Warwick, expressed his concern that in local government
the models in order to consider the nature of any uncertainty
there were views about which authorities were the most effi-
and how it might best have been brought explicitly into the
cient, but he was interested in exploring the possibility of a
analysis. Our interest lies in the interface between uncertainty
more objective approach and questioned whether DEA might
as met in practice and the theoretical tools proposed in the
literature. be an appropriate mechanism. The Audit Commission at that
time was charged with identifying and spreading best practice
The origins of DEA lie with Farrell (1957), who advo-
throughout the system of local government and, interestingly,
cated identifying an empirical efficient frontier, that is, one
explicitly agreed not to publish comparative tables of perfor-
formed by a set of real units and based on observed best prac-
mance (which have obsessed governments and DEA analysts
tice, rather than formulating a theoretical frontier based on a
in recent years). Their approach, therefore, was primarily
production function, fashionable at that time in economics.
developmental rather than the more summative stance preva-
Chames et al (1978) developed the approach to encompass
lent in the public sector today with the widespread publica-
tion of league tables and a name and shame mentality. (See
* Correspondence: RG Dyson, Operational Research and Management
Sciences, Warwick Business School, University of Warwick, Coventry, CV4 Amado and Dyson (2009) for a recent developmental study in
7 AU UK the public sector.) The Audit Commission at that time sought
E-mail: r.g.dyson@warwick.ac.uk best practice, studied the mechanisms delivering it and then

This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
26 Journal of the Operational Research Society Vol. 61, No. 1
aimed to spread it throughout the system by the use of best
practice guidelines. This is entirely consistent with the oper-
ational research (OR) approach of studying real situations,
engaging with the key personnel and developing methods for
improvement as developed by Blackett (1950). Risk analysis,
an area we will make appeals to later in this paper, also has
a history of involving managers. This engagement is a vital
source of information for model building and engenders confi-
dence in the results of the analysis
This paper proceeds as follows. In the next section,
we examine four situations where DEA has been applied,
consider the nature of any sources of uncertainty in the data
and illustrate some of the ways that the nature of that uncer-
tainty would impact on the efficiency scores of the units. We
then review various approaches to dealing with uncertainty
in DEA in the literature and consider to what extent they are
appropriate vehicles for handling the situations we outline.
Finally, we discuss the challenges facing the use of DEA
under uncertainty from an OR perspective.
The nature of data in DEA applications
In this section, we consider four situations where DEA has
been applied but focus our attention on the data itself and the
nature of any uncertainty in that data. The examples selected
come from cases where there was no shortage of data so that
the issues discussed are as far as possible not driven by lack
of information.
UK universities
The first example relates to the assessment of universities,
see Athanassopoulos and Shale (1997), Sarrico et al (1997)
and Sarrico and Dyson (2000). In the latter two papers, DEA
models were used to assess the relative efficiency of depart-
ments in universities, for example physics, business studies,
across the UK university sector. We will concentrate on the
model in Sarrico et al (1997), which was from the perspective
of the students prior to identifying their preferred universities,
updated with the types of data used by Sarrico and Dyson
(2000). The outputs were a measure of teaching quality, a
measure of research quality, the employment prospects of
students, the amount of accommodation on the campus, the
cheapness of living at the university and the expenditure on
libraries. A nominal input was used as the outputs were not
volume related. At that time the measurement of the quality
of teaching in the universities was by direct assessment with a
team of assessors spending 2-3 days with a university depart-
ment, sitting in on lectures, examining materials, looking
at assessment schemes and so on. The assessment had four
dimensions and up to six points could be awarded on each,
giving a potential range of scores from 0 to 24. There was
a well-developed and consistent process for the assessment
that the different teams complied with. The result of this was
that it was rare for scores to be considered to be incorrect
by more than a single point on each of the four different
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
dimensions. It would therefore be reasonable to assume that
there was a high chance (say 95%) of the true score being
within two points either side of the recorded score. From this
information, we could construct a normal distribution on the
data subjectively. At first sight, this seems problematic at the
top end where a recorded score of 24 might be given but we
could either use a different distribution at the extremity or
simply assume that the underlying continuum went beyond 24.
In this way, a deterministic teaching rating could be replaced
by a subjective probability distribution as in the risk analysis
approach (Hertz, 1964). With only a general understanding of
the nature of the uncertainty, the recorded score would have
to be taken as the mean value (or mode for a skewed distribu-
tion). If, however, underlying information about the construc-
tion of the recorded score were available, it may be that the
true mean would be estimated to be a little lower or a little
higher than the recorded score, in which case that value could
be used. Also the degree of uncertainty could conceivably be
different for different departments. This however would need
detailed information about the assessments of every depart-
ment which was not available at the time.
The approach to assessing the quality of research was done
through a national research assessment exercise (RAE) where
the research quality of individuals in each department was
assessed by consideration principally of their output in terms
of journal articles and books. In determining a departmental
score some account would also have been taken of the number
of doctoral students, the value of research grants and contracts
and the esteem of the department. The research quality of the
department was given a rating on a scale 1-7 (actually, 1, 2,
3(a), 3(b), 4, 5 and 5* were used) where the highest value was
intended to describe a department in which the majority of
the output was of international standard and at the bottom end
the research was of sub-national standard. The most recent
RAE has adopted a different scoring mechanism although
the methodology is broadly similar. Given the thoroughness
of the assessment process it would have been unlikely that
errors of more than one point in either direction would be
made, so again a subjective symmetrical distribution can
be constructed around the recorded score. The seven-point
scale and the descriptors used suggest that the assumption of
an interval scale is not unrealistic.
The employability of the students was measured by the
percentage in employment or further training 6 months after
graduation. The uncertainty in the data here relates to the
fact that not all students could be contacted after 6 months
and the information was self-reported. The self-reporting can
lead to a bias as it is understood that graduates in a job
are more likely to respond to an enquiry than those unem-
ployed. Consequently, it was felt that an optimistic estimate
of the true score would be 2% higher than the recorded score
but a pessimistic one would be 5% lower. Here, we would
have an asymmetric distribution which could be represented
by say a lognormal or beta distribution. In this analysis, a
key perspective was that of the student seeking to apply to
 RG Dyson and EA Shale - Data envelopment analysis, operational research and uncertainty 27
the university, and for accommodation the assumption was
that a higher percentage of university-owned accommodation
made the university more attractive. As the university would
know the addresses of their students the measure would have
a relatively small error associated with it. A further measure
was the cost of living at a particular university based on the
average debt of graduates of that university. This was felt to
be important, at least for many students, and London was
known to be rather expensive. The actual measure used was
the difference between the maximum average debt at any
university and the average debt of that particular university,
so that the one with the lowest debt would have the highest
score. As the level of debt might well depend on the demo-
graphics of the student population as well as the cost of living
this measure would have a relatively high degree of uncer-
tainty. Finally, the library expenditure per student was seen as
a surrogate for academic support costs, but this was not avail-
able for the entire sector and so was not always used in the
analysis. No explicit measure of staff resources was used as
it was assumed that the level of resource would be reflected
in the teaching and research scores. The assumption here is
that from a student perspective more means better, but from
an institutional perspective, this might hide inefficiencies in
the use of resources - Sarrico and Dyson (2000) take an insti-
tutional perspective.
It can be noted from the previous discussion that the data
points could have been replaced by subjective probability
density functions, rather than, for example, simply ranges and
that some of the measures exhibited relatively high degrees of
uncertainty whereas others such as the percentage of univer-
sity accommodation exhibited a relatively low degree. This
phenomenon can have profound implications for the efficiency
scores and this issue is considered in the section on approaches
to uncertainty.
Rates departments
In the application to rates departments (see the Introduction
section for context and references), there was a single input
which was the cost or budget of the department and three
outputs, one relating to the number of households from which
the rates or taxes were to be collected, a second relating to
the number of businesses and a third relating to the number
of distress warrants issued. This latter measure was one of the
steps taken in seeking payment from recalcitrant rate payers
and was included as it was clear that those rate payers would
consume significantly more resource. Here again it would be
reasonable to assume that errors in the data could be repre-
sented by subjective distributions but the distress warrants
measure presents an interesting complexity. The issue here is
that the actual data itself would be known with a consider-
able degree of accuracy; however, the uncertainty lies more
in the impact on the department of pursuing recalcitrants and
this might vary considerably from individual to individual and
indeed from department to department. It could be argued
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
that the distress warrants measure is in fact a surrogate for
some measure of difficulty in collecting rates due to socio-
economic conditions such as a highly mobile population and
this issue of surrogate measures will be considered in more
detail in the next example.
Public houses
The second author led an investigation into the efficiency of
a chain of branded public houses (Shale, 1996, 2009). The
inputs here were labour costs, population within a 5 mile
radius of the public house (pub) and weighted managerial
talent. The outputs were the 'wet' and 'dry' take, that is the
revenue from beverage and food sales, and a proxy measure
of the attractiveness of the location. The labour cost data
were held in two locations collected by different mechanisms
and for separate purposes, accountancy and human resources
(HR). The latter function was responsible for the company's
profit related pay scheme and the control of labour costs was
a major contributor to that scheme. These data can therefore
be expected to be accurate as there is a clear validation mech-
anism available. The population within a 5 mile radius was
chosen as a measure of potential custom. The use of the entire
population was not a compromise but actively selected due
to the marketing of facilities for all ages from 'toddlers' to
senior citizens. The radius was chosen as a result of statis-
tical analysis but confirmed by exit polls that showed that
80% of customers travelled 5 miles or less. There remain
futher sources of uncertainty in these data however. Using
these data as a measure of potential custom assumes a homo-
geneous response in terms of leisure preference and dispos-
able income across all the locations and ignores potential
access barriers such as major trunk routes which are known
to modify consumption behaviour. Furthermore, as these data
are based on census returns demographic changes have to
be estimated in the years between the census returns by the
database provider. With respect to the original data, the UK
census includes statements on coverage and completeness
of returns (Office for National Statistics, 2005). This type
of measure can be categorised as both environmental and a
proxy or surrogate for the latent variable of demand for these
services if delivered in the specified location.
The weighted managerial talent measure was constructed
in different ways in the two studies but there were signifi-
cant similarities. In both cases, the measure was constructed
as the product of the floor area available for the genera-
tion of revenue and an index designed to capture the talent
of the manager for running this type of outlet (the use of
floor area was necessary to convert the talent index into a
volume measure for compatibility with the other measures).
The floor area was estimated from the development blueprint
and although there may have been some layout changes this
would have a minor impact. The managerial talent measure
was much more open to debate. Initially, a qualitative judge-
ment was made based on all available evidence by the HR
28 Journal of the Operational Research Society Vol. 61, No. 1
department and the area managers. This was accepted within
the organisation to be subject to some bias, profitability of an
outlet sometimes being seen as 'proof of talent. The company
invested in a well-respected psychometric profiling exercise
run by an internationally recognised external provider. This
removed the potential for bias and within the exercise itself
reduced the uncertainty of the numeric measure which was
a count of desirable responses to a battery of 108 ques-
tions. The greatest remaining source of uncertainty is whether
talent can be measured in this way. To deny this type of
exercise on principle is possibly to denigrate psychometric
methods as a whole. Such a debate is beyond the scope of this
paper.
The revenue data were held in the company's accounting
database and derived directly from electronic point of sale
data. Although some errors are inevitable given the number of
'keystrokes' the tills receive and the potential for till malfunc-
tion, this type of data is subject to intense scrutiny, routine
cross-checking and cleaning and can therefore be expected
(in the absence of undetected fraud) to be accurate.
The final measure chosen (population within 1 mile) was
intended to capture the attractiveness of the site; the company
thinking was in terms of 'aspirational' locations. Irrespec-
tive of whether the public houses were located in a rural or
semi-rural position, adjacent to park land or open water, or
in an upmarket low density housing area, in all these cases
the population in a 1 mile radius was low. This is a quanti-
tative proxy measure for a qualitative aspect of the location.
Its appropriateness is context dependent. As a data item, it is
subject to some of the same issues as the population figures
discussed above. However, from the uncertainty perspective
it is its nature as a proxy that is the source of greatest doubt.
The precise relationship between the measure used and the
concept of an 'aspirational' location cannot be fully known as
any data depending upon perception, such as attractiveness,
must be subject to uncertainty. There is a significant inter-
disciplinary literature on the measurement of perceptions and
expectations, for example Bitner (1990) looked at the effect
of the physical surroundings on the perception of service
quality. Attractiveness might seem to be naturally an input
but the measure used meant that more - higher density of
population - was less attractive so was designated an output in
the model. Using an anti-isotonic input variable as an output
is a technical issue discussed in Shale (1996) and Dyson
et al (2001).
Travel agencies
A larger scale DEA exercise involved a chain of travel agents.
Their core business included the sale of flights and 'package'
holidays and it is this aspect of the operation that was anal-
ysed. Two outputs were used and in both cases the data was
expected to be accurate. The revenue generated was extracted
from the accountancy system and subject to all the usual
checks of audit. The number of customers would be accurately
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
known because of the regulations surrounding the booking of
flights. In the minority (for this type of business) of book-
ings without flights or other travel, insurance issues should
confirm numbers.
The model contained 1 1 inputs which will be reviewed here
in order of increasing data uncertainty. The number of service
points in an agency is known and layout or other changes can
easily be incorporated as non-integer values for the period of
analysis. Sales, support and managerial staff were recorded
in the model as full time equivalents. It is expected that these
would be known very accurately as employment contracts are
legal documents. There might be issues of absence, overtime
or the number of employees varying across the period of anal-
ysis but these could be resolved by using numbers of hours
salaried. Branch costs (excluding payroll) were taken from the
accountancy system and thus were expected to be consistent
within the limits of such a system. The average industry expe-
rience of staff in years might be subject to some doubt about
candidates' exaggerating their experience before joining the
company - otherwise it would be accurate - what is uncertain
is the relationship between experience and generating sales.
The floor space dedicated to customer facing tasks is known
but not every square metre will be equally easy to use just as
every member of staff is not equally effective. This is not a
serious problem even though the DEA model works on the
assumption of equality. If a unit is found to be inefficient then
a review of the quality as well as the quantity of the resources
is a natural managerial reaction to the assessment.
The remaining four inputs are external, environmental
and proxy measures subject to far greater uncertainty.
Market potential was included by using the population at the
'town' level; the company's preferred measure as the outlets
were located on high streets and in shopping centres that
drew customers from a wide radius. This is subject to the
same provisos as the similar figures in the pubs case. This
'potential' was complemented firstly by a count of travel
agents in the town as a measure of competitive pressure,
although there is again a homogeneity of effect issue, and
secondly by a measure of average household spend taken
from an external database, an indicator of wealth and hence
potential for spending disposable income on travel. Finally,
there was a variable intended to capture the quality of the
retail location itself. In effect this recorded how far the
agent was from the nearest Marks and Spencer (M&S) store!
Although this measure did not appeal to the analysts involved,
the company management were absolutely convinced it
worked on the basis of their knowledge of M&S's location
decisions and the similarity of the customer bases of the two
companies.
Our experience of DEA model building and knowledge of
the literature suggests that there are generic data categories
and sources of uncertainty that may have wide applicability.
We have summarised some of the most common of these and
their characteristics in Table 1 , although the categories are not
exclusive or exhaustive.
 RG Dyson and EA Shale - Data envelopment analysis, operational research and uncertainty 29

Table 1 Data uncertainty, sources and effects

Type of data/uncertainty Sources of uncertainty Effects and characteristics

Measurement error Human error or technical Symmetric if random error but skewed under bias. Large errors
malfunction or systematic bias likely to be detected by routine data cleaning.

Judgmental data Subjective nature of the quan- A highly variable category. Undertaken by a team of experts
tification of the measure using a carefully controlled process subject to wide-ranging
(perhaps public) scrutiny the measure is likely to be accurate,
with limited uncertainty. Individual or small group assessment
using no formal methodology and not subject to further review
may be highly uncertain.

Environmental or external data Complexity, quality control Environmental factors may be required to summarise a complex
and timeliness situation in a single or few numbers, for example a count of
the number of competing establishments does not necessarily
capture the strength of the competitive effect. By its nature a
summary omits the fine detail and is hence a potential source
of uncertainty. Data sourced from external providers potentially
has quality issues outside the control of the user. The review
and updating of the data may not coincide with the internally
generated data leading to timeframe mismatch.

Self-reported data Behavioural bias Non-response bias and behavioural side-effects when certain
responses are perceived to be more desirable.

Psychological data for example Effectiveness of measurement The analytical response to this type of data varies widely by
perceptions/expectations and instruments and behavioural discipline. It is the bedrock of the work of psychophysicists
psychological profiling bias but is subject to criticism by scientists trained in the classical
paradigm. Accepting that people are difficult to quantify but
that it is essential to do so for the realistic DEA of a social
system, the focus needs to be on the design of the data collection
and identification of sources of uncertainty and bias within it.

Proxy/surrogate data Indirect measurement Here the data used is either a substitute for a measurable
but unrecorded factor or is representing a latent or unmeasur-
able quantity. The former is regrettable and may potentially be
addressed in the longer term. The onus in both cases is on the
analyst to understand as far as possible the relationship between
the data possessed and that desired. There is a considerable
literature on latent variables (in many disciplines).

In summary, from our approach in the above cases and Approaches to uncertainty in DEA
in other real-world applications, we conclude the following
about the nature of data uncertainty in DEA. There may be In this section we introduce and discuss a number of

uncertainty in any of the inputs and outputs. We would not
necessarily expect to be able to describe this from panel
data as suggested, for example, by Kao and Liu (2009), as
we could not always assume stationarity through time due
to learning effects and technological change. Where data
approaches to handling uncertainty in frontier analysis and
DEA. We focus only on the basic methods, as our aim is
to differentiate and compare the main approaches. We do
not attempt to survey the entire set of approaches and their
variations.

exhibit uncertainty we would envisage both symmetrical
and skewed distributions, often in the same application, and
these would typically be constructed subjectively. For an
approach to constructing subjective distributions see, for
example, Hertz and Thomas (1983). Our units are character-
ized by a common set of inputs and outputs but they are not
necessarily conforming to a common production function.
A clear example of multiple production functions would be
that described in Chames et al (1981) where they analysed
two different educational programmes.
A well-known approach to determining an efficient frontier
under uncertainty is stochastic frontier analysis (SFA) (Aigner
et al, 1911 ' Meeusen and van Den Broeck, 1977). Here the
frontier is not an empirical one as in DEA. It is assumed
that the true/theoretical efficient frontier is generated by a
model linking a single output to multiple inputs. There are
two sources of error, one, often represented by a half-normal
or exponential distribution, attributed to controllable factors
leading to inefficiencies and thus less than optimal output,
and a second, assumed to be normally distributed, due to

This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
30 Journal of the Operational Research Society Vol. 61, No. 1
Ac
10
,,-^m J. '
5 1 ExC
Figure 1 The DEA efficient frontier.
uncontrollable factors often external to the unit. To specify the
stochastic frontier it is necessary to specify a functional form
for the production frontier and assume distributional forms
for the two error terms. A misspecification of the functional
form can lead to errors in the efficiencies.
The original formulations assumed a single output; later
developments address multiple outputs in the model using
distance functions at the cost of some complexity (Fare and
Grosskopf, 1990). The approach does not consider errors in
the inputs and outputs per se but can include them only as
a part of the uncontrollable error term. However from our
perspective, we consider that the distributional assumptions
required, the lack of focus on the uncertainty in individual
inputs and outputs, the difficulty in handling multiple inputs
and outputs and the focus on a theoretical frontier make the
approach unattractive.
IDEA was first introduced by Cooper et al (1999) and
covered bounded and ordinal data. We will concentrate here
on illustrating and discussing the use of bounded data (ranges)
to describe any uncertainty using the simplified formulation of
Zhu (2003). Figure 1 illustrates the basic DEA model for a set
of schools where the two outputs are academic achievement
per pupil and achievement in extra curricular activities such
as sport and the performing arts. School A is a high academic
achiever but is perhaps over focussed on exam performance
and neglects extra curricular activities as a result. School I
on the other hand focuses on performance at the expense of
academic subjects. DEA allows units to weight their outputs
(and inputs) favourably so that the basic model would identify
A, B, C and I as all being efficient and defining the efficient
frontier. The model would also identify FT as defining target
output levels for the underperforming school H, and C and
I are peer schools as they have the closest output mix to H
among the efficient schools. Best practice from C and I could
potentially be transferred to H to improve its performance.
Importantly, for the following discussion DEA identifies a
unique efficient frontier. This is achieved conceptually either
by finding the weights on the inputs and outputs showing a
target unit in its most favourable light or constructing a linear
combination of the other units to see if a composite unit can
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
Ac "
10 A A
w d
5 D* #F 1
5 10 _ -
_ ExCu -
Figure 2 Imprecise DEA.
Ac ''
10 A A'
5 D# p ^W C
5 10 ExCu
Figure 3 Imprecise DEA.
outperform the target unit. An example of such a composite
unit would be H' in Figure 1 .
If the output levels exhibit uncertainty and as in IDEA the
uncertainty is captured by ranges then we have the situation in
Figure 2. In the example, it is assumed that the uncertainty in
the academic measure is relatively low compared to the uncer-
tainty in the extracurricular measure. If we focus on school G
which was just underperforming in DEA, in IDEA not only
do we allow the weights to favour G, but also the values of
the outputs. To achieve that, the outputs of G are allowed to
move to their highest levels in the ranges (denoted by G') and
the outputs of the other schools forced to their lowest levels
(eg denoted by a for school A). The frontier for G is now
(a, G', i), and G is now efficient. Also with B and C taking
on their lowest values they become just inefficient against G's
frontier. In general, the additional flexibility of IDEA will
thus allow some inefficient units to become efficient.
When we focus on each school in turn, IDEA will poten-
tially produce a frontier peculiar to each unit. For example,
the efficient frontier for A is (A', c, i), see Figure 3, and
C, which regains its efficiency has a frontier of (a, C) with
I now being inefficient. Schools A, B, C, G and I will
each have their own frontier under IDEA and be efficient
whilst the other schools will all be enveloped by the frontier
(a, c, i). They will still be inefficient but have a higher effi-
ciency score than under DEA. In general, IDEA will yield
 RG Dyson and EA Shale - Data envelopment analysis, operational research and uncertainty 31
Ac -
10
F G ' ' CC (0.05)
# ' '
/ ' '
J ! '
5 10 ExC
Figure 4 Chance-constrained DEA.
a higher number of efficient units than the standard DEA
model.
The use of ranges appears equivalent to imposing a uniform
distribution on the uncertain parameters, that is from a
managerial perspective there is no further information on the
likely value within the range. It should be noted also that the
IDEA approach is effectively selecting only extreme values of
these ranges although the extent of the ranges should reflect
different degrees of uncertainty in the inputs and outputs.
IDEA is thus perhaps more akin to sensitivity analysis than
to stochastic analysis. However, it could identify the more
robust efficient units and would therefore be more helpful
for benchmarking and identifying best practice than DEA.
In CC DEA (Land et al, 1988; Olesen and Petersen, 1995;
Cooper et al, 1996) the typical assumption is that the inputs
are deterministic and the outputs are stochastic described
by a joint distribution, usually the normal distribution for
tractability. The uncertainty is handled by allowing a unit to
lie outside the efficient frontier with a pre-determined prob-
ability. The CC DEA approach converts the CC models to
deterministic equivalents which can be solved to determine
a new efficient frontier enveloping the original DEA frontier.
In Figure 4, the ellipse around unit B is an equi-probability
contour of unspecified probability level representing the joint
distribution of B. Note that, due to the convexity of the DEA
frontier, there will be a greater than a 0.5 chance of the true
value of B lying outside the efficient frontier. In CC DEA, we
wish to locate an efficient frontier such that an efficient unit
will have only a small chance 0.05 of falling outside the fron-
tier. This is illustrated in Figure 4 by the frontier labelled CC
(0.05), for example. Olesen (2006) shows how to construct CC
frontiers where there are both stochastic inputs and outputs.
The approach can soon become intractable if it is desired to
incorporate uncertainty in the inputs as well as the outputs
and if a mix of distributions is required.
Monte Carlo simulation has been advocated by a number
of researchers as a means of incorporating uncertainty in data
into DEA (Banker et al (1987) being an early example) but
almost all studies use the approach with artificially generated
data to compare techniques and/or look at the accuracy of the
generated efficiency scores. That is not our arena here - we
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
Cu I a
10 S^^5-^ a
5 10 ExC
Figure 5 Monte Carlo simulation.
are interested in using real-data to examine the impact of
uncertainty on the identification of best practice. An example
of perturbing observed data to examine the effect upon DEA
efficiency estimation can be found in Kao and Liu (2009).
This approach circumvents the intractability of many analyt-
ical methods by employing simulation where the uncertainty
in the data can be described by any discrete or continuous
distribution. As it can involve solving large numbers of DEA
models to obtain approximations to the distributions of the
efficiencies it can be computationally greedy. However, there
are no restrictions in principle on the nature of the uncer-
tainty affecting the variables. Kao and Liu (2009) advocate
the use of several years of data to construct distributions of
the uncertain variables and illustrate the advantage of this
approach over using averages. They found that in their bank,
for example, 2000 replications were sufficient to produce a
result close to the true distributions of the efficiencies. Apart
from any computational difficulties there is the concern that
the processes are unlikely to be stable over time due to
changes in technologies and learning effects, particularly if
the aim of any analysis is to discover and spread best practice
across the system. This does raise starkly the contrast between
summative and formative approaches. In a summative culture,
the best performers are rewarded and there is no incentive
(or even a disincentive) to share best practice. An alternative
to using panel data is to construct subjective distributions of
the uncertain variables as advocated in risk analysis (Hertz,
1964). In this approach, experts are interrogated about the
uncertainty in the variables providing, for example, esti-
mates of most likely, optimistic and pessimistic values.
In this way, parameters can be constructed and often a
beta distribution is assumed where the estimates indicate a
skewed distribution. This simulation approach is illustrated in
Figure 5.
The distribution of the outputs of each unit is represented
by equi-probability contours and just two are shown. We will
assume for illustrative purposes that in each replication the
output values are either at the mean or on one of the contours.
In the simulation, the frontier with each unit taking on its
mean output values will have the highest likelihood of occur-
rence and this is represented by the thickest line A, B, C and I.
32 Journal of the Operational Research Society Vol. 61, No. 1
A replication joining points on the outer contours will be the
least likely and shown by the thinnest lines. One joining a
mix of mean value and contour points will have an interme-
diate thickness. Each replication will give an estimate for the
frontier and a distribution of the efficiency of each unit will
be generated. It can be seen from the figure that the DEA
efficient unit I will often be dominated by C and in general
units deemed efficient because of their high score under the
measure with low degree of uncertainty, for example school
A, are stable in terms of their efficiency score, whereas for
units on the efficient frontier due to high scores on the measure
exhibiting high variability, for example school I, the chances
of them remaining consistently on the efficient frontier are
much reduced.
Simar and Wilson (1998) first advocated bootstrapping as
a way of capturing the uncertainty in DEA via constructing
confidence intervals about the efficiency scores. It is assumed
that there is a potentially infinite population of units corre-
sponding to a data generating process and the current set
is simply one sample from the population. There has been
development in the area of consistent estimators and we will
concentrate on these. Simar and Wilson (2008) provide an
overview. One such approach is based on repeatedly sub-
sampling the set of observations, the size of the sub-samples
being critical to adequate estimation of confidence intervals
on the efficiency scores. However, Kneip et al (2008, p. 1664)
state there is 'no reliable way of determining a reasonable
value of the sub-sample size in applied settings.' They propose
instead adopting an alternative bootstrap approach smoothing
both the distribution of the observations and the initial esti-
mate of the frontier. They argue that the two smoothing param-
eters required can be selected by logical means but the method
is considerably more complex than the original sub-sampling
approach particularly in the multi-input, multi-output case.
This is well illustrated in Simar and Wilson (2008). In order
to render the method more tractable, they suggest a simpli-
fying assumption, namely that the distribution of efficiency is
homogeneous across input-output space - a potentially very
restrictive assumption. In essence, the method generates arti-
ficial data from the observations in order to create the multiple
bootstrap samples required. Full details are given in Simar
and Wilson (2008). Constructing the symmetric confidence
interval about the DEA frontier relies upon applying the one-
sided variability observed in the bootstrapped frontiers lying
within the DEA frontier outside that frontier also. These confi-
dence intervals are illustrated in Figure 6 where the thick lines
on the rays from the origin represent the confidence intervals.
They show, for example, that the DEA inefficient unit G could
outperform C and C could outperform the efficient unit I.
This method enables the analyst to give a unit (ray) specific
estimate of the uncertainty in the DEA efficiency scores and
can identify potentially robust efficient units. However, it
cannot include any specific information on data uncertainty
in the observed values of inputs and outputs. It subsumes
any particular source of variability into a general statement
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
Ac-
io :A
; / yy/^ i '
/< ^. - - - " **
5 10 ExC
Figure 6 Bootstrapped confidence intervals.
about uncertainty. Additionally, the homogeneity assumption
on efficiency makes it impossible to deal with the very real
instance of some observations being more uncertain than
others.
Applicability of the approaches
In this section, we return to the nature of uncertainty in the
types of applications we have encountered and consider the
applicability of the various approaches. We reiterate that
we are interested in applying DEA in a formative way to
identify and spread best practice. Additionally, the units do
not necessarily conform to a single production function. The
uncertainty of interest relates to the inputs and outputs of the
units and may exist due to measurement errors, uncertainty
in external data, perceptions and proxy measures. There
is typically no data to produce frequency distributions and
the approach to capturing the uncertainty would be through
management engagement and the construction of subjec-
tive probability distributions which could be skewed. This
approach has support within the DEA literature, see, for
example, the arguments of Stewart (1996). The uncertainty
may also be unit specific rather than system wide.
First of all, we repeat our concerns about the efficacy of
SFA in our contexts. These relate to the distributional assump-
tions required, the lack of focus on the uncertainty in indi-
vidual inputs and outputs, the difficulty in handling multiple
inputs and outputs and the emphasis on a single theoretical
production function rather than one or more empirical fron-
tiers.
IDEA would be a possible approach, as it focuses on
actual units and allows uncertainty in the inputs and outputs.
Also it makes no assumptions about an underlying produc-
tion function and could accommodate the different potential
sources of uncertainty. It would therefore be consistent with
our 'spreading best practice' approach. The approach would
have two disadvantages however. Firstly, capturing the uncer-
tainty by ranges is equivalent to specifying a uniform distri-
bution with equal probabilities within the ranges and zero
elsewhere. We believe that in many situations more realistic
 RG Dyson and EA Shale - Data envelopment analysis, operational research and uncertainty 33
probability density functions can be constructed. Secondly, the
optimisation procedure will move the target unit to the most
favourable data point and the others to the least favourable so
that the approach has a second process in favour of the target
unit (the first being the choice of weights). This may be over-
compensation and will lead to a reduction in discrimination.
Note that there is now no longer a unique frontier but poten-
tially each unit has its own frontier. There will also of course
be no distribution of efficiency for each unit. However, IDEA
can nevertheless be valuable in identifying robustly efficient
units which will be superior candidates for spreading best
practice.
CC DEA focuses on the uncertainty in the inputs and
outputs although typically at a system level and again is
concerned only with the real set of units. The approach
involves specifying new frontiers enveloping the original one
and as with IDEA may identify the more robustly efficient
units as the frontiers will be closest together where the vari-
ability in efficiency is the lowest. The approach can become
intractable where there are uncertainties in inputs and outputs
and the normality assumption is relaxed. For our purpose,
it is also not clear to what extent it adds value beyond that
obtained by IDEA.
Bootstrapping produces confidence limits on the efficien-
cies of the units thus attempting to capture the true efficient
frontier within the specified interval. The units with the confi-
dence limits of highest efficiency could thus be examples of
best practice. There are however two key drawbacks from
our perspective. The sources of uncertainty are at the unit
level and inferred implicitly from the distribution of the units
in input output space, rather than addressed explicitly at the
individual inputs and outputs, and the assumptions necessary
to make the approach tractable, in particular the assumption
of homogeneity of efficiency across the input output space,
could be unrealistic. Inefficiency across the space would not
be homogeneous if operating practices within an organisation
are not equally well developed, that is mature across all the
functions captured in the model.
Having indicated the drawbacks of the previous approaches
we can now assess the potential of the Monte Carlo simulation
approach - the last, but often effective, resort of the stochastic
modeller. The approach focuses on the real set of units and
makes no underlying assumptions about a production func-
tion. In principle, it transforms the best information available
into subjective probability distributions with the uncertainty
on a particular input or output systematic across the set of
units or peculiar to an individual unit. The replications of the
simulation will lead to a frequency distribution for the effi-
cient frontier and there will be a distribution of efficiency
scores for each unit.
Thus, there will be no strict division between efficient and
inefficient units but the units with the greatest concentration
at the highest efficiency score would be candidates for best
practice. Desai et al (2005) compare CC DEA and Monte
Carlo simulation and conclude that the latter is more flex-
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
ible and computationally less cumbersome. The drawbacks or
challenges relate to constructing the distributions, but there is
a wealth of practice on that issue, and the potential computa-
tional greediness of the approach.
Conclusions and challenges
Our tentative conclusions on the various approaches to
handling uncertainty in DEA are as follows. Given our focus
on real units and best practice, with the sources of uncer-
tainty in both inputs and outputs of those units, we are not
attracted to bootstrapping, principally due to its approach
to uncertainty and to the simplifying assumptions necessary
to make the approach tractable. IDEA does focus on real
units and incorporates uncertainty on individual inputs and
outputs with the ranges being constructed subjectively. The
IDEA analysis can identify robust efficient units which may
be examples of best practice, but cannot provide informa-
tion on the distribution of efficiencies. It is computationally
feasible. CC DEA, like IDEA focuses on real units and
uncertainty in individual inputs and outputs. It could handle
subjective distributions. However, it can become intractable
where there are skewed distributions and uncertainties in
both inputs and outputs. It could identify robust efficient
units but it is not clear whether the added complexity over
IDEA gives significant added value. Monte Carlo simulation
appears to have all the flexibility necessary for the situations
with its only disadvantage being that it is computation-
ally greedy. We would thus consider IDEA as a relatively
simple and useful step in the direction of incorporating
uncertainty into DEA and Monte Carlo simulation as the
most flexible and comprehensive approach in terms of our
requirements.
Finally, in this 60th anniversary issue of JORS, we would
like to look forward by reflecting upon two more general
questions related to our foregoing discussion. What are some
of the challenges facing those interested in applying DEA but
aware of the tension between a deterministic technique and an
uncertain world? Stepping back from the detail of our partic-
ular topic and field, what generic issues have been revealed
by the investigation? Firstly, we suggest that there has been
too much emphasis on the 'research' element of OR and too
little on the 'operational'. Models and model extensions are
being published because we can develop the mathematical
and statistical tools. At the extreme this becomes a branch of
applied mathematics or perhaps not even particularly applied.
Real-world, organisational issues remain unresolved or any
resolution provided by OR practitioners goes unpublished.
We do not believe that this is unique to DEA but is a general
malaise of OR, nor is it a new problem but that does not
render it less important (Ackoff, 1979). The debate over the
definition and role of OR continues in the 21st century. WW
Cooper argued for application-driven theory in his Blackett
lecture to the Operational Research Society (Cooper, 1999).
The operational researcher applying DEA needs to engage
34 Journal of the Operational Research Society Vol. 61, No. 1
both with the operational context and the nature and source of
the uncertainties in order to realise the better in The Science
of Better'. There seem to be too many models whose sole
published use is illustrated with unrealistic data and with little
discussion of the practical significance of the development.
It would be interesting if mainstream OR journals required
from papers, upon submission, not only an indication of appli-
cability (currently often cursory in the extreme) but paired
submissions; one paper outlining the theoretical development
perhaps with a simple example and a second paper illus-
trating its use in a realistic scenario. We understand that this
might be impractical but it would certainly be a challenge
for the OR community. It might require greater collaboration
between academics and practitioners or greater involvement
of academics with practice. We do not view either of these as
undesirable developments.
References
Ackoff RL (1979). The future of operational research is past. J Opl
Res Soc 30: 93-104.
Aigner D, Lovell CAK and Schmidt P (1977). Formulation and
estimation of stochastic frontier production function models.
J Econometrics 6: 21-37.
Amado CAF and Dyson RG (2009). Exploring the Use of DEA for
Formative Evaluation in Primary Diabetes Care: An Application
to Compare English Practices. / Opl Res Soc 60: 1469-1482.
Athanassopoulos AD and Shale EA (1997). Assessing the comparative
efficiency of higher education institutions in the UK by means of
data envelopment analysis. Educ Econ 5: 1 17-134.
Banker R, Chames A, Cooper WW and Maindiratta A (1987). A
comparison of data envelopment analysis and translog estimates of
production frontiers using simulated observations from a known
technology. In: Dogramaci A and Fare R (eds). Applications of
Modern Production Theory: Efficiency and Productivity, Kluwer
Academic Publishers: Boston: MA, pp 33-55.
Bitner MJ (1990). Evaluating service encounters: the effects of
physical surroundings and employee responses. J Marketing 54:
69-82.
Blackett PMS (1950). Operational research. Opl Res Quart 1(1): 3-6.
Charnes A, Cooper WW and Rhodes E (1978). Measuring the
efficiency of decision making units. Eur J Opl Res 2: 429-444.
Charnes A, Cooper WW and Rhodes E (1981). Evaluating program
and managerial efficiency: An application of data envelopment
analysis to program follow through. Mngt Sci 27: 668-697.
Cooper WW (1999). Operational research/management science:
Where it's been. Where it should be going? The Blackett Memorial
Lecture 18 November 1997. J Opl Res Soc 50: 3-11.
Cooper WW, Huang ZM and Li S (1996). Satisficing DEA models
under chance constraints. Ann Opns Res 66: 279-295.
Cooper WW, Park KS and Yu G (1999). IDEA and AR-IDEA: Models
for dealing with imprecise data in DEA. Mngt Sci 45: 597-607.
Desai A, Ratick SJ and Schinnar AP (2005). Data envelopment
analysis with stochastic variations in the data. Soc Econ PI Sys 39:
147-164.
Dyson RG and Thanassoulis E (1988). Reducing weight flexibility in
data envelopment analysis. J Opl Res Soc 39: 563-576.
This content downloaded from 197.255.68.201 on Wed, 06 Jul 2016 17:35:29 UTC
All use subject to http://about.jstor.org/terms
Dyson RG, Camanho AS, Podinovski VV, Sarrico CS and Shale
EA (2001). Pitfalls and protocols in DEA. Eur J Opl Res 132:
245-259.
Fare R and Grosskopf S (1990). A distance function approach to price
efficiency. J Public Econ 43(1): 123-126.
Farrell MJ (1957). The measurement of production efficiency (with
discussion). J R Stat Soc Ser A 20: 253-281.
Hertz DB (1964). Risk analysis in capital investment. Harvard
Business Review 42: 95-106.
Hertz DB and Thomas H (1983). Risk Analysis and its Applications.
Wiley: Chichester.
Kao C and Liu S-T (2009). Stochastic data envelopment analysis in
measuring the efficiency of Taiwan commercial banks. Eur J Opl
Res 196: 312-322.
Kneip A, Simar L and Wilson PW (2008). Asymptotics and consistent
bootstraps for DEA estimators in nonparametric frontier models.
Econ Theory 24: 1663-1697.
Land KC, Lovell CAK and Thore S (1988). Chance-constrained data
envelopment analysis. Managerial Decision Econ 14: 541-554.
Meeusen W and van Den Broeck J (1977). Efficiency estimation
from Cobb-Douglas production functions with composed error.
International Economic Review 18: 435-444.
Office for National Statistics (2005). Census 2001 Quality Report
for England and Wales, http://www.statistics. gov.uk/downloads/
census2001/census_200 l_quality_report.pdf, accessed 17 April
2009.
Olesen OB (2006). Comparing and combining two approaches for
chance constrained DEA. J Prod Anal 26: 103-1 19.
Olesen OB and Petersen NC (1995). Chance constrained efficiency
evaluation. Mnet Sci 41: 442-457.
Sarrico CS and Dyson RG (2000). Using DEA for planning in
UK universities - An institutional perspective. J Opl Res Soc 51:
789-800.
Sarrico CS, Hogan SM, Dyson RG and Athanassopoulos AD (1997).
Data envelopment analysis and university selection. J Opl Res Soc
48: 1163-1177.
Shale EA (1996). Performance measurement of branded public
houses: An application of data envelopment analysis to the outcome
of investment decisions. MSc dissertation, University of Warwick.
Shale E A (2009). The integration of psychometric data into the
performance assessment of a retail service chain. Working Paper.
University of Warwick.
Simar L and Wilson PW (1998). Sensitivity analysis in efficiency
scores: How to bootstrap in nonparametric frontier models. Mngt
Sci 44: 49-62.
Simar L and Wilson PW (2008). Statistical inference in non-parametric
frontier models: Recent development and Perspectives. In: Fried
HO, Lovell CAK and Schmidt SS (eds). The measurement of
productive efficiency and productivity growth, Oxford University
Press: New York, p 638.
Stewart TJ (1996). Relationships between data envelopment analysis
and multicriteria decision analysis. J Opl Res Soc 47: 654-665.
Thanassoulis E, Dyson RG and Foster MJ (1987). Relative efficiency
assessments using data envelopment analysis: An application to
data on rates departments. J Opl Res Soc 38: 397-411.
Zhu J (2003). Imprecise data envelopment analysis (IDEA): A
review and improvement with an application. Eur J Opl Res 144:
513-529.
Received May 2009;
accepted September 2009 after one revision