NEWS AND VIEWS

Mismeasuring biological diversity: Response to Hoffmann and

Hoffmann (2008)
Lou Jost
Via a Runtun, Baos, Tungurahua, Ecuador
A R T I C L E D A T A A B S T R A C T
Article history:
Received 5 August 2008
Accepted 26 October 2008
Available online 26 December 2008
Common forms of reasoning about diversity in ecology and some other sciences are only
valid if diversity measures have certain mathematical properties. These properties define a
core diversity concept for these sciences. Most common diversity measures lack these
properties, so they do not support the kinds of reasoning typically applied to them. This
mismatch between diversity measures and rules of inference has caused serious errors in
many sciences. Measures which match the core concept of diversity are known in ecology as
Hill numbers, and in economics as the reciprocal of the HannahKay family of generalized
entropy concentration measures.
1. Introduction
Generalized entropy measures (Shannon entropy, the Gini
Simpson index, Renyi entropies, and others) are often equated
with diversity in biological sciences. In Entropy and diversity
(Jost, 2006) and later articles (Jost, 2007, 2008; Hardy and Jost,
2008) I argue that this practice leads to bad science many of
biologists' standard forms of reasoning about diversity are
only valid when applied to N
q
(the numbers equivalents of
generalized entropies of order q, symbolized by
q
D in my
articles). When biologists apply these standard forms of
reasoning to raw generalized entropies (as they often do),
their conclusions are frequently invalid. These other mea-
sures of compositional complexity should be distinguished
from measures of diversity.
Hoffmann and Hoffmann (2008) criticize my view, asserting
that my arguments do not uniquely specify N
q
, and that no
one concept of diversity can be called the true diversity. In a
later paper (Hoffmann, 2008) S. Hoffmann notes that virtually
all commonly used measures of diversity" are included in the
SharmaMittal family of generalized entropies. She argues
that any of these entropies can be used as diversity measures,
depending onthe mathematical needs of the application (such
as concavity, additivity, etc), and she proposes to call them all
diversities. This pluralistic viewis the dominant approach to
diversity measures in biology today, but as shown in the next
section, it can cause biologists to make serious errors.
2. Mismatches between biologists' diversity
concept and entropy measures
To illustrate the kinds of mismatches that occur when measures
other than N
q
are equated with diversity, consider the popular
GiniSimpson index
H
GS
= 1
X
S
i = 1
p
2
i
:
This is one of the most often recommended measures of
ecological diversity (e.g. Magurran, 2004). It is also the measure
most often equated with gene diversity in population genetics,
where it is called heterozygosity or simply diversity, and it is
one of Hoffman's (2008) diversities, the SharmaMittal general-
ized entropy of order 2 and degree 2. Howwell does this measure
capture the intuitive notion of diversity used by biologists?
Biologists frequently use measures of diversity to detect
changes in the environment due to pollution, climate change, or
other factors. For this application, a large drop in the diversity of
E-mail address: loujost@yahoo.com.
avai l abl e at www. sci encedi r ect . com
www. el sevi er. com/ l ocat e/ ecol econ
E C O L O G I C A L E C O N O M I C S 6 8 ( 2 0 0 9 ) 9 2 5 9 2 8
doi:10.1016/j.ecolecon.2008.10.015
the environment should cause a large drop in the value of the
diversity measure. Suppose a continent has a million equally-
commonspecies, andameteor impact kills 999,900 of the species,
leaving 100 species untouched. Any biologist, if asked, would say
that this meteor impact caused a large absolute and relative drop
indiversity. Yet H
GS
onlydecreases from0.999999to0.99, adropof
less than1%. Evidentlythe metric of this measure does not match
the intuitive concept of diversity as used by biologists, and
ecologists relying on H
GS
will often misjudge the magnitude of
ecosystem changes. This same problem arises when Shannon
entropy is equated with diversity. In contrast N
2
, the numbers
equivalent of H
GS
, drops by the intuitively appropriate 99.99%. So
does any other N
q
.
Diversity measures are also often used to guide conservation
decisions. Suppose a region has twenty forest fragments of equal
size and diversity, and suppose that no fragment shares species
with any other. Biologists might be asked how many of these
fragments should be saved in order to conserve most of the
diversity of the region. Intuitively, since each fragment contri-
butes equally to the total regional diversity, we should save at
least ten of the fragments. Yet if we equate diversity with the
GiniSimpson index, we get erroneous and self-contradictory
conservation guidance. For definiteness suppose that each
fragment has the same species abundances as the tree abun-
dances on Barro Colorado Island in Panama (Hubbell et al., 2005).
Eachfragment thenhasadiversity of 0.95, whiletheregionhasa
diversity of 0.998. This means we can preserve most of the
regional diversity by saving just one of the twenty unique
fragments, since saving one fragment conserves 95% (0.95/0.998)
of the regional diversity. Yet the pooled diversity of the 19
sacrificed fragments is 0.997, so we not only save 95% of the
regional diversity in this scenario, but we also lose 99.9% (0.997/
0.998) of the regional diversity! By focusing on one or the other of
these numbers, an unscrupulous conservation biologist could
prove whatever he or she wanted about this or any other
conservation program. This same problem arises if Shannon
entropy is equated with diversity.
In contrast the numbers equivalent N
2
gives the intuitively
reasonable, unambiguous, self-consistent answer: each island
has a diversity of 20.3 effective species, and the region has a
diversity of 406 effective species, so saving half the fragments
conserve exactly half (203 effective species) of the regional
diversity. Saving one fragment conserves 5%(1/20) of the regional
diversity, andsacrificing the other 19 fragments causes the loss of
95% (19/20) of the regional diversity. In general, when there are n
equally diverse, equally large, completely distinct communities,
N
q
of the portion saved plus N
q
of the portion destroyed add up to
the regional N
q
.
Ecologists also frequently use diversity measures to infer the
compositional similarity or differentiation among groups. They
take the ratio of within-group diversity to total diversity (the
proportionof total diversitycontainedinthe average community)
as a measure of similarity (Lande, 1996; Veech et al., 2002). If the
groups are similar incomposition, the within-group diversity will
be close tothe regional diversity, sothe ratiowill be close tounity.
If the groups are very different, biologists reasonthat the regional
diversity would be much higher than the within-group diversity,
so the ratio of within-group diversity to regional diversity will be
low. This wayof measuring compositional similarityis oftenused
with the GiniSimpson index or other entropy measures. We can
use this method to measure the compositional similarity of the
completely distinct forest fragments of the preceding example.
The ratio is 0.93/0.99, which is 0.93. This ratio is near unity,
supposedly indicating that the region is homogeneous and the
fragments are very similar to each other in composition, even
thoughthe fragments are infact maximally dissimilar toeachother
in composition (no shared species). This measure always
approaches unity when diversity is high, supposedly indicating
that the samples are nearly identical in composition, even if they
are actually completely distinct in composition (no shared
species). The many studies using the GiniSimpson index to
evaluate patterns of diversity are invalid, because the diversity
measures they use are mathematically inconsistent with their
forms of inference. The same problem arises when Shannon
entropy is equated with diversity. In contrast, if N
2
(or any other
N
q
) is equated with diversity, the similarity ratio is 1/20, which is
exactly the intuitive number when there are twenty completely
distinct, equally diverse, equally large fragments in a region.
This same ratio of mean within-group diversity to total
diversity is heavily used in population genetics, where differ-
entiation of allele frequencies between groups is one of the
principal concerns. The GiniSimpson index, usually called
heterozygosity in genetics, is the nearly universal measure of
genediversity inthisscience. Geneticdifferentiationisnormally
measuredbyF
ST
orG
ST
, thecomplement of thesimilaritymeasure
of the preceding paragraph (Nei, 1973). According to standard
genetics dogma, values of G
ST
near zero indicate low differentia-
tion among groups (e.g. Hartl and Clark, 1997, Balloux and Lugon-
Moulin, 2002). However, the problem illustrated in the preceding
paragraph makes nonsense out of this dogma. When within-
group heterozygosity is high, the value of G
ST
will approach zero,
supposedlyindicatingthat thegroups arenearlyidentical inallele
composition and frequencies, even if the groups are completely
differentiated (no shared genes)!
Thus some of the most important measures of population
structure in genetics and ecology do not really measure popula-
tion structure. The errors can approach 100%, suggesting that
populations are highly similar when they are actually completely
distinct. This has already happened in many genetic and
ecological studies; for example Balloux et al. (2000) observed
values of genetic differentiation near zero even when gene
sequencing proved that the populations under study shared no
alleles. They puzzledover the meaning of this but didnot findthe
real culprit, the use of an inappropriate measure of diversity.
Population genetics has been misled for more than 50 years
because of the mismatch between its diversity measures and its
rules of inference. Converting H
GS
to true diversity N
2
resolves
these paradoxes and leads to a deeper understanding the genetic
basis of evolutionary processes (Jost, 2008).
3. Characterizing biological diversity
Species-neutral measures of diversity per individual must
have the following properties if the standard forms of reason-
ing used in the preceding section are to be valid. (A species-
neutral measure of diversity depends only on the species'
frequencies, not on other characteristics such as their phylo-
genetic relations or their functional roles inthe ecosystem. The
926 E C O L O G I C A L E C O N O M I C S 6 8 ( 2 0 0 9 ) 9 2 5 9 2 8
qualification per individual is required because these mea-
sures are derived from the information-theoretic generalized
entropy per symbol, not per unit time or per message; see
Ricotta, 2003 for more on this distinction.)
1. Symmetry. The diversity function is symmetric in its
arguments.
2. Expandability or zero output independence. Adding a
species with zero abundance does not change the total
diversity.
3. Output transfers principle. Transferring unit abundance from
a common species to a rarer species should not decrease
diversity.
4. Homogeneity. The diversity depends only on species relative
frequencies and not on their absolute abundances.
5. Replication principle. Suppose N communities have identical
sets of species abundances, but nospecies are sharedbetween
any of the communities. All N communities necessarily have
the same diversityD(p). Suppose we pool all Ncommunities. If
the diversity measure D(x) obeys the replication principle,
then the diversity of the N pooled equally diverse, equally
large, completely distinct communities must be ND(p).
6. Normalization. If the diversity measure is applied to S equally
common species, its value is S.
Properties 14 are uncontroversial and are satisfied by all
standard species-neutral measures of compositional complexity
or entropy. Property 5, treated in Jost (2006), specifies the class of
complexity measures that match the biological concept of
diversity. If the examples in the preceding section are analyzed
carefully, it will beseenthat anymeasurewithProperty5will lead
to the intuitively correct answers, and any measure lacking this
property will not. Property 5 also ensures internal consistency, so
that when there are N equally diverse, equally large, completely
distinct communities, thediversityof anysubset of them, plusthe
diversity of the complement of the subset, equals the total
regional diversity. This property is essential when reasoning
about diversity in conservation applications.
Property 5 was first highlighted by Hill (1973) in ecology as a
prerequisite for diversity measures. Later, in economics, Hannah
and Kay (1977) eloquently argued that measures of diversity and
concentration should have this property and should therefore be
measured by N
q
. Chakravarty and Eichhorn (1991) proved that N
q
is the only function of the form 1=/
1
PS
i = 1
pi / pi
" #
which satisfies
Properties 16. (Their proof was in the context of the Hannah
Kay concentration measures, which are the reciprocals of true
diversities.) N
q
is the only known family of ecological diversity
measures to satisfy these six properties, but any other measure
satisfying them could also be considered a true diversity.
4. Converting other measures to true diversities
Distinguishing diversity from other measures of compositional
complexity has no cost, only benefits. Researchers can still take
advantage of the particular mathematical properties of any of the
standard generalized entropies, if these are needed for analyses.
The key to rigorous reasoning is to convert these other measures
to their numbers equivalents N
q
before interpreting them as
diversities. For a given system and given value of q (q1), the
numbers equivalents of all generalized entropies that are
functions of
P
S
i = 1
p
q
i
(including all SharmaMittal generalized
entropies) are identical:
N
q
=
X
S
i = 1
p
q
i
" #
1= 1q
=
q
D:
The equivalence of all generalized entropies of a given order q
extends to their within- and between-group components, as long
as the partitioning into these components is complete (the
within-group component does not contain information about
the between-group component, and vice-versa). When the
partitioning of any generalized entropy is complete, the numbers
equivalent of the within-group component times the numbers
equivalent of the between-groupcomponent equals the numbers
equivalent of theentropyof thepooledgroups. Eachof thesethree
numbers equivalents (within-group, between-group, and total)
depend only on the species frequencies and the order q, not on
the particular form of the generalized entropy used (Jost, 2007).
5. Conclusion
Hoffmann and Hoffmann rightly argue that there are other
concepts of diversity (for example those taking into account
phylogenetic differences between species) besides those satisfy-
ing Properties 16. Nevertheless many of these other concepts of
diversity may be considered generalizations or extensions of the
core diversity concept characterized by these properties. Hoff-
mann and Hoffmann write that no concept of diversity should be
calledthe true diversity. I donot want toargue about semantics.
It is not very important what name we give to the core concept of
(species-neutral) diversity per individual embodied by Properties
16. We could call it canonical diversity or mathematical
diversity or neutral diversity or anything else. The important
thing is to recognize that many of our standard rules of inference
about the biological concept of species-neutral diversity (in
particular the use of ratio comparisons) presuppose Properties
16. Biologists have made and will continue to make serious
mistakes whenever they apply these rules of inference to
measures of diversity lacking these properties. It seems helpful
tocall measures satisfying Properties 16 true diversities simply
because, whenbiologists apply their standard forms of reasoning
to such measures, they will reach valid conclusions.
Acknowledgement
I acknowledge the support from a grant by John V. Moore to
the Population Biology Foundation.
R E F E R E N C E S
Balloux, F., Lugon-Moulin, N., 2002. The estimation of population
differentiation with microsatellite markers. Molecular Ecology
11, 155165.
927 E C O L O G I C A L E C O N O M I C S 6 8 ( 2 0 0 9 ) 9 2 5 9 2 8
Balloux, F., Brunner, H., Lugon-Moulin, N., Hausser, J., Goudet, J.,
2000. Microsatellites can be misleading: an empirical and
simulation study. Evolution 54, 14141422.
Chakravarty, S., Eichhorn, W., 1991. An axiomatic characterization
of a generalized entropy index of concentration. Journal of
Productivity Analysis 2, 103112.
Hannah, L., Kay, J., 1977. Concentration in the modern industry:
theory, measurement, and the U.K. experience. MacMillan,
London.
Hardy, O., Jost, L., 2008. Interpreting and estimating measures of
community phylogenetic structuring. Journal of Ecology 96,
849852.
Hartl, D., Clark, A., 1997. Principles of Population Genetics, 3rd ed.
Sinauer Associates Inc., Sunderland, MA.
Hill, M., 1973. Diversity and evenness: A unifying notation and its
consequences. Ecology 54, 427432.
Hoffmann, S., 2008. Generalized distribution-based diversity
measurement: survey and unification. FEMM Working Paper
Series. University of Magdeburg, Germany (http://www.ww.
uni-magdeburg.de/fwwdeka/femm).
Hoffmann, S., Hoffmann, A., 2008. Is there a true diversity?
Ecological Economics 65, 213215.
Hubbell, S., Condit, R., Foster, R., 2005. Barro Colorado Forest
Census plot data. (http://ctfs.si.edu/datasets/bci).
Jost, L., 2006. Entropy and diversity. Oikos 113, 363375.
Jost, L., 2007. Partitioning diversity into independent alpha and
beta components. Ecology 88, 24272439.
Jost, L., 2008. G
ST
and its relatives do not measure differentiation.
Molecular Ecology 17, 40154026.
Lande, R., 1996. Statistics and partitioning of species diversity, and
similarity among multiple communities. Oikos 76, 513.
Magurran, A., 2004. Measuring biological diversity. Blackwell
Publishing, Oxford.
Nei, M., 1973. Analysis of gene diversity in subdivided populations.
Proceedings of the National Academy of Sciences 70,
33213323.
Ricotta, C., 2003. Parametric scaling from species relative
abundances to absolute abundances in the computation of
biological diversity: a first proposal using Shannon's
entropy. Acta Biotheoretica 51, 181188.
Veech, J., Summerville, K., Crist, T.O., Gering, J., 2002. The additive
partitioning of species diversity: recent revival of an old idea.
Oikos 99, 39.
928 E C O L O G I C A L E C O N O M I C S 6 8 ( 2 0 0 9 ) 9 2 5 9 2 8