SCIENTIFIC

REASONING
THE BAYESIAN
APPROACH
Colin Howson and Peter Urbach
THIRD EDITION
... if this [probability] calculus be condemned, then the
whole of the sciences must also be condemned.
-Henri Poincare
Our assent ought to be regulated by the
grounds of probability.
-John Locke
OPEN COURT
Chicago and La Salle, Illinois
To order books from Open Court, call toll -free 1-800-815-22110, or visit our
website at www.opencourtbooks.com.
Open Court Publi shing Company is a divisi on of Carus Publishing Company.
CopyrightC 2006 by Cani s Publishing Company
First printing 2006
All ri ghts reserved. No part of this publicati on may be reproduced. stored in a
ret ri eva l system. or transmitted, in any form or by any means. electronic. mcchani eal.
photocopyi ng. rccording. or ot herwise. without the prior written permission of the
publi sher, Open Court Publi shing Company. a division of Carus Publishing Company.
315 Fi fth Street. P.O. Box 300. Peru, Illinoi s 6 1354-0300.
Printed and bound in the l; nit ed States of America.
Library of Congress Cata loging-in-Publication Data
Howson. Colin.
Scient ific reasoning : the Bayesian approach Colin lIowson and Peter
Urbaeh.- 3rd cd.
p. cm.
Includes bibliographi ca l references (p. ) and index.
ISB'J-IJ: 9n-0-81 2(, -'}578-6 (trade pbk. : alk. paper)
IS8'J-10: 0- gI26-9578-X (t rade pbk.: alk. papcr)
I. Seicncc--Philosophy. 2. Reasoning. 3. Bayesi an statistical dccision theory.
I. Urbach, Peter. II. Titl e.
Q1 75. 11 87 2005
50 I de22
200502486S
Contents
Preface to the Third Edition Xl
Introduction
The Problem of Induction
Popper on the Problem of Inducti on 2
l ~ r Scientific Method in Practice 3
.iI Probabi li stic Induction: The Bayes ian
Approach 6
The Objectivity Ideal 9
The Pl an of This Book 10
The Probability Calculus 13
The Axioms 13
. ~
Useful Theorems of the Calculus 16 =c.,
Discussion 22
Countable Additivity 26
Random Variables 29
Distributions 30
Probability Densities 31
2.11
Expected Values 32
The Mean and Standard Deviation 33
VI CONTENTS
.1
Probabilistic Independence 35
~
Conditional Distributions 37
The Bivariate Normal 38
The Binomial Distribution 39
.n The Weak Law of Large Numbers 41
The laws of Probability 45
Prologue: Frequency-Probability 45
Measuring Uncertainty 51
Utilities and Probabilities 57
Consistency 63
The Axioms 67
The Principal Principle 76
Bayesian Probability and Inductive
Scepticism 79
3.h Updating Rules 80
The Cox-Good Argument 85
Exchangeability 88
Bayesian Induction:
Deterministic Theories 91
Bayesian Confirmation 91
Checking a Consequence 93
The Probability of the Evidence 97
The Ravens Paradox 99
The Duhem Problem 103
Good data, Bad Data, and Data Too
Good to be True 114
CONTENTS
Ad Hoc Hypotheses
Designing Experiments
Under-Determination and Prior
Probabilities
Conclusion
Classical Inference:
Significance Tests and Estimation
Falsificationism in Statistics
Fisheri an Significance Tests
Neyman-Pearson Significance Tests
S.d Significance and Inductive
Significance
Testing Composite Hypotheses
Classical Esti mation Theory
Point Estimation
Interval Estimation
Sampling
Conclusion
Statistical Inference in Practice:
VII
118
127
128
130
131
131
133
143
149
161
163
163
169
177
181
Clinical Trials 183
Clinical Trials: The Central Probl em 183
Control and Randomization 185
Signi ficance-Test Defences of
Randomization 188
6.d The Eliminative-Induction Defence of
Randomization
194
viii
Sequential Clinical Trials
Practical and Ethical Considerations
Conclusion
Regression Analysis
Simple Linear Regression
7.. The Method of Least Squares
7 Why Least Squares?
CONTENTS
197
202
203
205
205
207
209
Prediction 217
I. e Examining the Form of a Regression 220
7. 1' Conclusion 235
Bayesian Induction:
Statistical Theories 237
The Question of Subjectivity 237
{ ~
",
The Principle of Stable Estimation 245
Describing the Evidence 247
Sampling 252
Testing Causal Hypotheses 254
Conclusion 262
Finale: Some General Issues 265
The Charge of Subjectivism 265
I The Principle of Indifference 266
Invariance Considerations 273
Informationlessness 276
~ ) . a A Simplicity 288
9.13 Summary 296
CONTENTS
The Old-Evidence Problem
Conclusion
Bibliography
Index
IX
297
301
303
319
Preface to the Third Edition
How should hypotheses be evaluated, what is the role of evidence
in that process, what are the most informative experiments to per-
form? Questions such as these are ancient ones. They have been
answered in various ways, often exciting lively controversy, not
surprisingly in view of the important practical implications that
different answers carry. Our approach to these questions, which
we set out in this book, is the Bayesian one, based on the idea that
valid inductive reasoning is reasoning according to the formal
principles of probabi lity.
The Bayesian theory derives from the Memoir of the mathe-
matician and divine, Thomas Bayes, which was published posthu-
mously by his friend Richard Price in 1763. The principles set out
by Bayes had considerable influence in scientific and philosophi-
cal circles, though worries about the status of the prior probabili-
ties of scientific theories meant that the whole approach continued
to be dogged by debate. And by the 1920s, an alternative approach,
often called 'Classical', achieved dominance, due to powerful
advocacy by R. A. Fisher and many other distinguished statisti-
cians, and by Karl Popper and similarly distinguished philoso-
phers. Most of the twentieth century was dominated by the
classical approach, and in that period Bayesianism was scarcely
taught in universities, except to disparage it, and Bayesians were
widely dismissed as thoroughly misguided.
But in recent years, there has been a sea-change, a paradigm
shift. A search of the Web of Science database shows, during the
1980s, a regular trickle of around 200 articles published annually
with the word or prefix 'Bayes' in thei r titles. Suddenly, in 1991 ,
this number shot up to 600 and by 1994 exceeded 800; by 2000
it had reached almost 1,400. (Williamson and Corfield, 200 I,
p. 3). This book was one of the first to present a comprehensive,
XII PREFACE TO THE THIRD EDITION
philosophical case for the Bayesian approach to scientific rea-
soning and to show it s superiority over the classical. Its first and
second editions were published in 1989 and 1993, and from the
figures quoted it is clear that the book anticipated a powerful and
sophisticatcd resurgence of the once-dominant Bayesian
approach.
This new edition amends, updates, re-organi zes, and seeks to
make the subject more accessible. The text is intended to be sclf-
contained, calling, in the main, on only elementary mathcmatical
and statistical ideas. Nevertheless, somc parts are more complex,
and some more essential to the overall argument than others.
Accordingly, wc would suggest that readers who are not already
familiar with mathcmatical probability but who wish to gain an
initial understanding of the Bayesian approach, and to appreciate
its power, adopt the following plan of attack. First, read Chapter
I, which sets the scene, as it werc, with a brief historical
ovcrview of various approaches to scientific inference. Then,
look at section a of Chapter 2, which gives the simpl e principles
or axioms of the probability calculus, and section b, where there
are some of the probabi lity theorems that will be found useful in
the scientific context: the central theorem here is Bayes's theo-
rem in its various forms. We then suggest that the reader look at
the first few sections of Chaptcr 4, where Bayes's theorcm is
applied to some simpl e reasoning patterns that are found partic-
ularly when determini stic theorems arc handled; thi s chapter also
compares the Bayesi an approach with some others, such as
Popper's well-known falsificationist methodology. Chapters 5 to
7 deal with non-determini stic, that is, statistical hypotheses, giv-
ing a critical exposition of thc classical, or frequent ist, methods
that constitute the leadi ng alternative to the Bayesian approach;
the main classical ideas can be gleaned from sections a to d and
f and g of Chapter 5. The final part of the mini-course we are
suggesting is to examine Chapter 9, where somc of the more
widespread criticisms that have becn levell ed against the
Bayesian approach are discussed (and rejccted).
There are some marked differences between this third edition
and the preceding ones. For example, some of the objections to
the Bayesian theory we considered in the second edit ion have not
stood the test of time. There have also been changes of mind: one
PREFACE TO THE THIRD EDITION XIII
of the most prominent examples is the fact that now we accept the
strength of de Finetti's well-known arguments against countable
additivity, and have accordingly dropped it as a generally valid
principle. Other changes have been largely dictated by the desire
to make this edition more compact and thereby more accessible.
We hope that this indeed turns out to be the casco
CHAPTER 1
Introduction
1.0 The Problem of Induction
Scientific hypotheses have a general character relative to the
empirical observations they are supposed to explain, carrying
implications about phenomena and events that could not possibly
figure in any actual evidence. For instance, Mendel's genetic the-
ory concerns all inherited traits in every kind of flora and fauna,
including those that are now extinct and those that are yet to
evolve. There is therefore a logical gap between the information
derived from empirical observation and the content of typical sci-
entific theories. How then can such information give us reason-
able confidence in those theories'? This is the traditional Problem
of Induction.
One answer that has been suggested claims that our stock of
information is not in fact restricted to the empirical. A number of
philosophers have taken the view that there are certain principles
which are sufficiently rich to bridge the logical gap between
observations and theories, whose truth we are able to cognize a
priori. Kant, for example, held the proposition 'every event has a
cause' to be such a principle and he devoted much space and dif-
ficult argumentation to proving that it was indeed a priori. But
whether or not the argument is valid is beside the point, because
the principle would not solve the problem of induction anyway.
That problem is not essentially concerned with causality; and
where specifically causal theories are at issue, the question is not
whether every event has a cause, but what the particular cause or
causes of a particular observed effect are. Kant (1783, p. 9) tells
us that his "dogmatic slumber" was disturbed by Hume's brilliant
analysis of the problem of induction, yet he seems not to have
fully woken up to its significance.
2 CHAPTER 1
Another bridging principle that has been proposed is the so-
called Principle of the Uniformity of Nature, which Humc (1 777,
Section 32) summed up in the phrase "the future will resemble the
past". Some philosophers have held that whcn scientists defend
their thcorics, they are tacitly relying on this principle. But it too
cannot help with induction. The problem is that the principle is
empty, since it does not say in what particular rcspects the future
and the past are similar. And if it is to connect particular observa-
tions with a particular theory, it needs a more specific formula-
tion. For example, in ordcr to act as a bridgc between observations
that certain metals expandcd on certain occasions when they wcre
heated and the general proposition that those metals will expand
when they are heated in future, the principle nceds to be framed
in terms of those particular properties. And to infer that all met-
als expand when they are heated would require a more elaborate
formulation still. But, as Hume observed, such versions of the
Uniformity of Nature Principle are themselves general cmpiri cal
propositions, whose own claims are no less problematic than the
theories they are dcsigncd to guarantee.
l.b Popper on the Problem of Induction
It seems, then-and this is no longer controversial-that there is
no solution to the problem of induction that could demonstrate
with logical ccrtainty the truth of general scientific theori es.
Some, like Paul Fcycrabcnd, havc concludcd from the fact that no
theory is conclusivcly proved that all thcories arc therefore equal-
ly unproved, and epistemically on a par, and that the trust we com-
monly repose in science is completely irrational.
But Karl Popper, amongst others, was concerned to resist such
a sweeping scepticism, whose consequcnccs, if accepted, would
be alarming. In his attempt to defend the rationality of science and
to solve the problem of induction, he drew upon two familiar log-
ical facts. First, that while scienti fic theories cannot be deci sivcly
proved from observational evidence, observations may sometimes
refute them. Popper's strong emphasis of this possibility expl ains
why his philosophy is known as Falsificationism. The second log-
ical fact that Poppcr drew on is that deductive consequences of a
INTRODUCTI ON 3
theory can sometimes be verified through observat ion; when this
occurs, Popper said that the thcory was "corroboratcd". This ter-
minology suggests that corroboration confers some epistemic
merit on the theory, though it is not clcar what implications
Poppcr thought that had for its rational apprai sal. The predomi-
nant opinion now is that no such implications exist, for when a
particular theory is corroborated (in Popper's sense) by evidence,
so are infinitely many other theories, all ri val s to it and to each
othcr. Only one of these can bc true. But which?
Supposc, for exampl e, you were interested in thc general
law governing the coloration of swans. If the number of swans
that will ever exist is n and the number of colours is 111, then
there are m" colour combinations. This then represents a lower
limit for the number of theories conccrning the colours of
swans. If we take account of the further possibiliti es that these
birds alter their hues from time to time, and from place to place,
and that some of them are multi coloured, then it is apparent that
the number of poss ibl e theori es is immense, indeed, infinite.
The simple theory ' all swans arc white' that Popper often used
as an illustration is corroborated, as he said, by the observation
on particular occasions of white swans; but so arc infinitely
many of the othcr swan hypotheses. The question of how to sup-
port a rational preference amongst these hypotheses then
remains. And it is evident that Popper's ideas do nothing to
solve the problem of induction. I
l.c Scientific Method in Practice
Popper's idea that unrefuted but corroborated hypotheses enjoy
somc special epistemic virtue, led him to recommend that scien-
tists should seek out and give preference to such hypotheses.
There was also a descriptive aspect to this recommendation, for
Popper assumed that mainstream science is conducted more or
less as he belicved it ought to be.
I Other exampl es illustrating thc same point are given in Chapter 4. Scct ion i.
Sec also, e.g., Lakatos 1974, Sal mon 1981. and Howson 2000 fo r decisive criti-
cisms of Popper's views on induction.
4
CHAPTER 1
Popper's descriptive account does reflect two key features of
scientific reasoning. First, it sometimes happens in scientific
work that a theory is refuted by experimental findings, and when
this happens, the scientist usually recognizes that the theory is
therefore false and abandons it, perhaps re-adopting it in some
revised form. And secondly, when investigating a deterministic
theory, scientists frequently focus attention on certain of its logi-
cal consequences and then check these empirically, by means of
specially designed experiments; and if such consequences turn
out to be true, the practice is to conclude that the theory has been
confirmed and made more credible.
But Popper's methodology has very little further explanatory
power. This is for two reasons. First, it has no means of discrim-
inating between a particular theory that has been confirmed by a
successful empirical prediction and the infinity of other, conflict-
ing theories that make the same prediction. In practice, scientists
do rank such hypotheses according to their value, or credibility,
or eligibility for serious consideration- the scientific enterprise
would be impossible otherwise. Secondly, most scientific evi-
dence does not bear the logical relationship to theories that
Popper envisaged, for, more usually, such evidence is neither
implied by the theories they confirm, nor precluded by those they
disconfirm.
So, for example, many deterministic theories that appear in sci-
ence, especially the more significant ones, often have no directl y
checkable deductive consequences, and the predicti ons by which
they are tested and confirmed are necessarily drawn only with the
assistance of auxiliary theories. Newton's laws, for instance, con-
cern the forces that operate between objects in general and with the
mechanical effects of those forces. Observable predictions about
particular objects, such as the Earth's planets, can be derived only
when the laws are combined with hypotheses about the positions
and masses of the planets, the mass-di stribution of space, and so
on. But although they are not immediate logical consequences of
Newton's theory, planetary observations are standardly taken to
confirm (and sometimes disconfirm) it.
2
2 This obj ect ion to Popper's account was pressed with particular effect by
Lakatos, as we discuss in Chapter 4.
INTRODUCTION 5
Then there are scientific theories that are probabilistic, and for
that reason have no logical consequences of a verifiable character.
For example, Mendel's theory of inheritance states the probabili-
ties with which certain gene combinations occur during reproduc-
tion, but does not categorically rule out nor definitely predict any
particular genetic configuration. Nevertheless, Mendel obtained
impressive confirmation from the results of his plant trials, results
which his theory did not entail but stated to be relatively probable.
Finally, even deterministic theories may be confirmed or dis-
confirmed by evidence that is only assigned some probability.
This may arise when a theory's quantitative consequences need to
be checked with imperfect measuring instruments, subject to
experimental error. For example, the position of a planet at a cer-
tain time will be checked using a telescope whose readings are
acknowledged in experimental work not to be completely reliable,
on account of various unpredictable atmospheric conditions
affecting the path oflight to the telescope, as well as other uncon-
trollable factors, some connected with the experimenter and some
with physical vagaries. For this reason, quantitative measurements
are often reported in the form of a range of values, such as a h,
where a is the recorded reading and a - h and a + b are the bounds
of an interval in which it is judged the true value very probably
lies. This judgment is usually based on a theory giving the proba-
bilities that the instrument reading diverges by different amounts
from the true value of the measured quantity. Thus, for many
deterministic theories, what may appear to be the checking of log-
ical consequences actually involves the examination of experi-
mental effects which are predicted only with a certain probability.
Popper tried to extend his falsificationist ideas to the statisti-
cal realm, but insuperable difficulties stand in the way of any such
attempt, as we show in Chapter 5. In that chapter, we shall review
the ideas of R.A. Fisher, the eminent statistician, who was also
inspired by the idea that evidence may have a decisive negative
impact on a statistical hypothesis, akin to its falsification. He
called a statistical hypothesis under test the 'null hypothesis' and
expressed the view that
the null hypothesis is never proved or established, but is possibly dis-
proved, in the course of experimentation. Every experiment may be
6 CHAPTER 1
said to exist only in order to give the facts a chance of disproving the
null hypothesis. (1947, p. 16)
Fisher's theory of so-called 'significance tests', which prescribes
how statistical hypotheses should be tested, has drawn consider-
able criticism, and other theories have been advanced in opposi-
tion to it. Notable amongst these is the modified theory of
significance testing due to Jerzy Neyman and Egon Pearson.
Though they rejected much of Fisher's methodology, their theory
owed a good deal to his work, particularly to his technical results.
Above all, they retained the idea of bivalent statistical tests in
which evidence determines one of only two possible conclusions,
that is, the acceptance or rejection of a hypothesis.
1.d Probabilistic Induction:
The Bayesian Approach
One of the driving forces behind the development of the above-
mentioned methodologies was the desire to vanquish, and provide
an alternative to, the idea that the theories of science can be and
ought to be appraised in terms of their 'probabilities' . This ' prob-
abilistic induction' is in fact a well-established position in science
and philosophy. It has long been appreciated that scientific theo-
ries extend beyond any experimental data and hence cannot be
verified (that is, logically entailed) by them; yet while it is agreed
that absolute certainty is therefore unavailable, many scientists
believe that the explanations they think up can secure for them-
selves an epistemic status somewhere between the two extremes
of certainly right and certainly wrong and that that status depends
on the quality of the evidence and can be altered by new evidence.
This spectrum of degrees of certainty has traditionall y been
characterised as a spectrum of probabilities. For example, Henri
Poincare, the noted mathematician and physicist, asked himself
what right he had as a scientist to enunciate a theory such as
Newton's laws, when it may simply be chance that they are in
agreement with all the available evidence. How can we know that
the laws will not break down entirely the next time they are test-
ed? "To this objection the only answer we can give is: It is very
INTRODUCTI ON 7
improbable" (1905, p. 186). Poincare believed (pp. 183-84) that
"the physicist is often in the same position as the gambler who
reckons up his chances. Every time that he reasons by induction,
he more or less consciously requires the calculus of probabilities."
And summing up hi s approach, Poincare remarks (p. 186): "if this
calculus be condemned, then the whole of the sciences must also
be condemned."
Similarly, the philosopher and economist W. S. Jevons:
Our inferences . . . always retain more or less of a hypothetical char-
acter, and are so far open to doubt. Only in proportion as our induc-
tion approximates to the character of perfect induction does it
approximate to certainty. The amount of uncertainty corresponds to
the probability that other objects than those examined may exist and
falsify our inferences; the amount of probabili ty corresponds to the
amount of informati on yielded by our examination; and the theory of
probability will be needed to prevent us from over-estimating or
under-estimating the knowledge we possess. ( 1874, Volume 1, p.
263)
Many scientists have voiced the same idea, namely, that theories
have to be judged in terms of their probabiliti es in the light of the
evidence. Here are two quotations from Einstein which explicitly
manifest a probabili stic view:
I knew that the constancy of the velocity of li ght was something
quite independent of the relati vity postulate and I weighted which
was the more probable. (From a letter quoted in Stachel 1998)
and
Herr Kaufmann has determined the relation [between electric and
magnetic defl ection] of [3-rays with admirabl e care ... Using an
independent method, Herr Planck obtained results which fully agree
[with the computati ons of] Kaufmann ... it is further to be noted that
the theories of Abraham and Bucherer yield curves which fit the
observed curve considerably better than the curve obtained from rel-
ativity theory. However, in my opinion, these theori es should be
ascribed a rather small probability because their basic postulates
concerning the mass of the moving electron are not made plausible
by theoretical systems which encompass wider complexes of phe-
nomena. (Quoted in Pais 1982, p. 159)
8
CHAPTER 1
Einstein is here using a basic probabilistic idea, that a very high
likelihood (given by the closeness of fit to the data) can combine
with a small enough prior probability to give an overall small pos-
terior probability. In fact, as Jon Dorling (1979, p. 180) observed,
it is rare to find any leading scientist writing in, say, the last three
hundred years who did not employ notions of probability when
advocating his own ideas or reviewing those of others.
Philosophers from James Bernoulli in the seventeenth century
to Rudolf Carnap, Harold Jeffreys, Bruno de Finetti, Frank
Ramsey and E. T. Jaynes in the twentieth century have attempted
to explicate these intuitive notions of inductive probability. There
have been two main strands in this programme. The first regards
the probabilities of theories as objective, in the sense of being
determined by logic alone, independent of our subjective attitudes
towards them. The hope was that a way could be found to ascer-
tain by logical anal ysis alone the probability that any given theo-
ry is true, and so allow comparative evaluations of competing
theories to be placed on an objective footing. This would largely
solve the problem of induction and establish an objective and
rational basis for science. But, in fact , it is now generally
acknowledged that no one has succeeded in this approach, and
that objections that have been made against it have proved crip-
pling and unanswerable.
The other strand in the programme to explicate inductive prob-
ability treats the probability of a theory as a property of our attitude
towards it; such probabilities are then interpreted, roughly speaking,
as measuring degrees of belief. This is called the subjectivist or per-
sonalis! interpretation. The scientific methodology based on this
idea is usually referred to as the methodology of Bayesianism,
because of the prominent role it reserves for a famous result of the
probability calculus known as Bayes's theorem.
Bayesianism has experienced a strong revival in recent years,
due in part to its intrinsic plausibility and in part to the weakness-
es which have gradually been exposed in the standard methodolo-
gies. It is fair to claim that we are in the midst of a revolution, in
which Bayesianism is becoming the new paradigm for inductive
inference. [n the chapters to come, we shall present a detailed
account and defence of the Bayesian methodology and will show
how it illuminates the various aspects of scientific reasoning.
INTRODUCTION 9
1.e I The Objectivity Ideal
The sharpest and most persistent objection to the Bayesian
approach is directed at one of its defining features, namely that it
allows certain subjective factors a role in the appraisal of scientif-
ic theories. Our reply to this objection will be that the element of
subjectivity is, first of all, minimal and, secondly, exactly right.
Such a response contradicts an influential school of thought that
denies that any subjectivity at all should be admitted in theory-
appraisal; such appraisal, according to that school, should be
completely objective. Lakatos (1978, Volume 1, p. 1) expressed
this objectivist ideal in uncompromising style, thus:
The cognitive value of a theory has nothing to do with its psycholog-
ical influence on people's minds. Belief, commitment, understand-
ing are states of the human mind. But the objective, scientific value
of a theory is independent of the human mind which creates it or
understands it, its scientific value depends only on what objective
support these conjectures have infacts.
3
It was the ambition of Popper, Lakatos, Fisher, Neyman and
Pearson, and others of their schools to develop this idea of a cri-
terion of scientific merit, which is both objective and com-
pelling, and yet non-probabilistic. And it is fair to say that their
methodologies, especially those connected with significance
testing and estimation, which comprise the bulk of so-called
Classical methods of statistical inference, have achieved pre-
eminence in the field and become standards of correctness with
many scientists.
In the ensuing chapters, we shall show that these classical
methods are in fact intellectually quite indefensible and do not
deserve their social success. Indeed, we shall argue that the ideal
of total objectivity is unattainable and that classical methods,
which pose as guardians of that ideal, actually violate it at every
turn; virtually none of those methods can be applied without a
generous helping of personal judgment and arbitrary assumption.
3 These characteristic italics were edited out of the posthumously published ver-
sion of Lakatos's original mimeographed paper.
10
CHAPTER 1
1.f I The Plan of This Book
The thesis we propound in this book is that scientific reasoning is
reasoning in accordance with the calculus of probabilities.
In Chapter 2 we introduce that calculus, along with the princi-
pal theorems that will later serve in an explanatory role with
respect to scientific method; and we shall also in that chapter
introduce the notion of a probability density, and describe some of
the main probability distributions and associated theorems that
will be needed later on.
We shall then, in Chapter 3, consider different interpretations
of the probability calculus and introduce the notion of degrees of
belief.
In Chapter 4 we examine how scientific theories may be con-
firmed or disconfirmed and look at characteristic pattcrns of
inductive reasoning, particularly in relation to deterministic theo-
ries; we argue that these patterns are best understood as arguments
in probability and that non-Bayesian approachcs, by comparison,
offer littl e or no insight into them.
Then, in Chapter 5, we turn to statistical hypotheses, where
the philosophy of scientific inference has mostly been the pre-
serve of statisticians. Far and away the most in fl uential voice in
statistics in recent times has been that of the classical statisti-
cian, and we shall therefore first give an account of the classical
viewpoint and of its expression in the theories of significance
tests and estimation.
In Chapter 6, we examine the practical areas of agricultural
and clinical tri als and argue that restrictions placed on such trials
by classical principles of inference are largely unjustified and
often harmful to their efficient and ethical conduct.
Another practical area where class ical principles have been
highly influential is in the study of the regression, or correlation
of one physical parameter on others, and in Chapter 7 we argue
that standard classical methods of regression analysis are also
misconceived and harmful to scientific advancc.
In Chapter 8, we outline the Bayesian approach to statistical
inference and show its merits over its chief rivals. And finally, in
the last chapter, we consider the commonest objections brought
INTRODUCTION \I
against the Bayesian theory, principally the complaint that the
subjectivi sm implicit in it is at vari ance with any proper under-
standing of the scientific process and of inductive inference. Thi s,
and the other objections we show are baseless.
CHAPTER 2
The Probability Calculus
2.0 The Axioms
The rules governing the assignment of probabilities, together with
all the deductive consequences of those rules, are collectively
called the probability calculus. Formally, the rules, or axioms, of
the probability calculus assign non-negative real numbers (the
probabilities), from among those between 0 and 1 inclusive, to a
class of possible states of affairs, where these are represented under
some appropriate manner of description. For the time being all that
we shall assume about this class of representations, called the
domain of discourse, or domain for short, is that it is closed under
conjoining any two items with 'and', them with 'or', and
negating any single item with 'not'. Thus if a and b represent pos-
sible states of affairs, so do respectively 'a and b', symbolised a &
b; 'a or b', symbolised a v b; and 'not-a', symbolised -a.
We shall allow for a certain amount of redundancy in the way
the members of this possibility structure are characterised, just as
we do in ordinary discourse. For example, '--a' is just another,
morc complicated, way of saying a, and a and --a are logically
equivalent. In general, if a and b are logically equivalent repre-
sentations of any possible state we shall symbolise the fact by the
notation a b. It is useful (actually indispensable in the devel-
opment of the formal theory) to consider as limiting cases those
possible states of affairs which must necessarily occur, such as
the state of its either raining or not raining, and those which nec-
essarily cannot occur, such as its simultaneously raining and not
raining (in a particular place). The symbolism a v -a represents
a necessary truth, and is itself is called a logical truth, while a &
-a represents a neccssary falsehood, and is called a logicalfalse-
hood, or contradiction. In what follows, t will be the generic
14 CHAPTER 2
symbol of a logical truth and ..l that of a contradiction. To any
reader who has had exposure to an elementary logic course these
concepts and the notation will be familiar as the formal basics of
a propositi onal language, and for that reason we shall call these
items, a, b, c, .. . and the compounds we can form from thcm,
using the operat ions -, v and &, propositions. Thc ' propositi on'
terminology is not ideal, but thcrc is no better general-purpose
term around to refer to classes of possibl e states of affairs, be they
locali sed in spacet ime or larger-scal e types of possible world.
A word to the wise, that is, to those who have at some point
consultcd textbooks of probability, clcmentary or advanced.
These texts frequently start off by defining a probabilitl'-c\ystem,
which is a tripl e (S, :S, P) , where P is a non-negative, real-va lued
function on ~ whi ch is called a field of subsets of S, where the
latter is called variously the class oj' elementary events, sample-
space or possibili(v .\pace. That . ~ . is a fie ld of subsets of S means
that it contains S itself, and is closed under the set-theoretic oper-
ations of complementation vvith respect to S, union and intersec-
tion. It follows that . ~ . contains 0, thc cmpty set, since this is the
complement of S with respect to itself. We can relate this to our
own rather (in fact deliberatcly) informal treatment as follows. 2i
corresponds to our domain of propositi ons (referring to a class of
possi bl e states of affairs hcre represent ed by S), with negation rep-
resented by re lative complement, conjuncti on by intersection, and
disjuncti on by union. The only signi f icant difference is that the
set-theoretic for mali sm is purel y extensional: there is no room for
equival ent yet distinct descriptions of the same events in S. Thus,
for example, S is the singlc extension of all the logicall y true
propositions like a v -a, -(a & -a), and so forth), and 0 the sin-
gle extension of all logical falsehoods. By writing t and ..l as
generic logical truths and falsehoods wc arc in effect pcrforming
notationall y the same collapsing operati on as is achieved by going
set -theoretical.
A word to the very wise. Sometimes thc probability function
is said to be defined on a Boolean algebra, or algebra for short. A
celebratcd mathcmatical result li es bchind thi s terminology,
namel y Stone's Theorem that every Boolean algebra is isomor-
phic to a field of sets. Thus we ean talk of an algebra of scts,
implicitly referring to the unique algebra isomorphic to the given
THE PROBABILITY CALCULUS I S
field. Also, the propositional operations of conjunction and di s-
junction arc often symbolised using the Boolean-algebraic sym-
bols for meet and join, 1\ and v. The reason for this is that if we
identify logicall y equivalent elements of a propositional language
we also obtain a Boolean algebra, the so-call ed Lindenbaum (zlge-
bra of the language. Sometimes, for this reason, people speak of
an algebra of propositions. Strictly speaki ng, however, the ele-
ments of a propositional language are not isomorphic to a
Boolean algebra, merely homomorphic, because the mapping is
only many-one from the propositions to corresponding elements
of the algebra (all logical truths map to the unique maximal ele-
ment I of the algebra, and all logical falsehoods map to the unique
least element 0, and in general all equivalent propositions map to
the same member of the algebra; the reader mi ght like to check
that the algebra determined by one propos iti onal variable has four
members, that generated by two has sixteen, and that generated by
11 has 2 raised to the power 211 members).
So much for peripheral technicaliti es. In what follows we shall
regard probabiliti es as defined on domains of propositions closed
under negat ion, conjunction, and disjunction, with the probability
function on a particular domain denoted by P, and P(a) read as 'the
probability of a' . Thi s brings us to the question of what P(a) actu-
ally means. A remarkable fact about the probability calculus, dis-
covered two hundred years ago, is that such statements can be
endowed with two quite distinct types of meaning. One refers to
the way the world is structured, and in parti cul ar the way it appears
to endow certain types of stochastic (chance-like or random)
experiment with a disposition to deliver outcomes in ways which
betray marked large-scale regularities. Herc the probabilities are
objective, numeri cal measures of these regul arities, evaluated
empirically by the long-run relative frequencies of the correspon-
ding outcomes. On the alternative interpretati on the meaning of
P(a) is epislemic in character, and indi cates something like the
degree to whi ch it is felt some assumed body of background
knowl edge renders the truth ofa more or less likely, where a mi ght
be anything from a prediction about the next toss of a particul ar
coin to a statement of the theory of General Relativity. These sens-
es of Pray are not entirely unrelated. Knowing the objective prob-
ability of getting heads with a particul ar coi n should, it seems
16 CHAPTER 2
reasonable to believe, also tell you how likely it is that the next
toss of the coin wi II yield a head.
We shall investigate these interpretative issues in more detail
later. The task now is to get a feel for the formal principl es of the
probability calculus, and in particular see what the fundamental
postulates are and discover some useful consequences of them.
The fundament al postulates, known as the probability axioms, are
just four in number:
(1) P(a) 0 for all a in the domain of P
(2) P(t) = I.
(3) P(a v b) = pray + P(b) if a and b are mutually inconsis-
tent; that is, if a & b ..l.
(1)- (3) above suffice to generate that part of the probability cal-
culus dealing with so-called absolute or unconditional, probabil-
ities. But a good deal of what follows will be concerned with
probability functions of two variables, unlike P above which is a
function of only one. These two-place probability functions are
called conditional probabilities, and the conditional probability of
a given b is written P(a / b). There is a systematic connection
between conditional and unconditional probabilities, however,
and it is exprcssed in our fourth axiom:
P(a & b)
(4) P(a/b) = where P(b) ;z' O.
P(b)
Many authors take P(a I b) actually to be defined by (4). We prefer
to regard (4) as a postulate on a par with (1)- (3). The reason for this
is that in some interpretations of the calculus, independent mean-
ings are given to conditional and unconditional probabilities, which
means that in those (4) cannot be true simply by definition.
2.b Useful Theorems of the Calculus
The first result states the well-known fact that the probability of a
proposition and that of its negation sum to I :
THE PROBABILITY CALCULUS 17
(5) P(-a) = I - P(a)
Proof:
a entails -- a. Hence by (3) P(a v a) = P(a) + P(-a). But by
(2) P(a v -a) = I, whence (5).
Next, it is simple to show that contradictions have zero probability:
(6) P(.l) = O.
Proof:
-.l is a logical truth. Hence P(-.l) = 1 and by (5) P(.l) = O.
Our next result states that equivalent sentences have the same
probability:
(7) If a ? b then P(a) = P(b) .
Proof:
First, note that a v -b is a logical truth if a ? b.
Assume that a ? b. Then P(a v -b) = I . Also if a ? b then a
entails -- b so P(a v - h) = P(a) + P(-b).
But by (5) P(-b) = I - P(h) , whence P(a) = P(h).
We can now prove the important property of probability func-
tions that they respect the entailment relation; to be precise, the
probability of any consequence of a is at least as great as that of
a itself:
(8) If a entails b then pro) :s; P(b).
Proof:
If 0 entails b then [a v (h & -a)] ? b. Hence by (7) P(b) =
pra v (b & -a)}. But a entails -(b & -a) and so pra v (b &
-a)] = pray + P(b &-a). Hence P(h) = P(a) + P(b &-a). But
by (1) P(b & -a) 2: 0, and so P(a) :s; P(h).
From (8) it follows that probabilities are numbers between 0 and
I inclusive:
18
(9) $ P(a) $ I, for all a in the domain of P.
Proof:
CHAPTER 2
By axiom I , P(a) ~ 0, and since a entails t, where t is a logi-
cal truth, we have by (8) that P(a) ~ P(t) = I.
We shall now demonstrate the general (finite) additivity condition:
(10) Supposc a
i
entails -ai' where I $ i <j $ n. Then P(a I v
... va,) = P((lI) + ... + P(a
n
).
Proof:
P(a
l
v ... va,, ) = P[(a
l
v ... van I) va), assuming that n >
L if not the result is obviously trivial. But since a. entails -a,
I I
for all i ~ j. it follows that (a 1 v . .. v an 1) entails -a
l1
, and
hence P(a 1 v ... va,) = P(a I v ... v a" I) + P(a,). Now sim-
ply repeat this for the remaining ai' .. . ,a
l1
I and we have (10) .
(This is essentially a proof by mathematical induction.)
Corol/arv. If (II v . . . v a is a logical truth, and (I. entails -a.
1/ [I
for i ~ j then I = P(a,) + ... + P(a,). .
Our next result is often called the 'theorem of total probability' .
(11) If P((li v ... va) = I, and a. entails -a. for i ~ j. then
11 I .I
P(b) = P(b & a
l
) + ... + P(b & a,). for any proposition
b.
Proof:
h entails (b & (1 , ) v ... v (b & an) v [b & -(a, v ... va,)}.
Furthermore, all the disjuncts on the right-hand side arc mutu-
ally exclusive. Let a = (/1 V ... v a". Hence by (10) we have that
P(h) = P(b & a
l
) + ... + P(b & a,) + P(h & -a). But P(h &
-0) $ P(-a). by (8), and P(-a) = I - P(a) = 1 - 1 = 0. Hence
P(b & -a) = 0 and (11) follows.
Coro/fan) I. Ifa
l
v ... v a is a logrical truth, and a. entails -a.
~ 1/ I J
for i ;z!j, then P(b) = 'LP(b & (Ii)'
Corollary 2. P(h) = P(b I e) Pre) + P(h I -c) P(-c), for any c
such that pre) > 0.
THE PROBABILITY CALCULUS 19
Another useful consequence of (11) is the following:
(12) If P(a
j
v ... va) = I and a. entails ~ a for i '" J', and
II I .I .
P(a) > 0, then for any b, P(b) = P(b I aj)P(a
j
) + ... +
P(b I a) P(a,).
Proof:
A direct application of (4) to (11).
(12) itself can be generalized to:
IfP(a
j
v ... va) = I and P(a & a) = for all i =J', and P(a)
II 'I I' . f
> 0, then for any b, P(b) = P(bl a
j
) P(a
j
) + ... + P(b I
a,)P(a
n
).
We shall now develop some of the important properties of the
function P(a I 17). We start by letting b be some fixed proposition
such that P(b) > and defining the function Q(a) of one variable
to be equal to P(a I b), for all a.
Now define 'a is a logical truth modulo b' simply to mean '17
entails a' (for then a and t are equivalent given b), and 'a and e
are exclusive modulo b' to mean '17 & a entails -e'; then
(13) Q(a) = 1 if a is a logical truth modulo b; and the corol-
lary
(14) Q(h) = I;
(15) Q(a v c) = Q(a) + Q(e) , if a and e are exclusive modulo
b.
Now let Q'(a) = P(a I c), where pre) > 0; in other words, Q' is
obtained from P by fixing e as the conditioning statement, just as
Q was obtained by fixing h. Since Q and Q' are probability func-
tions on the same domain, we shall assume that axiom 4 also
Q(a & d)
holds for them: that is, Q(a I d) = ,where Q(d) > 0, and
Q(d)
similarly for Q '. We can now state an interesting and important
invariance result:
20
(16) Q(a I c) = Q' (a I b).
Proof:
Q(a & c) P(a & b I c)
Q(a & c) = . Q(c) P(c I b)
P(a&blc) Q'(a&b)
---- = Q(a I b).
P(b I c) Q' (b)
CHAPTER 2
P(a & b & c)
P(b & c)
Corollary. Q(a I c) = P(a I b & c) = Q'(a I b).
(16) and its corollary say that successively conditioning P on b
and then on c gives the same result as if P were conditioned first
on c and then on b, and the same result as if P were simultaneously
conditioned on b & c.
(17) If h entails e and P(h) > 0 and Pre) < 1, then P(h I e) >
P(h).
This is a very easy result to prove (we leave it as an exercise), but it
is of fundamental importance to the interpretation of the probabili-
ty calculus as a logic of inductive inference. It is for this reason that
we employ the letters hand e; in the inductive interpretation of
probability h will be some hypothesis and e some evidence. (17)
then states that {f"h predicts e then the occurrence ole will, if the
conditions of" (17) are satisfied, raise the probability C?lh.
(17) is just one of the results that exhibit the truly inductive
nature of probabili stic reasoning. It is not the only one, and more
celebrated are those that go under the name of Bayes :So Theorems.
These theorems are named after the eighteenth-century English
clergyman Thomas Bayes. Although Bayes, in a posthumously
published and justly celebrated Memoir to the Royal Society of
London (1763), derived the first form of the theorem named after
him, the second is due to the great French mathematician Laplace.
Bayes's Theorem (First Form)
pre I h) P(h)
(18) P(h I e) = - - - - p ~ T . where P(h) , pre) > O.
THE PROBABILITY CALCULUS 21
Proof:
P(h & e
l
P(h I e) = /
Pre)
p re I h) P(h)
_ ._. __. - --
Pre)
Again we use the letters hand e, standing for hypothesis and evi-
dence. This form of Bayes's Theorem states that the probability of
the hypothes is conditional on the evidence (or the posterior prob-
ability of the hypothesis) is equal to the probability of the data
conditional on the hypothesis (or the likelihood of the hypothesis)
times the probability (the so-called prior probability) of the
hypothesis, all divided by the probability of the data.
Bayes's Theorem (Second Form)
(19) i/P(h
l
V ... v hJ = I
P(h), Pre) > 0 then
and h . entails i()r i ;" J' and
, J.I'.
pre I hk) P(h
k
)
P(h
k
I e) = "LP(e Ih,) P(h)
Coro/tm:v. If hi v ... V hl/ is a logical truth, then if Pre), P(h)
> 0 and h entail s for i or ,', then
/ .
pre I hk) P(h
i
)
P(hk I e) = "LP(e I h)
, ,
Bayes's Theorem (Third Form)
Plh)
(20) P(h 1 e) = P(h) +
pre 1 h)
From the point of view of induct ive inference, this is one of
the most important forms of Bayes's Theorem. For, since =
I - P(h). it says that P(h I e) =f{P(h) , pre I -h)) wherefis an in-
\ pre I h)
creasing function of the prior probability P(h) of h and a decreas-
ing function of the likelihood ratio pre 1 -h) . In other words, for
pre 1 h)
22 CHAPTER 2
a given value of thc likelihood ratio, the posterior probability of h
increases with its prior, while for a givcn value of the prior, the
posterior probabi I ity of h is the greatcr, the less probable e is rel-
ative to ~ h than to h.
2.c Discussion
Despite their scemingly abstract appearance, implicit in axioms
1 ) ~ 4 ) is some very interesting, significant and sometimes sur-
prising information, and a good deal of this book will be taken up
with making it explicit and explaining why it is significant.
To whet the appctite, consider the following apparently simple
problem, known as the Harvard Medical School Test (Casscells,
Schoenberger, and Grayboys 1978), so called because it was given
as a problem to students and staff at Harvard Medical School,
whose rcsponses we shall come to shortly. I A diagnostic test for a
disease, D, has two outcomes 'positive' and 'negative' (supposed-
ly indicating the presence and absence of D respectively). The test
is a fairly sensitive one: its chance of giving a false negative out-
come (showing 'negative' when thc subject has D) is cqual to 0,
and its chance of giving a false positivc outcome (showing 'posi-
tive' when the subject does not have D) is small : let us supposc it
is equal to YYo. Suppose the incidence of thc disease is very low,
say one in onc thousand in the population. A randomly selected
person is given the test and shows a positive outcome. What is the
chance they have D?
One might reason intuitively as follows. They have testcd pos-
itive. The chance of testing positive and not having D would bc
only one in twcnty. So the chance of having D given a positive
result should be around ninetcen twcntieths, that is, 95%. This is
the answer given by the majority of thc respondents too. It is
wrong; very wrong in fact: the correct answer is less than two in
one hundred! Let us see why.
Firstly, anyone who answered 95% should have been suspi-
cious that a piece of information given in the problem was not
used, namely the incidence of D in the population. In fact, that
information is highly relevant, because the correct calculation
I The discussion here fo ll ows Howson 2000, Chapter 3.
THE PROBABILITY CALCULUS 23
cannot be performed without it, as we now show. We can repre-
sent the false negative and false positive chances formally as con-
ditional probabilities P(-e I h) = 0 and pre I -h) = 0.05
respectively, where h is 'the subject has D' and e is 'the outcome
is positive'. This means that our target probability, the chance that
the subject has D given that they tested positive, is P(h I e), which
we have to evaluate. Since the subject was chosen randomly it
seems reasonable to equate P(h), the absolute probability of them
having D, to 0.00 I, the incidence of D in the population. By (5)
in section b we infer that pre I h) = I, and that P(-h) = 0.999. We
can now plug these numbers into Bayes's Theorem in the form
(20) in b, and with a little arithmetic we deduce that P(h I e) =
0.0196, that is, slightly less than 2%.
Gigerenzer (1991) has argued that the correct answer is more
naturally and easily found from the data of the problem by trans-
lating the fractional chances into whole-number frequencies with-
in some actual population of 1,000 people in which one individual
has D, and that the diagnosis of why most people initially get the
wrong answer, like the Harvard respondents, is due to the fact that
the data would originally have been obtained in the form of such
frequencies, and then been processed into chance or probability
language which the human mind finds unfamiliar and unintuitive.
Thus, in the Gigerenzer-prescribed format, we are looking to find
the frequency of D-sufferers in the subpopulation of those who
test positive. Well, since the false negative rate is zero, the one
person having D should test positive, while the false negative rate
implies that, to the nearest whole number, 50 of the 999 who don't
have D will also test positive. Hence 51 test positive in total , of
whom I by assumption has D. Hence the correct answer is now
easily seen to be approximately I in 51 , without the dubious aid
of recondite and unintelligible formulas .
Caveat empfor! 2 When something is more difficult than it
apparently needs to be, there is usually some good reason, and
there is a compelling reason why the Gigerenzer mode of reason-
ing is not to be recommended: it is invalid! As we shall see later,
there is no direct connection between frequencies in finite sam-
ples and probabilities. One cannot infer directly anything about
2 Buyer beware!
24 CHAPTER 2
frequencies in finite samples from statements about a probability
distribution, nor, conversely, can one infer anything directly about
the latter from frequencies in finite samples. Tn particular, one is
certainly not justifi ed in translating a 5% chance of e conditional
on ~ into the statement that in a sample of 999, 50 will test pos-
itive, and even less can one, say, translate a zero chance of e con-
ditional on h into the statement that a single indi vidual with D will
test positive. As we shall also see later, the most that can be assert-
ed is that with a high probability in a big enough sample the
observed frequency wi II lie within a given neighbourhood of the
chance. How we compute those neighbourhoods is the task of sta-
ti stics, and we shall discuss it again in Chapters 5 and 8.
It is instructive to reflect a little on the significance of the
probability-calculus computation we have just performed. It
shows that the criteria of low false-positive and fal se-negative
rates by themselves tell you nothing about how reliable a positive
outcome is in any given case: an additi onal piece of information
is required, namely the incidence of the di sease in the population.
The background incidence also goes by the name of 'the base
rate', and thinking that valid inferences can be drawn just from the
knowledge of false positive and negative rates has come to be
called the ' base-rate fallacy'. As we see, if the base-rate is suffi-
ciently low, a positive outcome in the Harvard Test is consistent
with a very small chance of the subject having the disease, a fact
which has profound practical implications: think of costl y and
possibly unpleasant follow-up investigations being recommended
after a positive result for some very rare di sease. The Harvard Test
is nevertheless a chall enge to the average person's intuition, which
is actually rather poor when it come to even quite elementary sta-
tistical thinking. Translating into frequency-l anguage, we see that
even if it can be guaranteed that the null hypothesis (that the sub-
ject does not have the disease) will be rej ected only very infre-
quently on the basis of an incorrect (positive) result, this is
nevertheless consistent with almost all those rejections being
incorrect, a fact that is intuitively rather surpri sing-which is of
course why the base-rate fallacy is so entrenched.
But there is another, more profound, lesson to be drawn. We
said that there are two quite distinct types of probability, both
obeying the same formal laws (1)-(4) above, one having to do
THE PROBABILITY CALCULUS 25
with the tendency, or objective probability, of some procedure to
produce any given outcome at any given trial, and the other with
our uncertainty about unknown truth-values, and which we called
epistemic probability, since it is to do with our knowledge, or lack
of it. Since both these interpretations obey the same formal laws
(we shall prove this later), it follows that every formally valid
argument involving one translates into aformally valid argument
involving the other.
This fact is of profound significance. Suppose hand e in the
Harvard Test calculation had denoted some scientific theory
under scrutiny and a piece of experimental evidence respectively,
and that the probability function P is of the epistemic variety
denoting something we can call 'degree of certainty'. We can
infer that even if e had been generated by an experiment in which
e is predicted by h but every unlikely were h to be false, that
would still by itse(j'give us no warrant to conclude anything about
the degree of certainty we are entitled to repose in h
3
To do that
we need to plug in a value for P(h) , the prior probability of h. That
does not means that you have to be able to compute P(h) accord-
ing to some uniform recipe; it merely means that in general you
cannot make an inference ending with a value for P(h I e) without
putting some value on P(h) , or at any rate restricting it within cer-
tain bounds (though this is not always true, especially where there
is a lot of experimental data where, as we shall see, the posterior
probability can become almost independent of the prior).
The lessons of the Harvard Medical School Test now have a
much more general methodological applicability. The results can be
important and striking. Here are two examples. The first concerns
what has been a major tool of statistical inference, significance
] That it does is implicit in the so-called Neyman-Pearson theory of statistical
testing which we shall discuss later in some detail. And compare Mayo: if e 'fits'
h [is to be expected on the basis of h] and there is a very small chance that the
test procedure ' would yield so good a fit if h is false', then 'e should be taken as
good grounds for h to the extent that h has passed a severe test with e' (1996,
p.I77; we have changed her upper case e and h to lower case). Mayo responds to
the Harvard Medi cal School Test example in Mayo 1977, but at no point docs
she explain satisfactorily how obtaining an outcome which gives one less than a
2% chance of having the disease can possibly constitute ' good grounds' for the
hypothesis that one has it.
26 CHAPTER 2
testing, a topic we shall discuss in detail in Chapter 5. A Neyman-
Pearson significance test is a type of so-called likelihood ratio
test, where a region in the range of a test variable is deemed a
rejection region depending on the value of a likelihood ratio on
the boundary. This is determined in such a way that the probabil-
ities of (a) the hypothesis being rejected if it is true, and (b) its
being accepted if it is false, are kept to a minimum (the extent to
which this is achievable will be discussed in Chapter 5). But these
probabilities (strictly, probability-densities, but that does not
affect the point) are, in effect, just the chances of a false negative
and a false positive, and as we saw so graphically in the Harvard
Medical School Test, finding an outcome in such a region conveys
no information whatever by Uselfabout the chance of the hypoth-
esis under test being true.
The second exampl e concerns the grand-sounding topic oLvei-
entitie realism, the doctrine that we are justified in inferring to at
least the approximate truth of a scientific theory r if certain con-
ditions are met. These conditions are that the experimental data
are exceptionally unlikely to have been observed if r is false, but
quite likely if it is true. The argument, the so-called No Miracles
argument, for the inference to the approximate truth of r is that if
T is not approximately true then the agreement between r and the
data are too miracul ous to be due to chance (the use of the word
'miraculous', whence the name of the argument, was due to
Putnam 1975). Again, we see essentially the same fallacious infer-
ence based on a small false positive rate and a small false nega-
tive rate as was committed by the respondents to the Harvard Test.
However much we want to believe in the approximate truth of the-
ories like quantum electrodynamics or General Relativity, both of
which produce to order predictions correct to better than one part
in a billion, the No Miracles argument is not the argument to jus-
tify such belief (a more extended discussion is in Howson 2000,
Chapter 3).
2.d Countable Additivity
Before we leave this general discussion we should say something
about a further axiom that is widely adopted in textbooks of math-
THE PROBABILITY CALCULUS 27
ematical probability: the axiom of countable additivity. This says
that ifa), a
2
, a
3
, ... are a countably infinite family (this just means
that they can be enumerated by the integers I, 2, 3, ... ) of mutu-
ally inconsistent propositions in the domain of P and the state-
ment 'One of the G. is true' is also included in the domain of P
I
then the probability of the latter is equal to the sum of the P(aJ
Kolmogorov included a statement equivalent to it, his 'axiom of
continuity' , together with axioms (1)- (4) in his celebrated mono-
graph (1950) as the foundational axioms of probability (except
that he called (4) the ' definition' of conditional probability), and
also required the domain of P to be closed not only under finite
disjunctions (now unions, since the elements of the domain are
now sets) but also countable ones, thus making it what is called a
0-field, or a-algebra. These stipulations made probability a
branch of the very powerful mathematical theory of measure, and
the measure-theoretic framework has since become the paradigm
for mathematical probability.
Mathematical considerations have undoubtedly been upper-
most in thi s decision: the axiom of countable additivity is required
for the strongest versions of the limit theorems of probabi lity
(characteristically prefaced by ' almost certainly', or ' with proba-
bility one', these locutions being taken to be synonymous); also
the theory of random variables and distributions, particularly con-
ditional distributions, receives a very smooth development if it is
included. But we believe that the axioms we adopt should be driv-
en by what logicians call 'soundness' considerations: their conse-
quences should be true ofwhatcvcr interpretation we wish to give
them. And the brute fact is that for each of the principal interpre-
tations of the probability calculus, the chance and the epistemic
interpretation, not only are there no compelling grounds for think-
ing the countable additivity axiom always true but on the contrary
there are good reasons to think it sometimesja/se.
The fact is that if we measure chances, or tendencies, by lim-
iting relative frequencies (see Chapter 3) then we certainly have
no reason to assume the axiom, sincc limiting relative frequen-
cies, unlike finite frequencies in fixed-length samples, do not
always obey it: in particular, if each of a countable infinity of
exclusive and exhaustive possible outcomes tends to occur only
finitely many times then its limiting relative frequency is zero,
28
CHAPTER 2
while that of the disjunction is 1. As for the epistemic interpreta-
tion, as de Finetti pointed out (1972, p. 86), it may be perfectly
reasonable (given suitable background information) to put a zero
probability on each member of an exhaustive countably infinite
partition of the total range of possibilities, but to do so contradicts
the axiom since the probability of the total range is always 1. To
satisfy the axiom of countable addivity the only permissible dis-
tribution of probabilities over a countable partition is one whose
values form a sufficiently quickly converging sequence: for
example, 112, 1/4, 1/8, ... , and so forth. In other words, only
very strongly skewed distributions are ever permitted over count-
ably infinite partitions!
In both case, for chances and epistemic probabilities, there-
fore, there are cases where we might well want to assign equal
probabilities to each of a countable infinity of exclusive and
exhaustive outcomes, which we can do consistently if countable
additivity is not required (but they must receive the uniform value
0), but would be prevented from doing so by the principle of
countable additivity. It seems wrong in principle that an apparent-
ly gratuitous mathematical rule should force one to adopt instead
a highly biased distribution. Not only that: a range of apparently
very impressive convergence results, known in the literature as
Bayesian convergence-of-opinion theorems, appear to show that
under very general conditions indeed one's posterior probabilities
will converge on the truth with probability one, where the truth in
question is that of a hypothesis definable in a 0-field of subsets of
an infinite product space (see, for example, Halmos 1950, p. 213,
Theorem B). In other words, merely to be a consistent probabili s-
tic reasoner appears to commit one to the belief that one's poste-
rior probability of a hypothesis about an infinite sequence of
possible data values will converge on certainty with increasing
evidence. Pure probability theory, which we shall be claiming is
no more than a type of logic, as empty 0/ spec(/ic content as
deductive logic, appears to be all that is needed to solve the noto-
rious problem of induction!
If this sounds a bit too good to be true, it is: these results all
turn out to require the principle of countable additivity for their
proof, and exploit in some way or other the concentration of prob-
ability over a sufficiently large initial segment of a countably infi-
THE PROBABILITY CALCULUS 29
nite partition demanded by the principle. To take a simple exam-
ple from Kelly 1996, p. 323: suppose h says that a data source
which can emit 0 or I emits only 1 s on repeated trials, and that
P(h) > O. So h is false if and only if a 0 occurs at some point in an
indefinitely extended sample. The propositions all saying that a 0
occurs first at the nth repetition are a countably infinite disjoint
family, and the probability of the statement that at least one of the
a
i
is true, given the falsity of h, must be 1. So given the front-end
skewedness prescribed by the axiom of countable additivity, the
probability that h is false will be mostly concentrated on some
finite disjunction a
1
v ... van' It is left to the reader to show, as
an easy exercise in Bayes's Theorem in the form (20), section b
above, that the probability that h is true, given a sufficiently long
unbroken run of 1 s, is very close to 1.
There is (much) more to be said on this subject, but for further
discussion the reader is encouraged to consult de Finetti 1872,
Kelly 1996, pp. 321 - 330, and Bartha 2004. Kelly's excellent book
is particularly recommended for its illuminating discussion of the
roles played not only by countable additivity but also (and non-
neglibly) by the topological complexity of the hypotheses in prob-
abilistic convergence-to-the-truth results.
2.e I Random Variables
In many applications the statements in the domain of P are those
ascribing values, or intervals of values, to random variables. Such
statements are the typical mode of description in statistics. For
example, suppose we are conducting simultaneous measurements
of individuals ' heights and weights in pounds and metres.
Formally, the set S of relevant possible outcomes will consist of
all pairs s = (x, y) of non-negative real numbers up to some big
enough number for each of x and y , height and weight respective-
ly (measuring down to a real number is of course practically
impossible, but that is why this is an idealisation).
We can define two functions X and Yon S such that X(x, y) =
x and Y('C, y) = y . X and Yare examples of random variables: X
picks out the height dimension, and Ythe weight dimension of the
various joint possibilities. In textbooks of mathematical probabil-
30 CHAPTER 2
ity or statistics, a typical formula might be P(X > x). What does
this mean? The answer, perhaps not surprisingly, will depend on
which of the two interpretations of P mentioned earlier is in play.
On the chance interpretation, P(X > x) will signify the tendency
of the randomising procedure to generate a pair of observations
(x', y') satisfving the condition that x' > x, and this tendency, as
we observed, will be evaluated by inspecting the frequency with
which it does generate such pairs,
On the other, epistemic, interpretation, P(X > x) will signify a
degree of uncertainty about some specific event signified by the
same inequality formula X> x. For example, suppose that we are
told that someone has been selected, possibly but not necessarily
by a randomising procedure, but we know nothing about their
identity. We are for whatever reason interested in the magnitude
of their height, and entertain a range of conjectures about it,
assigning uncertainty-probabilities to them. One such conjecture
might be 'The height of the person selected exceeds x metres ',
and P(X > x) now symbolises the degree of certainty attached to
it.
This second reading shows that 'random variable' does not
have to refer to a random procedure: there, it was just a way of
describing the various possibilities determined by the parameters
of some application. Indeed, not only do random variables have
nothing necessarily to do with randomness, but they are not vari-
ables either: as we saw above, X, Y, etc. are not variables at all but,
since they take different values depending on which particular
possibilities arc instantiated,/ime/ions on an appropriate possibil-
ity-space (in the full measure-theoretic treatment, their technical
name is measurable jill1ctions).
2.f Distributions
Statements of the form 'X < x', 'X:s; x', play a fundamental role
in mathematical statistics. Clearly, the probability of any such
statement (assuming that they are all in the domain of the proba-
bility function) will vary with the choice of the real number x; it
follows that this probability is a function F(x:) , the so-called dis-
tributionfimction, of the random variable X. Thus, where P is the
THE PROBABILITY CALCULUS 31
probability measure concerned, the value of F(x) is defined to be
equal, for all x to P(X::; x) (although F depends therefore also on
X and P. these arc normally apparent from the context and F is
usually written as a function of x only). Some immediate conse-
quences of the definition of F(.x:) are that
(i) if Xj < x
2
then F('(j) ::; F(x
2
). and
(ii) P(''(j < X::; x
2
) = F(x) - F(.x:
j
).
Distribution functions arc not necessarily functions of one
variable only. For example, we might wish to describe a possible
eventuality in terms of the values taken by a number of random
variables. Consider the 'experiment' which consists in noting the
heights (X, say) and weights (Y) jointly of members of some
human population. It is usually accepted as a fact that there is a
joint (objective) probabil ity distribution for the vector variable
(X Y), meaning that there is a probability distribution function
F(x. y) = P(X::; x & Y::; y). Mathematically this situation is straight-
forwardly generalised to distribution functions of n variables.
2.9 Probability Densities
It follows from (ii) that ifF('() is differentiable at the point x, then
the probability density at the point x is defined and is equal to
dF(x)
f(x) =
dx
in other words, if you divide the probability that X
is in a given interval (x. x + h) by the length h of that interval and
let h tend to 0, then if F is differentiable, there is a probability
density at the point x, which is equal tof('(}. If the density exists
at every point in an interval, then the associated probability dis-
tribution of the random variable is said to be continuous in that
interval. The simplest continuous distribution, and one which we
shall refer to many times in the following pages, is the so-called
uniform distribution. A random variable X is uniformly distrib-
uted in a closed interval I if it has a constant positive probabili-
ty density at every point in I and zero density outside that
interval.
32 CHAPTER 2
Probability densities are of great importance in mathematical
statistics-indeed, for many years the principal subject of
research in that field was finding the forms of density functions
of random variables obtained by transformations of other ran-
dom variables. They are so important because many of the prob-
ability distributions in physics, demography, biology, and
similar fields are continuous, or at any rate approximate contin-
uous distributions. Few people believe, however, in the real-as
opposed to the mathematical-existence of continuous distribu-
tions, regarding them as only idealisations of what in fact are
discrete distributions.
Many of the famous distribution functions in statistics are
identifiable only by means of their associated density functions;
more precisely, those cumulative distribution functions have no
representation other than as integrals of their associated density
functions. Thus the famous normal distributions (these distribu-
tions, of fundament al importance in statistics, are uniquely deter-
mined by the values of two parameters, their mean and standard
deviation, which we shall discuss shortly) have distribution func-
tions characterised as the integrals of density functions.
Some terminology. Suppose X and Yare jointly distributed
random variables with a continuous distribution function F(X, Y)
and density function ./(t. y). Then F(XJ = r f(x, y)dy is called
the marginal distribution of X. The operation of obtaining margin-
al distributions by integration in this way is the continuous ana-
logue of using the theorem of total probability to obtain the
probability P(a) of a by taking the sum 'i:.P(a & b). Indeed, if X
and Yare discrete, then the marginal distribution for X is just the
sum P(X = x) = IP(X = x. & Y = v). The definitions are
I I I j
straightforwardly generalised to joint distributions of n variables.
2.h Expected Values
The expected value of a function g(X) of X is defined to be (where
it exists) the probability-weighted average of the values of g. To
take a simple example, suppose that g takes only finitely many
valuesg
l
, ,gil with probabilities a
l
, .. ,an' Then the expect-
ed value E(g) of g always exists and is equal to 'i:.g;G;. If X has a
THE PROBABILITY CALCULUS 33
probability density function.f(x) and g is integrable, then E(g) =
f x, g(x)[r-,)dx where the integral exists.
X, In most cases, functions of random variables are themselves
random variables. For example, the sum of any n random vari-
ables is a random variable. This brings us to an important proper-
ty of expectations: they are so-called linearfimctionals. In other
words, if XI' ... , Xn are n random variables, then if the expecta-
tions exist for all the ~ then, because expectations are either
sums or limits of sums, so does the expected value of the sum X =
Xl + ... + Xn and E(X) = E(X
I
) + . .. + E(XJ
2.i I The Mean and Standard Deviation
Two quantities which crop up all the time in statistics are the mean
and standard deviation of a random variable X The mean value of
X is the expected value E(X) of X itself, where that expectation
exists; it follows that the mean of X is simply the probability-
weighted average of the values of X The variance of X is the
expected value of the function (X - mi, where that expectation
exists. The standard deviation of X is the square root of the vari-
ance. The square root is taken because the standard deviation is
intended as a characteristic measure of the spread of X away from
the mean and so should be expressed in units of X Thus, if we write
s.d.(X) for the standard deviation of X, s.d.(X) = vE[(X - m)2J.
where the expectation exists. The qualification 'where the expec-
tation exists' is important, for these expected values do not always
exist, even for some well-known distributions. For example, if X
has the Cauchy density - a ~ _ then it has neither mean nor
. n(a
2
+ x
2
)
vanance.
We have already mentioned the family of normal distributions
and its fundamental importance in statistics. This importance
derives from the facts that many of the variables encountered in
nature are normally distributed and also that the sampling distri-
butions of a great number of statistics tend to the normal as the
size ofthe sample tends to infinity (a statistic is a numerical func-
tion of the observations, and hence a random variable). For the
moment we shall confine the discussion to normal distributions of
34 CHAPTER 2
one variable. Each member of this family of distributions is com-
pl etely determined by two parameters, its mean ,Ll and standard
deviation 0: The normal distri bution functi on itself is given by the
integral over the values of the real variabl e t from - 00 to x of the
density we mentioned above, that is, by
I _ 1 ( I - ~ )2
l (t ) = - e 2 -iT
. aV2ii:
It is easily veri f ied from the analytic expression for F(x) that
the parameters ,Ll and (J are indeed the mean and standard devi-
ati on of X. The curve of the normal density is the famili ar bell-
shaped curve symmetrical about x = ,Ll wi th the points x = jLl
(J corresponding to the points of maximum slope of the curve
(Figure 2.1). For these distributions the mean coincides with
the median, the value ofx such that the probability of the set {X
< x} is one hal f (these two points do not coincide for all other
types of di stribution, however). A fact we shall draw on later is
that the interval on the x- axis determined by the di stance of
1.96 standard deviations centred on the mean supports 95% of
the area under the curve, and hence receives 95% of the total
probability.
f(x)
'11 - 0" x
FIGURE 2.1
THE PROBABILITY CALCULUS 35
2.; Probabilistic Independence
Two propositions h I and h2 in the domain of P are said to be prob-
abilistically independent (relative to some given probability
measure P if and only if P(h
l
& h
2
) = P(h
l
)P(hJ It follows
immediately that, where P(h I) and P(h
2
) are both greater than
zero, so that the conditional probabilities are defined, P(h I 1 h
2
) =
P(h f) and P(h II h f) = P(h), just in case h I and h2 are probabilis-
tically independent
Let us consider a simple example, which is also instructive in
that it displays an interesting relationship between probabilistic
independence and the so-called Classical Definition of probability.
A repeatable experiment is determined by the conditions that a
gi ven coin is to be tossed twice and the resulting uppermost faces
are to be noted in the sequence in which they occur. Suppose each
of the four possible types of outcome-two heads, two tails, a
head at the first throw and a tail at the second, a tail at the first
throw and a head at the second-has the same probability, which
of course must be one quarter. A convenient way of describing
these outcomes is in terms of the values taken by two random
variables XI and X
2
, where XI is equal to 1 if the first toss yields a
head and 0 if it is a tail, and X
2
is equal to I if the second toss
yields a head and 0 if a tail.
According to the Classical Definition, or, as we shall call it,
the Classical Theory of Probability, which we look at in the next
chapter (and which should not be confused with the Classical
Theory of Statistical Inference, which we shall also discuss), the
probability of the sentence XI = I' is equal to the ratio of the
number of those possible outcomes of the experiment which sat-
isfy that sentence, divided by the total number, namely four, of
possible outcomes. Thus, the probability of the sentence 'XI = \'
is equal to \ 12, as is also, it is easy to check, the probability of
each of the four sentences of the form 'X = X " i = \ or 2, x = 0
I I I
or I . By the same Classical criterion, the probability of each of the
four sentences 'XI = XI & X
2
= xc' is \/4.
Hence
36 CHAPTER 2
and consequently the pairs of sentences 'XI = x I' , 'X
2
= x
2
' are
probabilistic ally independent.
The notion of probabilistic independence is generalised to n
propositions as follows: hi' ... ,hn are said to be probabilistically
independent (relative to the measure P) if and only if for every
subset hi!' ... ,hik of hI' ... ,h
n
.
It is easy to see, just as in the case of the pairs, that if any set of
propositions is probabilistically independent, then the probability
of anyone of them being conditional on any of the others, where
the conditional probabilities are defined, is the same as its uncon-
ditional probability. It is also not difficult to show (and it is, as we
shall see shortly, important in the derivation of the binomial dis-
tribution) that if hI' ... ,hn are independent, then so are all the 2"
sets h /' ... , hn' where +h is hand -h is ~ h
Any n random variables XI' ... ,X
n
are said to be independent
if for all sets of intervals II' ... ,In of values of XI' ... ,X
n
respec-
tively, the propositions XIE 11' ... ,X
n
E In are probabilistically
independent. We have, in effect, already seen that the two random
variables XI and X
2
in the example above are probabilistically
independent. If we generalise that example to that of the coin's
being tossed n times, and define the random variables XI ' ... ,X
n
just as we defined XI and X
2
, then again a consequence of apply-
ing the Classical 'definition' to this case is that XI' ... ,x" are
probabilistically independent. It is also not difficult to show that
a necessary and sufficient condition for any n random variables
XI' ... ,x" to be independent is that
where F(x
l
, ,x) is the joint distribution function of the vari-
ables XI' ... ,X
n
and F(x) is the marginal distribution of Xi'
Similarly, if it exists, the joint density f(xI' ... , x,) factors
into the product of marginal densitiesj(x
l
) .. . f(x,) if the Xi are
independent.
THE PROBABILITY CALCULUS
2. k I Conditional Distributions
According to the conditional probability axiom, axiom 4,
(1) P( X < x I y < Y < Y + 6 y) =

pry < Y y + 6y)
37
The left-hand side is an ordinary conditional probability. Note
that if F(x) has a densityHx) at the point x, then P(X = x) = 0 at
that point. We noted in the discussion of(4) that P(a I b) is in gen-
eral only defined if P(b) > O. However, it is in certain cases pos-
sible for b to be such that P(b) = 0 and for P(a I b) to take some
definite value. Such cases are afforded where b is a sentence of
the form Y = Y and there is a probability densityf(y) at that point.
For then, if the joint density f(x, y) also exists, then multiplying
top and bottom in ( I) by 6 y, we can see that as 6 y tends to 0, the
right-hand side of that equation tends to the quantity
I
Xf(u,y)dU ,
- x .1M
wherej(v) is the marginal density ofy, which determines a distri-
bution function for X, called the conditional distribution .Iimction
of X with respect to the event Y = Y. Thus in such cases there is a
perfectly well-defined conditional probability
P(x] < X x21 Y = y),
even though pry = y) = O.
The quantity f('(,y! is the density function at the point X =
.1M
x of this conditional distribution (the point Y = y being regarded
now as a parameter), and is accordingly called the conditional
probability density of X at x, relative to the event Y = Y. It is of
great importance in mathematical statistics and it is customarily
denoted by the symbol.f(x I y). Analogues of (18) and (19), the
38
CHAPTER 2
two forms of Bayes's Theorem, are now easily obtained for densi-
ties: where the appropriate densities exist
1(, I y) = f(y I Y j l x
f(y)
and
Rv I x)frx)
f(, I y) =- --_.- .
. fXxf(y I x)fMdx
2.1 The Bivariate Normal
We can illustrate some of the abstract formal notions we have di s-
cussed above in the context of a very important multivariate di s-
tribution, the bivariate normal distribution. This distribution is, as
its name implies, a distribution over two random variables, and it
is determined by five parameters. The marginal distributions of
the two variables X and Yare both themselves normal, with means
,u
x
' fly and standard deviations ox' Oy. One more parameter, the
correlation coefficient p, completely specifies thc di stribution.
The bivariate density is given by
f(x,y) =
This has the form of a more-or-Iess pointed, more-or-I ess elongat-
ed hump over the x. y plane, whose contours are cllipses with
eccentricity (departure from circularity) determined by p. plies
between -1 and + 1 inclusive. When p = 0, X and Yare uncorre-
lated, and the contour ellipses are circles. When p is either + 1 or
-1 the ellipses degenerate into straight lines. In this case all the
probability is carried by a set of points of the form y = ax + b. for
specified a and b. which will depend on the means and standard
deviations of the marginal distributions. It follows that the condi-
tional probability P(X = x I Y = y) is 1 if y = ax + b. and if not.
The conditional distributions obtained from bivariate (and
more generally multivariate) normal distributions have great
THE PROBABILITY CALCULUS 39
importance in the area of statistics known as regression a n a ~ v s i s
It is not difficult to show that the mean ,u(X I y) = r :"(./("( I y)dx
(or the sum where the conditional distribution is disErete) has the
equation Il(X I y) = Ilx + P d' (y - Ilv)' In other words, the depend-
) .
ence of the mean on y is linear, with gradient, proportional to p,
and this relationship defines what is called the regression of X on
Y. The linear equation above implies the well-known phenomenon
of regression to the mean. Suppose Px = P
y
and ,u
x
= Il
y
= m. Then
,u(X I y) = m + pCv - m), which is the point located a proportion p
of the distance between y and m. For example, suppose that peo-
ple's heights are normally distributed and that Y is the average of
the two parents' height and X is the offspring's height. Suppose
also that the means and standard deviations of these two variables
are the same and that p = 112. Then the mean value of the off-
spring's height is halfway between the common population mean
and the two parents' average height. It is often said that results like
this explain what we actually observe, but explaining exactly how
parameters of probability distributions are linked to what we can
observe turns out to be a hotly disputed subject, and it is one
which will occupy a substantial part of the remainder of this book.
Let us leave that topic in abeyance, then, and end this brief
outline of that part of the mathematical theory of probability
which we shall have occasion to use, with the derivation and some
discussion of the limiting properties of the first non-trivial ran-
dom-variable distribution to be investigated thoroughly, and
which has no less a fundamental place in statistics than the nor-
mal distribution, to which it is intimately related.
2.m The Binomial Distribution
This was the binomial distribution. It was through examining the
properties of this distribution that the first great steps on the road
to modern mathematical statistics were taken, by James Bernoulli,
who proved (in Ars Conjectandi, published posthumously in
1713) the first of the limit theorems for sequences of independent
random variables, the so-called Weak Law o.fLarge Numbers, and
40 CHAPTER 2
Abraham de Moivre, an eighteenth-century Huguenot mathemati-
cian settled in England, who proved that, in a sense we shall make
clear shortly, the binomial distribution tends for large n to the nor-
mal. Although Bernoulli demonstrated his result algebraically, it
follows, as we shall see, from de Moivre's limit theorem.
Suppose (i) Xi' i = I, ... , n, are random variables which take
two values only, which we shall label 0 and 1, and that the prob-
ability that each takes the value 1 is the same for all i, and
equals p:
P(J\ = 1) = = 1) = p.
Suppose also (ii) that the Xi are independent; that is,
P(XI = Xl & ... & Xn = X) = P(XI = XI) x ... X P(J\, = xJ
where Xi = 1 or O. In other words, the Xi are independent, identi-
cally distributed random variables. Let Yin) = XI + ... + XII. Then
for any f; 0 S r s n,
since using the additivity property, the value of P is obtained by
summing the probabilities of all conjunctions
where r of the X are ones and the remainder are zeros. There
I
are "C, of these, where "C, is the number of ways of selecting
b
f d . 1 n! h . I
r 0 out 0 n, an IS equa to --- - ,were n! IS equa to
(n -r)!r!
n(n - l)(n - 2) ... 2.1, and O! is set equal to 1). By the independ-
ence and constant probability assumptions, the probability of each
conjunct in the sum is p'"(I - pr ',since P(X
i
= 0) = 1 - p.
is said to possess the binomial distrihution. The mean of
is np, as can be easily seen from the facts that
THE PROBABILITY CALCULUS
and that
E(X) = P . I + (1 - p) . 0 = p.
The squared standard deviation, or variance of Yin)' is
E(Y(n) - npj2 = E(Y(n/) + E(npj2-
= + (npj2- 2npE(Y(n)
= - (npj2.
Now
= "L(X2) +
= np + n(n - 1)p2.
Hence
2.m I The Weak Law of Large Numbers
41
The significance of these expressions is apparent when n becomes
very large. De Moivre showed that for large n, is approximately
normally distributed with mean np and standard deviation
vnp(f - p) (the approximation is very close for quite moderate
values of n). This implies that the so-called standardised variable
Z
O;n) - np) . . 1 11 d' 'b del
= . IS approximate y norma y Istn ute lor arge n,
ynp(l - p)
with mean 0 and standard deviation I (Z is called 'standardised'
because it measures the distance of the relative frequency from its
mean in units of the standard deviation). Hence
P(-k < Z < k) = <t>(k) - <t> ( - k),
where <t> is the normal distribution function with zero mean and
unit standard deviation. Hence
42
CHAPTER 2
P(p - k jl!-n
q
< '! < p + kip;;) = (k) - (- k),
n 11
where q = 1 - p. So, setting E = k 1 P: '
11
P(p - E < p + E) = (P - (- EI n \
11 pq \ pq !
Clearly, the right-hand side of this equation tends to 1, and we
have obtai ned the Weak Len!' oj"Large Numhers:
y
P( I pi < t") -- 1, for all E > O.
n
This is one of the most famous theorems in the history of
mathemati cs. James Bernoulli proved it originally by purely com-
binatorial methods. It took him twenty years to prove, and he
called it hi s "golden theorem". It is the first great result of the dis-
cipline now known as mathematical statistics and the forerunner
of a host of other limit theorems of probability. Its significance
outside mathematics I ies in the fact that sequences of independent
binomi al random vari abl es with constant probability, or Bernoulli
sequences as they are called, are thought to model many types of
sequence of repeated stochastic tri als (the most familiar being
tossing a coin n times and registering the sequence of heads and
tail s produced) . What thc theorem says is that for such sequences
of trials thc relative frequency of the particular character con-
cerned, like heads in the example we have just menti oned, is with
arbitrarily great probability going to be situated arbitraril y close
to the parameter p.
The Weak Law, as stated above, is only one way of appreciat-
ing the significance of what happens as 11 increases. As we saw, it
was obtained from the approximation
ftj f
-
pq Y pq
P(p - k ... <- < p +k - )=<I>(k)- (- k) ,
n 11 n
where q = 1 - p, by repl acing the vari able bounds (depending on
THE PROBABILITY CALCULUS 43
n) k ;-;;q by , and repl acing k on the right-hand sidc by /;-.
~ M
The resulting equation is equivalent to the first. In other words,
the Weak Law can be seen either as the statement that if we select
some fixed interval of length 2 centred on p. then in the limit as
n increases, all the distribution will li e within that interval, or as
the statement that if we f irst select any value between 0 and 1 and
consider the interval centred on p whi ch carries that value of the
probability, then the endpoints of the interval move towards p as
n increases, and in the limit coincide with p.
Another ' law of large numbers' seems even more emphatically
to point to a conncction between probabilities and frcquencies in
sequences of identically di stributed, independent binomial ran-
dom variables. This is the so-called Strong Law, which is usually
stated as a result about actually infini te sequenccs of such vari-
ables: it asserts that with probability cqual to 1, the limit of ~ 1 i / 1 1
exi sts (that is to say, the relati ve frequency of ones converges to
some fini te value) and is equal top.
So stated, the Strong Law requires for its proof thc axiom of
countable additivity, whi ch we have cautioned against accepting
as a general principle. Nevertheless, a ' strong enough' version of
the Strong Law can be stated which does not assumc countabl e
additivity (the other ' strong' limit theorems of mathematical prob-
ability can usually be rephrased in a similar way): it says that for
an infinite sequence XI ' X
2
, .... of {O, I }-valued random vari-
ables, if D, are any positive numbers, however small , then there
exi sts an 11 such that for all m>n the probability that ~ I I , III ) - P is
less than is greater than I-D.
What thi s version of the Strong Law says is that the conver-
gence of the ~ 1 1 is 1ll1i/hrm in the small probability. The Weak
Law is weak in the sense that it merely says that the probability
that the deviation of Y . from J) is smallcr than can be madc
( II I
arbitrarily close to I by taking n large enough; the Strong Law
says that the probability that thc deviation will become and
remain smaller than can be made arbitrarily close to I by tak-
ing n large enough.
At any rate, throughout the eighteenth and nineteenth cen-
turies people took these results to justify inferring, from the
44
CHAPTER 2
observed relati ve frequency of some given character in long
sequences of apparently causally independent trials, the approxi-
mate value of the postulated binomial probability. While such a
practice may seem suggested by these theorems, it is not clear that
it is in any way justifi ed. While doubts were regul arly voiced over
the validity of thi s ' inversion', as it was call ed, of the theorem, the
temptation to see in it a licence to infer to the value of p from
'large' samples persists, as we shall see in the next chapter, where
we shall return to the discussion.
CHAPTER 3
The Laws of Probability
3.a Prologue: Frequency-Probability
We pointed out in 2.c that there are two main ways of interpreting
the probability axioms. Throughout this book we shall be mainly
concerned with one of them, an epistemic interpretation in which
the probability function measures an agent's uncertainty. This
interpretation is also called Bayesian probability. However, some
discussion of the other interpretation of the axioms is unavoid-
able, because much of the application of Bayesian probability is
to hypotheses about these other probabilities. There is a slight
problem of terminology here, since there is no ready antonym to
'epistemic', but to emphasise the fact that these probabilities are
supposed to characterise objective factual situations and set-ups
we shall rest content with the not wholly satisfactory terminology
of 'objective probabilities'.
What makes these important to science is that in a variety of
contexts, from demography in the large to quantum mechanics in
the small , they seem to be readily amenable to empirical measure-
ment, at any rate in principle. For to say that an experimental
arrangement has a stable objective probability of delivering any
one of its possible outcomes is at least arguably to say that it has
a characteristic tendency to do so, the magnitude of which is plau-
sibly gauged in terms of the normalised, or relative, frequency
with which the outcome in question is actually produced; or in
other words, the number of times the outcome occurs in n trials,
divided by n. Thus Pitowsky:
The observational counterparts of the theoretical concept 'probabil-
ity distribution' are the relative frequencies. In other words, as far as
repeatable ... events are concerned probability is manifested in fre-
quency (1994, p. 98)
46 CHAPTER 3
But not just frequency: fundamental to the identification of objec-
tive probability with relative frequency is the long run, for it is
only in the long run, in general, that such frequencies behave as
sufficiently stable features to be the counterparts of the postulat-
ed probability-distribution, and indeed to be objects of scientific
enquiry at all. Moreover, there is a good deal of evidence that in
suitable experimental contexts the relative frequency with which
each of the various possible outcomes occurs settles down within
a smaller and smaller characteristic interval as the number of
observations increases. This is not very precise, however (how
should the interval vary as a function of sample size?), and we
shall follow von M ises (1939) in replacing the rather vague notion
ofa relative frequency 'settling down within an increasingly small
interval' by the precise mathematical definition of a limit. Thus,
where a describes a generic event A in the outcome-space of the
experiment, I and n(A) is the number of occurrences of A in n rep-
etitions of the experiment, we define the measure of the probabil-
ity (we shall often simply write 'probability' where the context
shows that it is objective probability that is being referred to) of a
to be thc limit of n(A)/n as n tends to infinity. This limit is to be
regarded as a characteristic attribute of the experimental condi-
ti ons themselves. Indeed, just as tendencies manifest themselves
in frequencics of occurrence when the conditions are repeatedly
instantiated, we can think of this postulated limiting relative frc-
quency as an exact measure of the tendcncy of those conditions to
deliver the outcomc in question (this idea has of course been pro-
moted under the name 'propensity theory of probability', by
Popper (1959) and others; it was also the view of von Mises
whose own frequency theory is usually, and incorrectly, regarded
as not being of this type).
A pleasing feature of the limit definition is that limiting fre-
quencies demonstrably satisfy the axioms of the probability cal-
culus. This is very easy to show for unconditional probabilities,
and we leave it as an exercise. For conditional probabilities the sit-
I A generic event-description is one which states simply that the event occurs.
without reference to any specific feature of the situati on, its time, place. and so
forth. So, if the experiment consists in tossing a given coin, A might be the event
or landing heads, in which ease a is the statement 'the coin lands heads'.
THE LAWS OF PROBABILITY 47
uation is only slightly more complicated, due to the fact that as yet
conditional probabilities have not been defined in a frequency-
context. But the definition is entirely natural and intuitive: where
a and b are generic event-descriptors as above, we define P(a I b)
to be the long-run relative frequency of outcomes of type A among
all those of type B. Hence P(a I b) = lim n---;.x n(A&Bln(B).
It immediately follows that P(alh) = lim [n(A&B)ln +
n(B)ln] where lim n(B) exists. Hence P(alb) = P(a&b)IP(b) where
P(h) > O.
While the use of the limit definition allows us to regard objec-
tive probabilities as probabilities in the purely formal sense of the
probability calculus, it has nevertheless elicited from positivisti-
cally-minded philosophers and scientists the objection that we can
never in principle, not just in practice, observe the infinite
n-limits. Indeed, we know that in fact (given certain plausible
assumptions about the physical universe) these limits do not exist.
For any physical apparatus would wear out or disappear long
before n got to even moderately large values. So it would seem
that no empirical sense can be given to the idea of a limit of rela-
tive frequencies. To this apparently rather strong objection anoth-
er is frequently added, that defining chances in terms of limiting
frequencies is anyway unnecessary: the mathematical theory of
probability in the form of the Strong Law of Large
Numbers shows how the apparent convergence of the relative fre-
quencies to a limit can be explained as a feature oflong sequences
of Bernoulli trials.
c
Let us look into this claim. The 'strongest' form of the Strong
Law refers to thc set of possible outcomes VI" of an infinitely repeat-
ed experiment which generates at each repetition either a 0 or a 1.
These possibilities are described by a sequence Xi of independent,
identically distributed random variables such that X/w) is equal to
the ith coordinate of VI' (0 or 1) and P(X
i
= I) = P for some fixed p
and all i. This Strong Law says that with probability 1 the limit, as
C A variant of the Strong Law, the LUll' of the Iterated Logarithlll. also seems to
answer the other question of how large the interval is within which the frcquen-
cies are confined with increasing n: with probability 1 the frequency oscillates
within an interval of length ([In In 11]/2n)1 " where In signifies the natural loga-
rithm. the inverse function to expo
48 CHAPTER 3
n tends to infinity, of n -/ LXi' where the sum is from I to 11, exists
and is equal to p (as we pointed out in Chapter 2, its proof in this
form requires the principle of countable additivity) . Thus, the
claim proceeds, the assumption that a sequence of coin tosses is
approximately modelled as a sequence of Bernoulli trials (we
shall return to this question in a moment) is sufficient to explain
the convergence of the frequencies of heads and tails to within an
arbitrarily small interval characteristic (because of the fixed value
of p) of the experimental arrangement.
More than a pinch of salt is advised before swallowing this
story. Even if we were to permit the use of countable additi vity,
and hence approve the Strong Law in its strongest form, it is not
difficult to see that in itself it explains nothing at all , let alone
sanctions the identification of probabilities with observed relative
frequencies, since no empirical meaning has yet been given to the
probability function P. Even were it true that a large value of P,
even the value J, attaches to a particular event cannot by itself
explain any occurrence of that event: no statement of the pure
probability calculus by itself makes any categorical prediction.
Not only that: even were it the case that in some acceptable sense
the Strong Law, or rather the modelling hypothesis implying it,
that the coin tosses are an instantiation of a Bernoulli process,
explained the convergence of relative frequencies, this by itself
would be no ground for accepting the explanation offered. As we
shall see in a subsequent chapter, there are infinitely many possi-
ble distinct and mutually inconsistent explanations of any
observed effect. There has to be some independent reason for
selecting anyone of these rather than any other, which the
Bayesian theory formally represents in terms of different prior
probabilities.
We can also reject the frequently-made claim that the Strong
Law shows why chance cannot be defined as limiting relative fre-
quency-because the identity only occurs on a set of measure I,
and so it is possible in principle for the relative frequency not to
converge, or to converge to some other value. But again, without
any independent meaning given to P, or independent reason to
accept the modelling hypothesis itself, this claim is empty. Some
additional assumption linking statements of the probability calcu-
lus to physical reality is, on the contrary, indispensable. We have
THE LAWS OF PROBABILITY 49
made long-run frequency a definitional link, because that is the
simplest procedure that avoids the obj ections. There have been
other suggestions. One, which seems first to have been explicitly
made by A.A. Coumot in the nineteenth century, and which often
goes under the name of Cournot :s' Principle, is that sufficiently
small probabilities should be assumed to be practi cal impossibil-
ities (the same rule was also proposed by Kolmogorov in his 1950
monograph 1950). After all , the minute probability that the kinet-
ic theory ascribes to ice forming spontaneously in a warm bath is
typically taken as an aff idavit that it won't happen in anyone's
lifetime. Conversely, if it happened rather often, that would almost
certainly be taken as a good indication that the kinetic theory is
incorrect.
It should be clear, however, that without some qualification
Coumot's Principle is false, for events with almost infinitesimal
probabiliti es occur all the time without casting any suspicion
upon the theories which assign them those probabiliti es: the exact
configuration of air molecules in a room at any given time has a
microscopic probability; so does any long sequence of outcomes
of tosses of a fair coin (even with so small a number as twenty the
probability of each possible sequence is already around one in a
million). Can the Principle be qualifi ed in such a way as to make
it tenable? This is just what the well-known and widely-used the-
ories of bi valent statistical tests of R.A. Fisher, and Neyman and
Pearson (all believers in a long-run frequency account of statisti-
cal probabilities), attempt to do in their diffe rent ways.
Unfortunately, as we shall show later, in Chapter 5, these attempts
al so fail.
But there remains the objection to a limiting frequency meas-
ure of obj ective probability that relative frequencies in finite
samples deductively entail nothing whatever about the behaviour
oflimits, an objecti on reinforced by the fact that these limits only
exist in an idealised modelling sense. The remarkable fact is that
empirical data do nevertheless give us information about the
limit, though to show how we need to add to that definition
another feature of actual random sequences besides their appar-
ent convergence to small neighbourhoods of some characteristic
value. Thi s is the fact that these sequences are random. The intu-
itive idea, which was made mathematically ri gorous by the
50 CHAPTER 3
American mathematician and logician Alonzo Church using the
theory of computable functions which he had independently
invented, is that a sequence is random if there is no algorithm, into
which a knowledge of the first 11 members can be inputted, 11 =
1, 2, 3, ... , for discovering subsequences in which the probabili-
ties (the limiting relative frequencies) differ: were such a possibil-
ity to exist then it could be exploited by a mathematically adept
gambler to generate a sure profit.
According to Richard von Mises the two principles of conver-
gence and randomnness, or immunity to gambling systems, deter-
mine a plausible mathematical model, which he called a Kollektiv,
of what is empirically observed in the field of stochastic phenom-
ena, either carefully cultivated in casinos, or occurring naturally
in the passing on of genes from two parents, radioactive emis-
sions, and so forth. One of their consequences is partitioning a
Kollektiv into n-termed subscquences, for each 11 = I, 2, 3, ... ,
generates a new Kollektiv of n-fold Bernoulli sequences, in which
successive members are probabilistically independent, with the
chance of any specified outcome occurring at the ith place the
same for all i, i = 1,2, ... ,11 (von Mises 1964, pp. 27, 28). In other
words, a Kollektiv with outcomes 0 and I (say) represents an infi-
nite sequence of n-fold random samples in which the outcomes
are characterised by the same chance, p, as in the original
Kollektiv. This remarkable fact might seem to preclude the appli-
cation of the theory to the important and extensive field of sto-
chastic phenomena where independence fails, and which instead
exhibit systematic probabilistic dependencies like Markov
processes for example. This is fortunately quite untrue: all that
follows is that the particular random variables Xi' i = L ... , 11,
defined on the set of all 211 n-tuples (x I' ... ,x,), .\ = 0 or 1, where
Xi = Xi' are independent. The random variables Y
k
= XI + ... + X
k
,
k = L .... , 11 , are certainly not independent.
Later, we shall show how it follows from these properties of
Kollektivs that data from finite sequences can and do actually
give information about the behaviour of limits. To do this we
need first to develop the theory of epistemic probability, the
Bayesian theory, for it is only within such a theory that this fact
can be demonstrated. That it can be demonstrated at all is
remarkable enough. That it needs a theory of epistemic probabil-
THE LAWS OF PROBABILITY 51
ity to do so underlines the indispensability of the latter for pro-
viding a basis of inductive inferencc; that this basis is also a
secure logical basis is even more remarkable. That is the pro-
gramme for the remainder of this chapter.
3.b Measuring Uncertainty
To carry out this programme we shall first have to explain more
exactly what epistemic probabilities are. The answers that have
been given, at different times over the past three centuries, ditTcr in
the details, but one thing on which nearly everyone is agreed is that
epistemic probabilities are numerical measures of uncertainty. So
far, so good, except that it is not very far, and the question is where
to go from there. Here opinions differ. The development we shall
favour is the one that seems to us the most natural and definitely
the simplest; it brings with it as a consequence that the rules of
epistemic probability arc nothing but rules of logic: not deductive
logic, but a logic that is very closely kin to it. We shall show that
the axioms of probability are a logic a/uncertain inference.
The idea that there is an authentic logic of uncertain inference,
complementing the deductive logic of 'certain' inference, has an
ancient pedigree, extending back to the beginnings of the mathe-
matical theory of probability in the seventeenth century. Leibniz in
the Nouveaux Essais and elsewhere said so explicitly, and the idea
runs like a thread, at times more visible, at times less, through the
subsequent development of the epistemic view of probability, right
up to the end of the twentieth century. Thus Ramsey: "The laws of
probability are laws of consistency, an extension to partial beliefs
of formal logic, the logic of consistency" ( 1931, p. 182). Ironically
enough, however, Ramsey did more than anyone else to deflect it
out of a logical path by choosing to embed his discussion not with-
in the theory of logic as it was then being (very successfully)
developed in continental Europe, but within the very different the-
oretical matrix of axiomatic utility, which Ramsey himself was the
first to develop axiomatically, and of which more anon.
In this chapter we shall try to establish just this link between
contemporary deductive logic and the laws of epistemic probabili-
ty. We shall proceed in stages, the first of which will be to show
52 CHAPTER 3
how it is possible to make numerical assessments of uncertainty,
and then see what follows from the properties these have. That it
is possible to make such assessments is hardly in doubt: people
have been doing so for centuries, albeit indirectly, in terms of
odds; for example:
SPEED: Sir, Proteus, save you! Saw you my master?
PROTEUS: But now he parted hence, to embark for Milan.
SPEED: Twenty to one, then, he is shipp'd already. (William
Shakespeare, Two Gentlemen oj" Verona. 3)
We say 'indirectly' in terms of odds because 'odds' traditionally
means 'betting odds', and betting odds directZv determine the
ratio in which money changes hands in a bet, a fact which raises
problems for any attempt to use the odds the agent gives or
accepts as a measure of their uncertainty. The method suggested
in de Finetti (1937, p. 102), of identifying your personal degree of
uncertainty with the odds at which you would be prepared to take
either side of a bet at arbitrary stakes will certainly not work, for
reasons which have been well-documented and are by now very
familiar (your willingness to bet will be sensitive to the size ofthc
stake, to your attitude to risk, maybe to moral considerations, and
to possible other 'external' factors), and which were historically a
powerful factor in making people think that there was no alterna-
tive to the explicitly utility-based approach which we shall discuss
in the next section. It is not possible to evade the problem by
requiring that the stakes are always sufficiently small that they can
be safely assumed to be approximately linear in utility, since the
Dutch Book argument for the conditional probability axiom
(Chapter 2, axiom (4)) requires that one be prepared to make a
combined bet against b and on a&b with a stake on b which can
in principle be arbitrarily large, and the Duteh Book argument fur-
nishes the prudential reason for making your betting quotients
'We arc grateful to Vittorio Ciirotto and Michel Gonzalez (200 I) for drawing our
attention to this quotation. A more recent example of the same thing is this: "The
betting among physicists. however. was that there was an even chance that the
sse [Superconducting Supercollider] would find exotic parti cles beyond the
Standard Model" (Kaku 1994. p. 1 X3).
THE LAWS OF PROBABILITY 53
obey the probability axioms: if they don't, you can be made to
lose come what may (for an elementary proof see Gillies 2000 pp.
59- 64).
But citing odds does not necessarily indicate a corresponding
propensity to bet that way, and Speed's 20: I odds were probably
not meant to be taken in any such way: they are more likel y to be
(for Shakespeare) hi s assessment of the relative chances of his
master shipping versus not shipping (we are using the word
' chance' here in an entirely informal vernacul ar sense). Call such
odds chance-based odds. It is important to keep in mind that
these odds are conceptually di stinct from betting odds: they are
(so far at any rate)judgments about the relative likelihoods of the
rel evant propositi on and its negation. That said, these odds are
nevertheless numericalZv identical to a special type of such odds,
the agent's fair betting odds, which is the reason that they are
referred to as odds at all. What are fair betting odds'? The answer,
as old as probability itself, is that betting at your personal,
chance-based odds balances the risk (according to your own
estimation) betH'een the sides oj'the bet. According to a standard
definition (Mood and Graybill 1963, p. 276, for example), risk is
expected loss, and expected loss is something that is straightfor-
wardly computable, using the normalisations 20/21 and 1/21 of
Speed's 20: I chance-ratio as Speed's respective personal proba-
bilities of his master having shipped and not having shipped: the
word ' probabiliti es' is justified by the fact that these normalisa-
tions are numbers in the probability scale of the closed unit inter-
val, summing to I .
It is now simpl e to see that odds in this rati o do indeed balance
the risk. Suppose Speed is observing two individuals actually bet-
ting on whether his master has shipped. One indi vidual collects Q
ducats from the other if the proposi tion is true, and loses R ducats
to the other if it is fa lse. It is easy to work out that, according to
Speed's evaluation, the risk of this bet to the first individual is
equal to the risk to the second if and only if R:Q = 20: I. Hence
20: I is Speed's fair odds. Note that the bet is fair according to
Speed o(the magnit ude oj' Q + R, i.e. oj'the stake:
its fairness, or otherwise, depends only on the odds R: Q. SO
chance-based odds are also fair betting odds. Normalised betting
odds are called betting quotients, and so normali sed chance-based
54 C HAPTER 3
odds arc also personal fair betting quotients. Thus we havc the
equations:
Personal probahilities = normalisedfair betting quotients =
normalised chance-based odds
There is one proviso in all this, applying whcn the truth-value
of the proposition in question is not fully decidablc. One of the
major applications of the Bayesian formalism is to the problem of
deriving posterior probabilities for scientific hypotheses (hencc
thc title of this book!). Yct, as Poppcr becamc famous for pointing
out, if they are sufficiently general these are at bcst only one-way
decidable, being refutable (with the help of suitable auxiliary
hypotheses) but not verifiable. Clearly, the only practical way to
equalise risk in a bet on these is for it to be at zero odds. But thcsc
may not, and often will not, correspond to your chance-based
odds. But all we have to do to re-establish equality is to make the
proviso, possibly countcrfactual, that any bct will be decided, if
necd bc by an omniscicnt oraclc (this is a fairly standard proce-
dure and, since nothing in this account stipulatcs that the agent
must or should bet at their fair odds, an unexceptionablc one; cf
Gai fman 1979, p. 134, n.4).
The condition of equal (and hence zero) risk is, of coursc,
equal to that of equal (and hence zero) expected gain, a quantity
that Laplace termed advantage (1820, p. 20); thus fair odds are
those also that confcr cqual, mcaning zcro, advantage on each
sidc ofthc bet. Appeal to zero expected gain as the critcrion offair
exchanges has of course come in for a good deal of adversc com-
ment, with the celebrated St Petersburg problem alleged to show
the practice must cither be abandoncd on pain of inconsistency, or
amended to a corrcsponding condition of zero cxpccted utilitv.
This piccc of conventional wisdom is incorrect. True, thc zero-
expected-gain condition, plus the additivity of expectation,
implies that any finite sum paid for the famous offer gives the
advantagc to thc buyer (thc offer is payment of$2/l if the first head
in a scqucnce of indefinitely many tosses of a fair coin occurs at
the nth toss, for 11 = 1,2,3, ... ). But all that follows from anyone
actually bcing willing to pay a substantial sum is that they are
cither foolish or they have money to burn.
THE LAWS OF PROBABILITY 55
What does not follow is that the bet is unfair, and to conclude
otherwise is simply to contlate equity and preference. That these
are conceptually distinct is evident from the fact that J may prefer,
for a variety of reasons, to accept a bet at greater odds than those
I believe are the fair ones-I might actually want my opponent to
have the advantage, for example. Nevertheless the contlation is
routinely made. Explicit in Savage's claim that "to ask which of
two 'equal ' betters has the advantage is to ask which of them has
the preferable alternative" (1954, p. 63 )4, it has been a systemat-
ic feature of di scussions of subjective probability throughout the
twentieth century, undoubtedly assi sting the acceptance of the
orthodox view that defining a fair bet in terms of expected gain
commits the error of assuming that money is linear in value. Of
course it is not, but nothing in the zero-expected-gain account
implies that it is, or indeed anything at all about the value that
should be placed on a gamble (the point is made forcibly in
Hellman 1997). Savage's claim is, moreover, as he half-confesses,5
a very considerable distortion of the hi storical record. From the
early eighteenth century, the zero-expected gain condition has
been a legall y enforceable condition offairness in games of chance
in lotteries and casinos, any divergence between the odds and the
assumed chances being thought almost certain to be manifested in
the average gains in repeated gambles failing to tend to equality.
Condorcet pointed this out over two centuries ago by way of
defending the fairness of the St Petersburg gamble (Todhunter
1886, p. 393), and in general it seems to have been for a long time
regarded as a fact, certainly about repeated games of chance:
We ean actually ' see' the profits or losses of<l persiSlent gamble. We
naturall y translate the lolal into average g<lin and thereby ' observe'
the expectation even more readily than the prob<lbilit y ... Certainly
a gambler could notice that one strategy is in (J<llileo's words "more
advantageous" than another. (Hacking 1975, p. 92)
-l This is typica l within the orthodox approach: "the fair price of a wager is the
Sllm of moncy at which [the buyer] should be equally happy to have either a
straight payment ()f the SUIll or the wager itsel f" (Joyce 1999, p. 13).
" " Perhaps I distort hi story sOlllewhat in insist ing that ea rl y problems were
framed in terill s of choi ce among bets" ( 1954, p. 63). Quite so. But every suc-
cessful revolution tends to rewrite history in it s own favour, and the utilit y revo-
lution is no except ion.
56 CHAPTER 3
Be that as it may (the scare-quotes suggest less than complete con-
fidence on the part of the author), the discussion shows one impor-
tant sense of fairness of odds which demonstrably has nothing to do
with any consideration of preference. We cannot, however, invoke
presumptive tendencies in long sequences of independent repeti-
tions, if for no other reason than that most of the propositions we
will consider do not describe repeatable events at all: they are the-
ories and evidence-statements whose truth, or falsity, are singular
'events' par excellence. Nor does the surrogate of calibration seem
to otTer an acceptable solution. Your probability assignments arc
calibrated if, for each probability value p, the proportion of true
propositions in the class of all propositions to which you ascribe p
is sufficiently close to p . To rule out unrepresentative swamping by
many repetitions of the same event (like tossing the same suitably
weighted coin) some condition of independence is required, but this
and the other conditi ons needed on size and structure of the equi-
probable reference classes beg more questions than are answered.
Van Fraassen proves that calibration in some possi ble world ensures
obedience to the probability axioms (1983), and Shimony proves a
similar result in terms of 'reasonable' constraints on truth-frequen-
cy estimation (1993), but the assumptions required are so strong as
to deprive the results of most of their significance.
We nevertheless believe that the old idea that fai r betting quo-
tients are identified with personal chance-based odds represents a
sound enough basic intuition which, despite orthodox claims to the
contrary, can be maintained consistently with acceptance of the
phenomena of risk aversion, concavity of utility functions, and so
forth. The one which begs fewest questions of all , and would be the
one presented if our ai m were unalloyed rigor, is R.T. Cox's (we
shall describe it briefly later), but it also requires fairly sophisticat-
ed mathematics. At any rate, we shall continue with our much sim-
pler development based on the traditional idea of fair odds being
those corresponding to the agent's 'true' odds- traditional, that is,
until becoming (unjustly) a casualty of the utility revolution. This is
a fitting opportunity to pause and take a longer look at that revolu-
tion, whose tendenti ous redefinitions of fairness and equity we
have considered, and rejected as question-beggi ng. Even a rigorous
axiomatic development is no protection against questions begged
or, as we shall see in the next section, fundamental problems left
unsol ved.
THE LAWS OF PROBABILIN 57
3.e Utilities and Probabilities
Inaugurated in Ramsey's seminal essay 'Truth and Probability'
(1926), the utility-revolution was and still is widely regarded as
culminating successfully thirty years later in Savage's classic
work (1954). Savage analyses an individual's uncertainty with
respect to any proposition a in terms of how he or she ranks their
preferences for gambles involving a. Thus, if two gambles with
the same payoffs but on different propositions are ranked differ-
ently it is assumed to be because the agent thinks one of the
propositions more likely than the other (an axiom states that this
preference must be independent of the absolute magnitudes of the
payoffs). Given suitable constraints on the agent's preference
ranking, this relation determines what Savage calls a 'qualitative
probability' ordering on propositions which, within a sufficiently
big space of possibilities (it must be infinite), is representable by
a unique probability function.
Secured on what seemed like a rigorous axiomatic base, this
utility-based account of epistemic probability by philosophers
became so dominant after the publication of Savage's text that it
is fair to call it now the orthodox account. It is certainly so among
philosophers, promulgated in Earman's declaration that "degrees
of belief and utilities have to be elicited in concert" (1992, pp.
43--44), and Maher's "You have subjective probability function p
just in case there exists a utility function u such that your prefer-
ences maximise expected utility relative to p and u" (1990, p.
382). Yet dissent is, we believe, well-founded. To start with, the
claim that a utility-based development is the only sure foundation
for a theory of personal probability is not true: the path via utili-
ty theory is actually far from sure. There is, in particular, a pro-
found problem with the way probabilities are supposed to emerge
from preferences. The gambles relative to which the agent's qual-
itative probability ordering is defined are a subclass of the class
of possible acts, and it is this more inclusive class on which the
preference relation is defined. Formally, they are fitnctions from
states to consequences (including constant functions which for
each consequence takes that consequence as value for all states),
and a critical assumption is that the consequences are capable of
being described in such a way that their value to the agent can be
58 CHAPTER 3
regarded as constant across possible states. This constancy
assumption is highl y unrcalistic/' and Savage himself acknowl-
edges that in practical situations the value-rclevant conse-
quences of our actions, as we represent them to ourselves at any
rate, will depend on states of affairs holding that as yet remain
uncertain: we necessarily contemplate what he calls 'small
worlds', whose states are coarsenings of the ' grand-world' states
that determine the ultimate consequences of our acts.
Knowledge of these being typically denied us, the consequences
we actually envi sage are therefore in reality 'grand-world' acts
with consequences indeterminate in terms of the 'small world'
states (1954, pp. 82- 91). Since to every' small world' act there
corresponds a unique 'grand world' act, it is a natural consistency
condition that the expected-utility ranking of the 'small world'
acts is consistent with that of their 'grand world' counterparts.
Yet, as Savage himself shows, there are 'small world'-'grand
world' pairs in which the preference ran kings are consistent with
each other but in such a way that the 'small world' probability
function is inconsistent with the ' grand world' probability
(1954, pp. 89-90).
While in itself thi s might seem, according to taste, either
merely mildly anomalous or something rather more serious, it
nevertheless portends something definitely morc to the latter end
of the spectrum. For a development of Savage's example by
Schervish et al. shows that the phenomenon of sameness of rank-
ing with di flerent probabilities can be reproduced entirely within
one 'grand world' (1990 p. 845). Their result is not, as it seems to
be, in contlict with Savage's well-known proof of the uniqueness
of the probability-utility representation, because that proof
assumes that the values attached to the consequences arc fixed
independently of the states. In the construction of Schervish et al.
this is actually true for each probability-utility representation sep-
arately, but the values assigned consequences in one representa-
tion are state-dependent in the other, and vice versa. While the
formal correctness of Savage's result is not impugned, therefore,
the fact nevertheless remains that altering the way consequences
(' CfAumann 2()()(), pp. 30506. Savage's reply to him is hardly convincing (ihid.
pp. 307- 310).
THE LAWS OF PROBABILITY 59
are valued can, at any rate in Savage's account1, actually alter the
probabilities elicited from the same set of preferences. Jeffreys
had earlier voiced scepticism about basing subjective probability
on revealed preferences among uncertain options, precisely on the
ground that such preferences are irreducibly "partly a matter of
what we want, which is a separate problem from that of what it is
reasonable to believe" (1961, p. 30), a judgment which now
seems fully vindicated. Schervish et al. themselves conclude that
the situation described above places "the meaning of much of this
theory . .. in doubt" (1990, p. 846).
The path from utilities to determinate probabilities is, para-
doxically, rendered even less firm by improving the conceptual
foundations of Savage's theory. Jeffrey's decision theory (1964),
published a decade after Savage's, replaces Savage's problematic
state-action partition with a more appealing single algebra of
propositions describing both the taking of actions and the possi-
ble consequences ensuing, generating thereby a modified expect-
ed-utility representation of preference (,desirability') whose
probability-weights are conditional probabilities of outcomes
given the action contemplated (in Savage's theory they are uncon-
ditional probabilities distributed over a partition of states). But a
well-known feature of Jeffrey's theory is that the desirability-
axioms permit a class of probability functions, and one so exten-
sive that it contains functions which disagree in their ordering of
the propositions.
It follows almost immediately that it is possible for an agent to
order propositions by degree of belief consistently with the
Jeffrey axioms in a way not representable by any single probabil-
ity function (Joyce 1999, p. 136). True, the class of functions can
in principle be narrowed down: the trouble is that all the attempts
to do so cause disruption elsewhere or beg the question. For
example, Bradley (1998) shows that uniqueness can be obtained
by adjoining to the underlying algebra of propositions a class of
conditionals satisfying the so-called Adams Principle (which says
essentially that the probability of a conditional is a conditional
probability). But then some famous results of Lewis (1976) imply
7 And not just in that: Schervish et al. point out that the same problem atnicts
Anscombe and Aumann's (1963) and de Finetti's (1974) theories.
60
CHAPTER 3
that the resulting logic must be non-Boolean, to say nothing of
these conditionals possessing very counterintuitive features: for
example 'If Jones is guilty then Jones is guilty' turns out to have
no truth-value if Jones is in fact not guilty. Jeffrey himself coun-
selled adding further axioms for qualitative probability orderings
(1974, pp. 77--78)8, and it is well-known how to do this to secure
uniqueness. But this strategy effectively sells the pass as far as the
programme for determining subjective probabilities by appeal to
properties of rational preference is concerned. The continued con-
centration of activity in the theory of rational preference should
not conceal the fact that one of its historically principal objec-
tives, the determination of personal probabilities by a suitable
elicitation of preferences
9
, is if anything farther from being
achieved than ever.
Not only do they seem to provide no secure foundations for
personal probability: decision theories like Savage's rather notice-
ably fail to discharge the function they themselves set, which is,
typically, identifying criteria for making optimal choices among
possible actions. Savage's theory is explicitly a theory of rational-
ity, of "the behaviour of a 'rational' person with respect to deci-
sions" (p. 7). The scare-quotes signify a degree of ideal isation in
the theory Savage is about to develop; to this end he calls it a
model. But a model of what? It is increasingly appreciated that the
theory's prescriptions are, at any rate in most interesting applica-
tions, beyond the capacity of even the most rational human agent
to obey, a feature remarked in observations to the effect that the
theory assumes an in-principle impossible "logical omniscience"
(Earman 1992, p. 124), and that its characteristically sharp (point)
probability-values are completely unrealistic. The objection is not
substantially deflected by appeal to the idealising character of
Savage's theory. A model of rational behaviour that makes no
allowance for the fact that people are highly bounded reasoners,
using irreducibly vague estimates of probability, is an inadequate
S A strategy endorsed by Joyce: "The bottom line is that JcfTrey-l3olker axioms
need to be augmented. not with further constraints on rational preference, but
with further constraints on rational belief" (1999. p. 137).
" It was the explicit goal of Ramsey. who proved. or more accurately sketched a
proof oC the first representation theorem for utilities.
THE LAWS OF PROBABILITY 61
model; hence the various attempts over the last forty or so years
to develop what Hacking called "slightly more realistic personal
probability" (1967).
What these last objections reveal, we believe, is not that the
standard formalism of personal probability is insufficiently
human-centric, but that it is misidentified as a model, even a high-
ly idealising one, of a rational individual's beliefs. On the other
hand, if it is not that, what is it? It is suggestive to ask what other
well-known theory has as its domain a full Boolean algebra of
assertions (up to equivalence) and rules for distributing values
over its elements. Answer: deductive logic. Nobody (presumably)
would regard this as the theory of the mental workings of even an
ideal human reasoner; it is a theory of necessary and sufTieient
conditions for inferences to be valid and sets of sentences to be
consistent, whose connection with human reasoning exists only in
the occasional ability to map in a more or less faithful way infer-
ences actually considered into the set defined and evaluated with-
in the model. The formalism ofepistemic probability suggests, we
believe, a similar conclusion: that formalism is a model of what
makes a valid probabilistic inference, not of what ideally rational
agents think or ought to think in conditions of uncertainty. Of
course, so far the kinship between epistemie probability and logic
is no more than suggestion, and certainly does not amount to a
proof, or even a semi-rigorous argument, to that effect. The differ-
ences between probability-values, inhabiting as they do an inter-
val of real numbers, and the two truth-values 'true' and 'false', as
well as their difference of interpretation, shows that any proof of
identity between the two models is out of the question. But
acknowledging their distinctively probabilistic features does not
mean that in some relevant sense the laws of probability are not
sufficiently close to those of deductive logic to merit being
assigned the status of logic.
Indeed, we believe such a logical interpretation is not only
possible (and the rest of this chapter will be devoted to arguing the
case for this) , but that 011(1' such an interpretation is capable of
accounting in a natural way for those features of the formalism of
epistemic probability which arc otherwise highly anomalous. For
example, on the logical view the charge of 'logical omniscience'
is clearly misplaced, for there is no knowing or reasoning subject,
62 CHAPTER 3
even an ideal one, appealed to or legislated for. The use of 'sharp'
probability values is also easily justified as a simplifying move
analogous to the adoption of 'sharp'-that is, strictly two-val-
ued-truth-values in the standard deductive models. These latter
would also be judged unrealistic were their function that of mir-
roring accurately the semantics of natural languages where, as the
many varieties of Sorites demonstrate, truth is typically vague if
not actually fuzzy (nor is even mathematics, that alleged para-
digm of precision, immune, a fact most notably pointed out by
Lakatos, who used the vagueness in the concept of a polyhedron
as the starting point from which he began a celebrated investiga-
tion into the foundations of mathematics (1963); and opinion is
still far from unanimous on the meanings of 'set' , ' function' ,
'continuous', 'number', and other basic mathematical notions).
Their role is not that of mirroring natural-language semantics,
however, but of playing a central role within simplifying models
of valid deductive inference whose payoff is information, about
the scope and limits of deductive systems: witness the signifi-
cance given the classic theorems of G6del, L6wenheim, Skolcm,
Church, Tarski, Cohen, and others. Similarly, sharp probability
values are justified by the explanatory and informational divi-
dends obtained from their use within simplifying models of
uncertain inference. And as we shall see in the remainder of this
book, these are very considerable.
One-indeed the principal-question we must answer is why
the probability axioms should be regarded as constraints on the
distribution of fair betting quotients. One well-known way of try-
ing to justify the probability axioms within a general betting-odds
approach appeals to a purely arithmetical result known as the
Dutch Book Theorem. First proved by de Finetti (1937), its con-
tent is this: a function P whose values are betting quotients can-
not generate, for any set of stakes, a positive net loss or gain
independently of the truth-values of the propositions bet 011, (fand
only P satisfies the/inite!y additive probability axioms. A system
of betting quotients invulnerable to such a forced loss de Finetti
termed 'coherent', usually translated into English as 'coherent'
though it is also translatable as 'consistent'. Thus the theorem
states that coherence for belief-functions measured in this way is
THE LAWS OF PROBABILITY 63
equivalent to their satisfying the probability calculus. For a sim-
ple proof of the thcorem see Gillics 2000, pp. 59-64.
As mathematics, the result is unquestionable. What is prob-
lematic about it is why being a guarantee of invulncrability to a
forced loss should authorise the probability axioms to constrain
the distribution of personal fair betting quotients whose defini-
tion, as we have taken pains to stress, implies no propensity of the
agent whose beliefs they characterise to bet at the corresponding
odds, or even to bet at all. There is an answer to this question, as
we shall see later in section e, but at the present stage of the dis-
cussion it is not obvious. In what follows we shall try a different
approach.
3.d Consistency
Ramsey and de Finetti both regarded the probability axioms as
consistency constraints. However, de Finetti identified proba-
bilistic consistcncy with coherence (1937, p. 102), which rather
begs the question, while Ramsey saw consistency as a property
of sets of preferences. That way, of course, is the orthodox way
of utility theory, which we have repudiated. Nevertheless,
despite this unpromising start, we shall persist with the claim
that the probability axioms are laws of consistency. Indeed, we
shall argue that they are laws of consistency in very much the
same way that the laws of deductive logie are laws of deductive
consistency.
Now this might seem an even more unpromising proposal than
either de Finetti's or Ramsey's, since to start with deductive con-
sistency is a property of sets of sentences, not of assignments of
numerical fair betting quotients to propositions. The proposition-
sentence difference is not the problem here, since for our purpos-
es they can be regarded as the same things. The problem, or what
seems to be a problem, is that in one case consistency is predicat-
ed of assignments of number-values, and in the other of sets of
sentences or propositions. Perhaps surprisingly, this gulf is less
formidable than it looks. At any rate, it is easily bridged by recog-
nising that the apparently distinct notions are merely subspecies
of a single more general concept, the familiar mathematical
64 CHAPTER 3
concept of consi stency defined as the solvability o/sets 0/ equa-
tions. To see why thi s is so, firstly note that a set of assignments
of number-values to a Boolean function is in effect a set of equa-
tions. Thus, considering a consistent set K of assignments to a
belief function Bcl( ) and a conditional beli ef function 8el( I ),
Paris writes
Consistent means that, with whatever additional conditions current-
ly appl y to Bel( ), Bel( I ), there is a solution satisfying these and the
equations in K. ( 1994, p. 6) 10
Thc next step consists in observing that deductive consistcncy is
also rcally nothing but solvability in this sense. This is easiest to
see in the context of a popular deductivc system for first order,
and in particular propositional logic, the semantic tableau or tree
system.
This is specifically designed to be an efficient test for the
deductive consistency of sets of sentences. Here is a simplc exam-
ple: we want to test the set {a, a ~ b, ~ h } for consistency. The
test consists of writing these sentences so
a
and beneath ~ h appending the tableau rule for the unnegated con-
ditional a ---c"> h
/ \
~ b
We have two branches from the root, on both of which are a sen-
tence and its negation. The tree is now closed, signifying that the
set is inconsistent, and the test is complete (the classic text is
Smullyan 1968; more elementary texts are Jeffrey 1989 and
Howson 1997) .
10 Pari s takcs K morc gcncra ll y to bc a sct of I in car constrai nts on belieffunctions.
THE LAWS OF PROBABILITY 65
Essentially the same tableau, or tree, could however have been
written in the 'signed' form
v(a) = 1
v(a --;. b) = I
v (b) = ()
II
v(a) = () v(b) = 1
(i)
(ii)
(ii i)
where I signifies truth and 0 falsity, and v is a truth-valuation
function II. What we see here is most revealing: the signed tree
shows that the initial assignment represented by the equations
(i)-(iii) is overdetermined. Those equations are unsolvable over
the set of propositional variables: there is no assignment of values
to a and b which satisfies them. Conversely, had the initial set
been consistent, a corresponding complete and open (signed) tree
could be constructed which would exhibit at least one single-val-
ued extension of the initial valuation to all the sentences of the
propositional language (Howson 1997, pp. 57-60). We should
note that this phenomenon is not restricted to propositional logic:
it is straightforward to show that signed trees perform an analo-
gous role for full first order logic.
We see, then, that the signed tree method shows explicitly that
the consistency or inconsistency oja set ofsentcnces is eqllivalent
to the saris/iability or Llilsatisjiabili(v ojsets ojequations. It should
also be evident that all signed tableaux can be converted into
unsigned ones (by first removing all the truth-values and then
negating any sentence to which 0 had been assigned), and converse-
ly. In other words, we see that the ordinary concept of deductive
consistency of a set of sentences in some interpretative structure is
equivalent to the solvability of an assignment of truth-values.
We can now paraphrase Paris's definition of a consistent set
K of assignments appropriately for this deductive example: K
(here the set {(i), (ii), (iii)}) is consistent if, with whatever addi-
tional conditions apply to v(), there is a solution satisfying these
I I Smullyan 1968, to whom the definition of signed tableaux is due, signs them
with T and F; we equate the sentences to 1 and 0 to emphasise the equational
character of the tableau.
66 CHAPTER 3
conditions and the equations in K. It follows that truth is involved
in the concept of deductive consistency o n ~ v via the constraints,
and suggests that corresponding to different types of constraint on
'values' attached to propositions we shall obtain different types
of logic: so for example the logic of consistent assignments of
truth-values subject to the constraints given by a classical truth-
definition is deductive logic; the logic of consistent assignments
offair betting quotients determined by appropriate constraints on
them we can legitimately call probability-logic.
The Big Question, of course, is what these 'appropriate con-
straints' are. We already have one: the scale of probabilities, since
we are dealing with normalised odds, which occupy the closed
unit interval (and add over pairs a, ~ a . And we also have the
primitive idea that those numbers represent fair betting quotients,
reflecting assessments of the agent's 'true' odds. The immediate
question is where to go from there.
A clue is suggested by the fact that since we are discussing
consistent distributions of probabilities, we should presumably be
looking for some collective condition or conditions. But what col-
lective condition? Joyce has recently argued that the probability
axioms follow from the condition that an overall assignment is
'truth-directed', but the conclusion depends on representing ' true'
by I and 'false' by 0 (1998): reverse this entirely conventional
proxying, and the result fails. But let us try to exploit the idea of
fairness a bit more. A very plausible condition is that fair gambles
should not become collectively unfair on collection into a joint
gamble, and it also points in the desired direction since (a)-(c)
below show that the laws of probability pretty well all follow from
the condition that a finite set of fair bets is fair (de Finetti first
showed this in his 1972, p. 77). A problem is that if the criterion
of fairness is zero-expected gain, we have as yet no information
on how to aggregate expectations on sums of bets; that they add
over finite sums of random variables is a consequence of the
finitely additive probability axioms, but to invoke the axioms at
this stage would clearly beg the question.
However, there is nothing to stop us adding as an independent
assumption that the class of fair bets, bets whose betting quotient
is the agent's fair betting quotient, is closed under finite sums. De
Finetti himself does so, in the form of a definition of fairness for
finite sums of bets (ibid.) For him, however, this 'definition' is in
effect a consequence of insisting monetary stakes be kept small:
THE LAWS OF PROBABILITY 67
in the limit of small sums utility-gains are equal to monetary ones,
and utility functions compensate for the departures from additiv-
ity (which he calls 'rigidity' [1974, p. 77]) caused by risk-aversion
where the stakes are in monetary units. But that, of course, is a
utility-based justification. Our own, where no condition is placed
on the size of the stakes, is merely that it is a natural assumption
that chance-based odds do not 'interfere' when combined in finite
sums of bets, one arguably as analytic in its way as the deductive
assumption that a conjunction is true just in case all its conjunc-
tions are. We do not even need the principle in its generality, only
the following special case:
(C) If the sum of finitely many (and in fact we never need to
exceed two) fair bets uniquely determines odds 0 in a bet
on proposition a, then 0 are the fair odds on a.
(C stands for 'Closure', that is, the fact that it is a closure princi-
ple on fair bets; "uniquely" means 'independently of choice of
stakes'; on those occasions when we will use (C) it is not difficult
to see that the condition is satisfied.)
3.e The Axioms
To keep things simple we shall now follow de Finetti 1972 in
identifying a proposition a with its indicator function: i.e. the
function taking the value 1 on those states of affairs making a true
and 0 on those making a false. This means that from now on we
are dealing with an algebra (of indicator functions). The random
quantity (de Finetti's terminology) 5(a - p) then represents an
unconditional bet on/against a with stake 5 (5 positive for 'on' and
negative for 'against') and betting quotient p, paying 5(J - p)
when a is true and - p5 when a is false. A bet on a conditional on
the truth of another proposition b is a bet on a if b is true, called
off if b is false. If the stake is 5 it is therefore the quantity b5(a-
p). p is your conditionalfair hetting quotient if it is your fair bet-
ting quotient in a conditional bet.
The following facts are easy to establish:
(a) If p is the fair betting quotient on a then 1- p is the fair
betting quotient on ~ a This anyway we know to be the
68 CHAPTER 3
case, since as we saw earlier p and I - p arc the normali-
sations of the agent's chance-based odds.
(b) If a and b are disjoint with fair betting quotients p and q
respectively then p + q is the fair betting quotient on a v
b.
(c) If p and q are the fair betting quotients on a&b and b
respectively, and q > 0, then plq is the conditional fair
betting quotient on a given b.
Proof:
(a) S(o - p) = -S(-a - [/ - p}). Now use (C).
(b) S(a - p) + S(b - q) = S(avb-(p + q)). Now use (C).
(c) If both p,q > 0 then there are nonzero numbers S. T, W
such that S(a&b - p) + T(b - q) = a W(o - p Iq) (TIS must
be equal to p Iq). Now use (C). The restriction to p > 0 can
be eliminated by noting that if p = 0 then S(a&b - p) =
Sa&b = Sab = bS(a - 0).
A point of interest is that (C) can be seen as a more general addi-
tivity principle: (b) tells us that it underwrites the familiar addi-
tivity principle for probabilities, but (c) tells us that it also
underwrites the quotient ' definition ' of conditional probabili-
ties. This kinship between two of the fundamental laws of prob-
ability usually regarded as bcing di stinct from each other is of
interest whatever view one might take of the naturc of probabi 1-
ity itself.
We now come to the main result of this chapter. Let m be some
algebra of propositions, assumed fixed for the following discus-
sion (as we shall see shortly, the result does not depend on the par-
ticular choice of m), and let K be an assignment of personal fair
betting quotient s (including conditional betting quotients) to a
t
'" b 1)\ flP
Illite su set . '0 0 at.
Theorem: K can bc extended to a single-valued function F on
m, wherc F satisfies (C), if and only if K is thc restriction of a
finitely additive probability function on m.
THE LAWS OF PROBABILITY 69
Prool
We shall actually prove something slightly stronger, namely
that the (nonempty) class of all single-valued extensions of K
to m which satisfy (C) is exactly the (nonempty) class of
finitely additive probability functions on m which agree with
K on mo'
(i) 'Only if'. It follows almost immediately from (a)-(e) that
any extension of K satisfying the conditions stated is a
finitely additive probability function on m (the additivity
principle is obvious, and we also have that the logieal
truth t is equal to a v -a for any a, and P(a v -a) = P(a) +
P(-a) = praY + 1 - praY = 1; the corresponding condition
that P(J..) = P(a&-a) = follows immediately).
(ii) 'If'. Suppose that K is the restriction of some probability
function P on m. It is sufficient to show that if p / .... ,Pk
are the values given by P to a /' ... ,a
k
, and a sum of bets
XI' ... ,X
k
at those betting rates is a bet on some proposi-
tion a with betting quotient P, then p = PraY. For then P
itself will be such a single-valued extension of K to m sat-
isfying (C). Suppose the antecedent is true. Since the
expectations of the X; with respect to P are all 0, it follows
that the expected value of Z = .LX; is 0 also, by the linear-
ity of expectations. Hence if Z = S(a - p) for some propo-
sition a then the expected value of the right hand side
must be zero, in which case P = pray (it is straightforward
to show that the result continues to hold if one or more of
the bets is a conditional bet). QED
Corollary 1 K is consistent just in case K is the restriction of
a finitely additive probability function.
We can now push the analogy with propositional logic even far-
ther. Define a model of a consistent assignment K to be any
single-valued extension of K to m which satisfies (C), and say that
an assignment K' is a consequence of another assignment K just
in case every model of K is a model of K '. We then have:
70 CHAPTER 3
Corollary 2 K'is a consequence of K just in case every prob-
ability function which extends K assigns values to members of
1ft as specified by K '.
Discussion. To say that? is a probability function over 1ft is to
say that the pair (1ft,?) is a system implicitly defined by the famil-
iar axioms of mathematical probability: the probability calculus is
just their deductive closure. Indeed, having now situated the enter-
prise of uncertain reasoning firmly within the province of logic,
we can construe those axioms as logical axioms, mediating the
drawing of consequences from specific sets of premises, which
we can regard as assignments like K (rather like in the signed
semantic tree representation of deductive inference, where the ini-
tial sentences are also value-assignments). A further corollary is
that valid uncertain reasoning, construed as reasoning involving
personal probabilities, is formally probabilistic reasoning-rea-
soning in accordance with the rules of the probability calculus.
Indeed, suppose that we can construct a representation of 1ft in a
deductive system, in which we can also express the ideas of a
probability system, and of an arbitrary assignment of real num-
bers to members of 1ft. Such a deductive system is a probability
calculus based on 1ft whose axioms, since they serve only to medi-
ate a consequence relation, can accordingly be regarded as having
logical status: they are in effect logical axioms in a theory of valid
probabilistic consequence.
And we can say more. The only 'facts' about the general prop-
erties of a probability function on an algebra are the logical con-
sequences of a suitable axiomatisation, together with a suitable
representation of the algebra. In a first-order framework these log-
ical consequences are just the formally provable consequences, in
which case the theorem above can be seen as a soundness and
completeness theorem fc)r probabilistic conseqllence: it implies
that K' is a consequence ol Kjust in case there is a prool of K'
fimn K in the corre:-,pondingformal calclilus. A concrete example
is provided by Paris (1994, pp. 8 3 ~ 8 5 , where the domain of the
agent's uncertainty is the set SL of sentences in the language L
generated by a finite set of propositional variables under the con-
nectives &. v, and -. Paris presents a formal deductive system n
for sequents of the form KI K' (to be read "K entails K ' '') , where
THE LAWS OF PROBABILITY 71
K and K' are now finite sets of linear constraints (i .e. sets of the
form
k
L uijB(a) = v"i = 1, ... , m

where B(a.) is the agent's probability assignment to a, and
j )
where the axioms and rules of inference are explicitly specified.
Paris then proves the following theorem:
For any pair K, K', the set alaI! probabilityfimctions extend-
ing K to SL is contained in those extending K' only
sequent K I K' is a theorem a/' n.
(We can straightforwardly adapt the result to a corresponding
algebra of propositions, or their indicator functions, by the usual
device of identifying logically equivalent members of SL.) Since
the antecedent of Paris's theorem is equivalent to the condition for
probabilistic consequence as defined above, that theorem is
indeed such a soundness and completeness theorem relative to an
explicitly-defined syntactical structure.
In the light of this discussion there seems no longer any room
for doubt that the probability calculus is interpretable as, and we
would argue should be interpreted as, a set of consistency con-
straints on distributions of personal probability in a way strongly
analogous to the way that standard deductive rules are consisten-
cy constraints on truth-value assignments.
Corol/ary 3 K is consistent if and only if no system of bets
with odds given by K will lead to certain loss.
This follows immediately from the result above and a sharpening
of the Dutch Book theorem by de Finetti (this states that a non-
negative real-valued function F on mo can be extended to a finite-
ly additive probability function on any algebra including mo iff
the betting quotients obtained from F are coherent on mo (de
Finetti 1972, p. 78; the proof is by induction on the well-ordered
set obtained from mo by adding an arbitrary proposition at each
stage; inn' is uncountable the proof uses transfinite induction on
72
CHAPTER 3
a well-ordering of ffi', for the existence of which the Axiom of
Choice has to be assumed}.
Disclission. While coherence is now not a primary, or in itself
any, justification of the probability axioms, merely a corollary of
the consistency of the assignments of fair betting quotients, this
result is of considerable importance from another point of view,
namely sanctions. It is precisely the issue of sanctions for viola-
tion of the rules of the probability calculus that might be thought
to present a big disanalogy between probabilistic and deductive
logic. Disobeying deductive rules of consistency invites the pos-
sibility of accepting a set of statements as simultaneously true
which in fact cannot be. This is, of course, a feature peculiar to
deductive logic, though it is one whose importance is, we believe,
greatly exaggerated. The constraints usuall y regarded as deter-
mining deductive consistency are the rul es of a classical truth-def:'
inition. Yet in ordinary discourse these rules are frequentl y and,
depending on the case, systematically violated- fortunately: oth-
erwise we could immediately infer the existence of God from the
apparently true premise that it is not true that if He exists we arc
free to do as we like. That the negation of a material conditional
implies the truth of its antecedent is an artefact of its definition as
a bivalent truth-function, resulting in classical first-order logic
providing a notoriously poor model of conditional reasoning (this
is a fact that is not emphasised in the standard textbooks, includ-
ing one by one of the authors, Howson 1997). And that is merely
an extreme case: the commutativity of conjunction is frequently
'infringed' without people declaring the speaker inconsistent: for
example, ' she got married and had a baby' does not necessarily
mean the same as 'she had a baby and got married'. These facts,
together with the development and serious candidacy of certain
types of non-class ical deductive logic, suggest that it is merely
na'ive to beli eve that deleterious consequences are any more like-
ly to flow from accepting a classically inconsistent set of sen-
tences. For that matter, none of our current foundational scientific
theories, including that for mathematics itself, are or even can be
known in any non-question-begging way to be classically consis-
tent: Godel's famous Second Incompleteness Theorem states that
there is no consistency proof for them which does not require
THE LAWS OF PROBABILITY 73
stronger premises than they themselves possess--unless they are
inconsistent.
So, surprisingly for those who accept the conventional wis-
dom, the case that failure to the canons of deductive consistency
placcs the violator in any sort of jeopardy has yet to be made, and
in view of the preceding observations it is most unlikely that it
ever will be. By contrast (and turning the conventional wisdom on
its head), the corollary above shows that there are much more pal-
pable sanctions against violating the rules of probabilistic consis-
tency. Personal probabilities determine fair betting quotients, and
being inconsistent in the use of these means potentially inviting
certain loss. It is no objection to say that the potential for certain
loss is merely that, a potentiality: also 'merely potential' are the
deleterious consequences supposedly arising from accepting an
inconsistent set of sentences.
Coro//ary 4 If K is consistent then it is so independently of
the particular m chosen; that is, K has a single-valued exten-
sion satisfying (C) to any algebra which includes mrr
This follows immediately from the theorem of de Finetti stated in
Corollary 4. Note that it is certainly not the case ifone substitutes
'countably' for 'finitely', since it is well-known that not every
subset of the unit interval, for example, is measurable (assuming
the Axiom of Choice).
Discussion. There is an important analogous property of deduc-
tive consistency: the consistency of a truth-value assignment
depends only on its domain; equivalently, a set of sentences is
deductively consistent, or not, independently of the constitution of
the language in which it is included. This local character of deduc-
tive consistency, and by implication deductively valid inference, is
therefore mirrored in the account of probabilistic consistency
(and probabilistic consequence; see the following corollary). The
importance of focusing on local assignments to sets of proposi-
tions which are not necessarily, or even usually, closed algebras,
is always stressed by de Finetti in his writings, and it is no acci-
dent that he often chose a logical or quasi-logical vocabulary to
describe his results: he certainly saw what he was doing as a part
of a more general theory of logic. It is no accident either that
74 CHAPTER 3
characteristic properties of deductive logic carryover to its prob-
abilistic analogue, and the fact that in Carnap's early systems of
inductive logic the 'degree of partial entailment' between hand e
did depend on the structure of the language of which hand e were
a part is a significant factor against them.
Historical Note. Most people who have tried to construct logical
theories of probability regard them as assigning probabilities to
formulas in a formal language. Interesting results have come out
of this research. For example, Gaifman proved a 'logical' ana-
logue of the Extension Theorem for measures: he showed that any
probability function defined on the quantifier-free sentences of a
first order language L with denumerably many constants has a
unique extension to the full set of sentences of L, which satisfies
the condition that the probability of an existentially quantified
sentence is the supremum (in the corresponding Lindenbaum
algebra) of the probabilities of its instantiations with constants
(Gaifman 1964; this condition is now known as the Gaifman con-
dition). Similar results were proved for infinitary languages by
Scott and Krauss (1966), and 'logical' analogues of Bayesian con-
vergence-of-opinion theorems have been obtained by Gaifman
and Snir (1982).
Our view is that the assignment of probabilities to formal sen-
tences or more generally formulas is neither a necessary nor a suf-
ficient condition for an authentically logical interpretation of
probability. It is not sufficient, because the case for making epis-
temic probability authentically logical arguably requires finding
some way of interpreting the rules of assignment, the probability
axioms, as recognisably logical rules, and this is not done simply
by assigning non-negative real numbers to formulas in a way that
satisfies the probability axioms. Nor is it necessary, we believe,
because the account given above provides an authentically logical
interpretation of the probability calculus. It is actually quite com-
patible with the numbers-assigned-to-formal-sentences one, since
we can if we wish take the algebra of propositions to be the sets
of realisations (models) of the corresponding sentences (the alge-
bra is then isomorphic to the Lindenbaum algebra of the lan-
guage), but it is much more general in that it includes in its scope
propositional structures not explicitly linguistic in character.
THE LAWS OF PROBABILITY 75
Standard formal languages are actually rather restrictive when
it comes to describing mathematical structures. First-order lan-
guages are well-known to be deficient in thi s respect, being
unable to characteri se up to isomorphism even very simple struc-
tures like the natural numbers, or indeed any infinite structure at
all, so the increase in determinateness provided by hi gher-order
languages has to be traded against the loss of a complete axioma-
ti sati on of the underl ying logic.
The logical perspecti ve provides compelling answers to some
of the most frequentl y raised problems, or what are taken to be
problems, in the literature on Bayesian probability. One of the most
prominent of these is the so-called 'problem of logical omnis-
cience'. This is that the incorporation of deductive conditions in
the probability axioms- for example the conditi on that all logical
truths have probability one, and all logical falsehoods have proba-
bility zero-means that the Bayesian agent has to be ' logically
omni sci ent' to apply those axioms correctl y. Thi s is a serious prob-
lem if the Bayesian theory is supposed to be a model of rational
agents, but no problem at all , as we noted carl ier, if the formal ism
of Bayesian probabili ty is seen merely as a model not of reasoners
but of valid reasoning in i t s l t ~ in which the axioms are consisten-
cy constraints in much the same way that the axioms of first order
logic are consistency constraints, or in other words as constraints
on the consistent distributi on of certain quantities independently of
any agent's thought-process or cognitive abilities.
These observati ons also effectively di spose of another fre-
quentl y-made objecti on, that the Bayesian theory requires that all
the possibl e hypotheses, and all the possi ble evidence statements,
one might ever consider must all be expl icitly present in any cal-
culation. The theory was invcnted by working scienti sts (we havc
mentioned Bayes, Laplace, Poincare, Jeffreys, Jaynes, Cox,
Savage, Lindley and ot hers) to help them codify, understand and
justify the principles they appeared to be applyi ng in making
inferences from data, and it seems hardly likely that they would
have produced a theory impossible in principl e to use. At any
given time there may be a limited number of hypotheses in play
that one would want to eonsidcr seriously, or find relevant to the
problem at hand. Probability is distributed over these, with per-
haps some kept in hand to be assigned to 'other causes' when and
76 CHAPTER 3
if these ever come to be considered (this is the view propounded
by Abner Shimony under the name of "tempered personalism"
(Shimony 1993, pp. 205-07). It may be that new information, or
merely more recent consideration, causes one to want to redistrib-
ute these probabilities, and not just by conditionalisation (that is,
after considering new data); perhaps the information is logical ,
and one sees logical dependencies where before one failed to; per-
haps new hypotheses are invented, or discovered, but then again
perhaps not, and one just feels dissatisfied that one 's original
assignments reflect what reflection deems they should. Well and
good; nothing in the Bayesian theory says that this should not be
allowed; it would be absurd to insist on any such statute of limi-
tation, and the theory does not do so. Its propositions are there to
be consulted, but in a sensible way, understanding that, just like
those of deductive logic, they are a servant and not a master.
3.f i The Principal Principle
Relative to the constraints that fair betting quotients I ie in the
closed unit interval , together with the collective condition (C), the
probability axioms are necessary and suffici ent conditions for
assignments of personal probability to be consistent. But there is
one aspect in which the constraints themselves are incomplete.
Suppose that a describes one of the possible outcomes of an
observation, over which there is a well-defined objective proba-
bility di stribution P*. Our assumptions (a) that objective proba-
bi I ities are numerical measures of tendencies scaled in the closed
unit interval , and (b) that personal probabilities reflect personal
assessments of relative likelihoods, strongly suggest that condi-
tional on your sole information that the data source possesses a
definite tendency to produce outcomes as measured by P*, your
personal probability of an a-type outcome should be equal to
P*(a). We can express this principle formally as follows . Let a be
a generic event-descriptor, and all be the prediction that a specif-
ic instance of a will be observed when suitable conditions have
been instantiated. Then
P(Q() I P*(a) = r) = r (2)
THE LAWS OF PROBABILITY 77
(2) is a version of the principle traditionally known (in the
English-speaking world) as the Principle of Direct Probability,
whimsically redubbed The Principal Principle by David Lewis in
a well-known paper (1 980).
It is (2) that enabl es posterior distributions to be calculatecL
via Bayes's Theorem, over values of these chances: without (2),
the subjective probability formali sm would be empty of any
methodological content. IndeecL one of the most striking results
obtained by using it provides the solution to a problem raised ear-
lier, with the promi se that it would be solved. Thi s is the problem
of how defining obj ective probabilities as limi ting relative fre-
quencies allows sampl e data, or records of what happens in finite
sequences of observations, to provide any information at all
about objective probabilities. After all, not only is any behaviour
in an initial segment of an infinite sequence compatible with any
limiting behaviour, but we can be assured that inf inite Kollekti vs
do not even exist, given that the lifetime of the Universe, at any
rate of that bit of it in which a coin, say, is repeatedly tossed, is
presumably finite. It seems frankly almost beyond belief that
such information could be provided. But it can, as we shall now
see.
There are threc steps in the explanation of how. Thc first is to
recall that, because thcy are defined in tcrms of limiting relative
frequencies, objective probabilities satisfy the axioms of the prob-
ability calculus. The second is also to recall that Kollektivs arc
Bernoulli sequences: the probability of any outcome at any point
is independent of the outcomes at other points, and is constant
from point to point. The third and final step is to use the resources
of our theory of epistemic probability. To this end, let h state that
the Xi are generated by a chance mechanism, in ot her wsords, for
some value of the chance that X = I, and h state that that chance
I I'
is p. Finally, let e(/: 11) be the statement that r ones are observed in
a sample of size 11 . Suppose that we have a fairl y smooth prior
density function h(p) for p, whose integral of h(p) between 0 and
1 is P(h) , since h just says that hI' is true for some value of p . By
Bayes's Theorem and the Principal Principle the posterior density
h(p I e(I:n)) is proporti onal to II C, p'(l - p)II - ' h(p). As 11 grows large
this density becomes dominated by the likelihood p' ( 1 - p) II , (for
large 11 the logarithm of the likelihood is approximately a constant
78
CHAPTER 3
times n), which is a maximum atp = rln and falls to zero very rap-
idly away from the maximum, leaving the posterior density of
points close to the relative frequency proportionally increased
more as the sample grows large. Anothcr way of putting this is
to note that the approximate likelihood ratio (rln)'(l--- (rln)),, - ' +
p'(l - p)" I increases without bound for p outside a small inter-
val which tends to 0 as n grows large, and if one accepts the plau-
sible Likelihood Principle (see Chapter 5, p. 156), which says that
the information gained from the sample is expressed in the likeli-
hood function, then the relative frequency in finite samples is cer-
tainly highly informative about the value ofp. Either way, we have
demonstrated, within the Bayesian theory of epistemie probabili-
ty, that finite sample data do provide information about a chance
parameter even ',vhen this paramete,. refers to an in-principle
unobservable limit.
The posterior probability of any interva l of values for p will
also depend on the prior probability P(h) of h, since the prior den-
sity h(p) must integrate to P(h) between the limits p = 0 and p =
1. P(h) , recall, is the prior probability that the data source will
generate a Koflektiv, and so it is the prior probability of a one-
dimensional statistical model, with undetermined parameter p.
Von Mises was extremely careful to base hi s theory on what he
took to be two already established empirical laws about random
phenomena: (i) that they are truly random, in the sense of being
immune to gambling systems, and (ii) that their relative frequen-
cies exhibit convergent behaviour. In SaFesian lams. von Mises
(Ihough he was not. at anv rate consciolls(v. a Sa.vesian) was in
effect {IIguingjor a considerable prior probabilitl ' TO be attached
10 the modelling hYPolhesis h. Thc posterior prohabili(v of h is of
course obtained by integrating the posterior density between the
limits 0 and I.
In the following chapters we shall apply the Bayesian tools to
more sophisticated, and also practically more important, statisti-
cal hypotheses than toy examples like h, involving models with
arbitrary numbers of adjustable parameters. But in terms of estab-
lishing one of the central claims of this book, that an appropriate
thcory ofepistemic probablity permits a demonstration of the fact
that finite sample data can indeed provide information about infi-
nite limits, the example above is anything but trivial.
THE LAWS OF PROBABILI TY 79
3.g I Bayesian Probability and Inductive Scepticism
David Hume produced a celebrated circul arity argument, that
any justification for believing ' that the future will resembl e the
past ' in any spec ified manner must expli citly or implictly
assume what it sets out to prove. We belicve (the claim is argued
at length in Howson 2000) that Hume 's argument shows infor-
mall y and in general terms that a valid inductive inference must
possess, in additi on to whatever observational or experimental
dat a is specified, at least one independent assumption (an induc-
ti ve assumption ) that in effect weights some of the possibilities
consistent with the evidence more than others. The Bayes ian
theory viewed in a logical perspective endorses this view in a
quite satisfactory way, and in so doing reveals a further illumi-
nating parallel with deductive logic. A nontrivial conclusion
(one which is not itselfa theorem of logic) ofa deducti vel y va lid
inference depends on at least one nontrivial premise. Simil arly,
a nontrivial conclusion (one which is not a theorem ofprobabil-
ity) of a valid probabilistic argument depends on one or more
nontrivial probabili stic premi ses. And just as the logical axioms
in Hilbert-style axiomatisations of cl assical logic are rcgarded
as empty offactual content because they arc universally valid, so
is the same true of the probabil ity axioms in the view wc have
been advocating. They too are logical axioms, empty of factual
content because uni versall y valid.
Putting all thi s together, we can deri ve a probabili stic ana-
logue of the celebrated conservation result of deductive logic,
that valid deductive inference does not beget new factual content,
but merely transforms or dimini shes the content already exi sting
in the premises. So too here: valid probabili sti c inference does
not beget new content, merely transforming or diminishing it in
the passage from premises to conclusion. Thi s was well under-
stood by de Finett i:
The ca/clI/lls o{probabilitl' call sav abso/ute/" nothillg ahout rea/-
itF . .. As with the logic of certa inty. the logi c of the probabl e adds
not hing of its own : it merely helps one to see thc implications con-
ta ined in what has gone before. (1974. p. 21 5; cmphasis in the
ori ginal)
80 CHAPTER 3
We can say more. The 'synthetic' premises in a probabilistic infer-
ence are generally prior, or unconditional, probabilities, and
because their exogenous nature is explicitly acknowledged within
the so-called subjective Bayesian theory they are often seen as its
Achilles heel. Hume's argument enables us to view them in a less
unfavourable light, for it implies that some degree of indetermi-
nacy is a natural and indeed inevitable feature in any adequatc
theory of valid inductive inference. Far, therefore, from being a
disabling feature of the subjective Bayesian theory, the exogenous
prior distributions, together with the partition of logical space
with respect to which they are defined (that is, the hypothesis-
space chosen), merely show where that indeterminacy is located.
3.h Updating Rules
There is probably no other controversial topic in the Bayesian the-
ory on which the logical perspective casts as much illumination as
that of so-called updating rules, and in particular that of the rule
of Bayesian conditionalisation. This rule is still widely regarded
as a fundamental principle on a par with the probability axioms
themselves, and a core feature of the Bayesian theory. We shall
deny that it deserves this status, but at the same time show that
under suitable conditions, which would be fulfilled in the sorts of
circumstances in which updating would be typically applied, the
rule is valid.
An updating rule tells 'Bayesian agents' how to adjust their
belief function globally when they acquire new evidence (the rea-
son for putting 'Bayesian agents' in quotes is because, as we have
said earlier, we do not think this is a theory of agents, Bayesian
or otherwise; it is a logic). There must, according to a widespread
view, be such a rule for doing this, otherwise there could be no
principled account of 'learning from experience'. The rule in this
case is this. Suppose that your probability of a just before the
time t, call it t -, at which you its learn its truth is ~ . (aJ. Your
new probability for a we can therefore write as ~ a ) , which is
presumably I. Obviously, you will have to change at least some
of your other probabilities to accommodate this change consis-
tently: for example, it might have been that ~ (h) < 1 where b
THE LAWS OF PROBABILITY
81
is some logical consequence of a, but consistency demands
that you now change this value too to I. Thi s prompts two ques-
tions: (i) are there any rul es whi ch will all ow one to make the
necessary changes consi stently, and (ii), if so, which should we
adopt? To avoid subscripts we shall henceforth write P for P, .
and Q for P,.
The answer to (i) is yes, and the rule almost universally recom-
mended in the Bayesian literature is this one, called the rule of
Bay esian conditionalisation:
If P(a) > 0, set Q( . ) equal to P( . I a) (3)
We already know (Chapter 2.e (14) and (15 that Q as defined
by (3) is a finit ely additive probability function and is therefore
a consi stent way of distributing probabilities. It is al so not di ffi -
cult to show that the single assignment Q(a) = I can be extend-
ed consistently in ways that do not satisfy (3) . So why should (3)
be adopted? There are various arguments in the literature. There
is, for example, the ' obvious ' reason that since P(e I a) is your
probability at time t- that c is true given a, then on learning a
and nothing el se your updated probability of c should be a.
Al so, (3) has some pleasing properti es: for example, that Q(b) =
I if b is entailed by a is now a consequence (Chapter 2e (13;
also, by (16), Chapter 2e, successively conditioning on e and
d is the same as successively conditioning on d and e, and
so on.
But there is also a simple, and compelling, reason why (3)
should not be adopted: it is inconsistent. Given the ' obvious'
argument for (3 ), and the fact that it is often advertised as itself
a principle of consistency, or of 'dynamic coherence ', this mi ght
seem a surpri sing claim. Nevertheless it is easy to prove.
Suppose that there is a contingent propositi on b of whose truth
you are P-certain; i.e. P(b) = I. Suppose also that for whatever
reason (the favourite example is that you are about to take a
mind-altering drug) you think it di stinctly P-possible that Q(b)
will take some value q less than I; that is, that P(Q(b) = q) > O.
Let a be 'Q(b) = q'. It follows by the probability calculus that
P(b I a) = I . But suppose at the appropriate time you learn a by
introspection; then Q(b) = q. IfyoLl conditionalise on a then you
82 CHAPTER 3
must set Q(b) = P(b I a) = 1.
12
Note that there is no conceptual
problem, nor any implicit appeal to 'second-order probabilities' (a
frequent but erroneous claim), in a statement like 'Q(b) = q' being
in the domain of an ordinary probability function: a makes a def-
inite factual assertion, just as much as 'The next toss of this coin
will land heads ' . In fact, a can be written in a formally unim-
peachable way as 'X" = q', where X/s) is the random variable,
defined on possible states of the world, whose value at s is the
value of your future probability Q of b.
Nor is it an objection that the shift to 'Q(b) = q' is not the
result of rational deliberation (whatever that may mean). Indeed,
such a consideration is rather obviously beside the point, which is
whether conditionalisation is a valid rulc. And it clearly is not: the
information that Q(b) = q cannot be conditionalised on in the cir-
cumstances described if you are consistent. Ramsey, in an of ten-
quoted passage, pointed out that (3) might fail ' for psychological
reasons' (1926, p. 180); what he did not seem to appreciate is that
it might also fail for purely logico-mathematical ones.
Invalid rules breed counterexamples. What is surprising in the
case of conditionalisation is that nobody seems to have realised
why it, and its generalisation to Jeffrey conditionalisation in the
case of an exogenous shift on any finite partition, is anything
other than a completely arbitrary rule when expressed uncondi-
tionally. Why should I unconditionally make Q( . ) equal P(. I a)
when pray shifts to Q(a) = I? Despite the 'obvious' argument
given above, there is actually nothing in the meaning of P(. la)
that tells me I should: for P(e I a) is my probability, given by.fil11c-
tirm P, of c on the assumption that a is true. Given that.f(J/' rea-
sons of" eonsistencv I now have to reject P, to say that my
probability of c should be equal to P(e I a) clearly begs the ques-
tion. Nor do the motley jumble of 'justifications' in the literature
for the rule succeed in doing anything else. The most widely
accepted of these is a so-called dynamic Dutch Book argument,
due to Teller ( 1973), who attributes it to David Lewis. This cer-
tainly shows that you would be fooli sh to announce a policy in
Ie Thus this is also a counterexample to the so-called 'Reflection PrincipJc ',
which says that P( . IO( . l = xl = x, 0 :5 X :5 I. Intuitively absur(L it has attracted
a disproportionate amount of discussion.
THE LAWS OF PROBABILITY 83
advance of lcarning a for setting Q(c) at a di fferent value to
P(e I a), and then give or accept bets at both betting quotients, but
the foll y is your willingness to bet in that way, not the updating
policy. Thus we see an interesting difference between ' dynamic'
and 'synchronic ' coherence: the former is entirely without proba-
ti ve significance, whil e the latter is merely a corollary of deeper
principles which do not rely on considerations of f inancial pru-
dence for justification.
Another alleged justification is that the updating function
given by Jeffrey's rul e, and hence ordinary conditionalisation
which is a special case, represents the smallest departure from the
initial distribution consistent with the shifted values on the parti-
tion, and hence should be the one chosen. Even if the antecedent
claim is true, and that turns out entirely to depend on the appro-
priate choice of mctric in function spacc, the conclusion simply
begs the question. An analogy with another putative (spurious)
deductive 'updating rule' is illuminating. Suppose at time t you
accept the conditional a -;> b and at t + I you learn (and hence
accept) a. The putative rule, which we might humorously call
'dynamic modus ponens' , says that if by t + I you have learned
nothing other than a you should acccpt h. Clearl y thc inference is
not deducti vel y vali d, since (to usc the terminology of Gabbay
1994) the statements have different label s. Indeed, it is easy, just
as it was in the probabili stic case, to exhibit choices of a and b
which actually make thi s spurious 'rule' yield an inconsistency.
For example, a mi ght be the negation -b of 17, so that a -;> b is just
a convoluted way of asserting -b (assuming that it is the material
conditional being used here), and learning a then in effect means
learning that a -;> b is false. In other words, learning a in this
example undermines the previously accepted conditi onal.
A similar undermining happens in the probabili stic counterex-
ample above: the initi al conditional probability of I of b given
Q(b) = q is undermined by learning the latter proposition, since
the conditional probability should then clearly change to q. This
is intuitively obvious, but in support we can cite the fact that a bet
on b conditional on a, i.e. on Q(b) = q, with unit stake and where
the betting quoti ents are determined by thc functi on Q, has the
following payoff table:
84
b Q(b) = q
T T
F T
F
I-q
-q
()
CHAPTER 3
This is of course the table of a bet on b given Q(b) = q with stake
I and betting quotient q.
We now have an explanation of why conditionalisation should
in suitable circumstances seem compelling. Those circumstances,
just as with ' dynamic modus ponens', consist in the preservation
of the relevant conditional valuations. The following probabilistic
analogue of a version of modus ponens is clearly valid (to obtain
the deductive rule replace b I a by a ~ b, let Q and P be valua-
tions with q in {O, I } instead of [0,1]):
Q(a) = I Q(b I a) = q
Q(b) = q
whence we obtain this conditional form of conditionalisation:
Q(a) = I Q(b l a) = P(bl a)
Q(b) = P(b I a)
(4)
But 4 ) : ~ validi(v is easilv seen to be derived from the ordinwy,
'cSynchronic', probabiliry axioms. Conditionalisation is thus not a
new principle to be added to the probability axioms as a 'dynam-
ic' supplement- indeed, that way, as we saw, lies actual inconsis-
tency- but a derived rule whose conditions of applicability are
given by the standard axioms together with assumptions which on
any given occasion mayor may not be reasonable, about relevant
conditional probabilities. As to a possible objection that the
Bayesian methodology relies crucially on conditionalisation, the
simple answer is that nothing of value can possibly be lost by
recognising that the limits within which any rule is valid.
The same conditions of validity, namely the maintaining of the
relevant conditional probabilities pre and post the exogenous
shift, are required for Jeffrey conditionalisation, or 'probability
kinematics ' as Jeffrey himself calls it. When an exogenous shift
THE LAWS OF PROBABILITY 85
from Pta) to Q(a) occurs on a proposition a, Jeffrey's rule is that
your new function Q should be defined by
Q( . ) = P( . I a)Q(a) + P(. l-a)Q(-a)
where Q(-a} is of course l-Q(a}.13 The equation above generalis-
es to arbitrary finite partitions. It is well known that the necessary
and sufficient condition for the validity of Jeffrey's rule (its valid-
ity rel ative to the 'synchronic' probability axioms) is that the fol-
lowing cquations hold: Q(. I a) = P(. I a), Q( . I-a) = P( . I -a).
Jeffrey conditional isation, despite alleged justificatory credentials
ranging from maximum-information principles (Chapter 9.d) to
commutative diagrams, 14 is no more absolutely valid than
Bayesian conditionalisation.
3.i The Cox-Good Argument
A compelling way of showing why the probability axioms can be
regarded as conditions for consistent uncertain reasoning is due
independently to R.T. Cox (1946, 1961) and U. Good (1950),
though as Cox's discussion was both earlier and more systematic it
is the onc wc shall brict1y describe here. Cox was a working physi-
ci st (he was Professor of Physics at Johns Hopkins University)
who sought to set out the fundamental laws of probabilistic reason-
ing independently of any particular scale of measurement (1961, p.
1). To this end he proved that any function on an algebra of propo-
sitions which obeyed these laws can always be rescaled as a prob-
ability function. To be more precise, suppose that M(a I b) is a
function where a and b are members of a set S of propositions con-
taining logical truths, falsehoods and closed under the operations
&, v, -, such that M takes values in an interval J of real numbers,
assigns the same values to equivalent propositions, and
U Jctlr cy's rule generalises straightforwardly to a simultaneous shift on the
members of a partition (i.e. on a set of exclusive and exhaustive propositions),
but not to a shift on any finite number of logically independent propositions.
1-1 Sec, for exampl e, Diaconis and Zabell 1982, Williams 1980, Shore and
Johnson 1980, and van Fraassen 1989, pp. 331-37.
86 CHAPTER 3
I. I c) =.f(M(a Ie))
2. M(a&blc) =g(M(alb&e), M(ble))
for any consistent e, wherefis strictly decrcasing and g is increas-
ing in both arguments and each is sufficiently smooth, i.e. satisfy
certain differentiability conditions ({must have continuous second
derivatives). The nub of Cox's proof is showing that the underlying
logical rules governing &, v and entail that f is associative in 1,
i.e. that/(.rf(v,z)) =.I(1(x,y),::), a functional equation whose gener-
al solution is of the form Ch(/(x,y)) = h(r). hev) where h is an arbi-
trary increasing continuous function. IS C is the value assigned to
certainty and can be set equal to unity. Thus using the rescaling
function h we obtain the so-called multiplication law of probabili-
ties, a form of axiom (4). Using the product form fort: Cox then
proved that g(r) must have the form (I - X 111) 1/ 111, and since the
product rule is al so sat isfied if h is replaced with hl11, we can with-
out loss of generality take m = I in both. From thi s it is a short step
to showing that there is an increasing (and hence order-preserving)
function h:/ [0, I] such that hM(a I c) is a conditional probabili-
ty Pea I c) on S. 16 Defining P' (a) = Pea I 1) where I is a logical truth,
it follows that P' obeys the unconditional probability axioms
together with the rule that P '(a & b) = P '(a I b)P '(b).
To sum up: Cox has shown that for any M satisfying I.and 2.
there is a probability function which orders the propositions
equivalently. This result might seem too weak to be useful : there
are infinitely many different algebras with the same total order-
ings of their clements but which admit quite di fferent represent-
ing probability functions. But there are additional considerations
which will determine the function uniquely. As de Finetti showed,
if the algebra is embedded in one permitting arbitrarily many
I' By taking logarithms t he general solution could equally wcll bc stated in the
form 11(/(\, . .1)) = H(x) + H(1).
1(, An objection to Cox's argument is that the associ at ivity of/is demonstrated
only for triples where.\ = ivl(o ll>&('&II) . . " = M{h lc&d). - /v/(e ld) for
arbitrary a.h.c.d, which certainly do not cx haust I' if thc domain of M is finite
(Halpern 1999). !\ reasonable response is that Cox is considering not only actu-
al but pO,l'sihle va lues of M in I for these arguments (this is implicit in the differ-
entiability assumptions ). Paris (1994, Chapter 3) gives a proof which does not
assume differentiability at all.
THE LAWS OF PROBABILITY 87
judgments of indifference over finite sets of alternatives, that
ordering will admit a unique representing finitely additive proba-
bility functions ( 1937, p. 101; Savage employs the same device in
hi s 1954, pp. 38-39).
Notice how formally similar these rules are to the Bool ean val-
uati on rules for propositional logic: they tell you that the values
(probability-values, truth-values) on two primitive Boolean com-
pounds depend functionally on the values on the components. In
the deductive case, since there are only two values in pl ay, it is
much easier for the rules to tell you exactly what the dependence
looks like. In the probabilistic case there is a continuum of values.
Nevertheless the negation case is very simple: the value on a
negat ion is a smooth increasing function of the value on the negat-
ed proposition. The deducti ve rule for conjunction assumes that a
Boolean valuation V, considered formally as a two-valued proba-
bility function, entails independence of the conjuncts: we have
V( A&B) = V(A)V(B). In the probabilistic case we must all ow for
the possibility of dependence, and Cox's axi om 2. does so in the
simplest possibl e way, by merely requiring that the joint 'proba-
bility' of A and B given consistent C depends only on the 'proba-
bility' of A given Band C and that of B given C.
It is indeed a profound result. We also have an answer to why
probability functi ons should be chosen as the canonical represen-
tati ves of belief functions satisfying l.and 2.: because of the facts
that additive functions have greater computati onal simplicity and
that they give a common scal e for objective and subjective prob-
abiliti es. The proof of Cox's result is not at all a trivial matter,
however, and for this reason among others we have chosen to
develop the argument for the probability axi oms in terms of con-
sistent distributi ons of fair bett ing quotients: these are more
familiar objects, and the proof that they generate the probability
axioms is elementary by comparison. That it requires a bit more
in the way of assumptions is the price paid.
3.j Exchangeability
We end this chapter by returning to the discuss ion which started
it. We have claimed that there are two distinct interpretati ons of
88 CHAPTER 3
the probability calculus, one as the formal theory of objective
probability, which plays a fundamental role in statistics, and more
generally in the natural, social and biological sciences, the other
as a system of consistency constraints on the distribution of per-
sonal probabilities. We have said quite a lot about the latter com-
pared with the former, because it is with epistemic probability that
we principally concerned in this book. But we shall conclude by
looking briefly at a very influential case made by de Finetti that a
separate theory of objective probability is redundant (he also
thought the idea of there being such things as objective probabil-
ities reprehensibly metaphysical).
To discuss his arguments we first need to look at what he
called exchangeable random quantities. Suppose that we are con-
sidering a possibility space W consisting of all denumerably infi-
nite sequences of Os and 1 s. We can think ofthis as the (ideal) set
of all possible outcomes in a sequence of indefinitely repeated
observations, where I and 0 record respectively the observation of
some specific characteristic. Suppose that P is a Bayesian person-
al probability function which assigns definite values to all propo-
sitions of the form X, = Xi' to be read as 'the ith outcome is Xi'
where Xi is 0 or 1. These variables are said to be exchangeable
(with respect to P) if the P-probability of any finite conjunction
x,! = Xi! & ... & X,1 = X
ik
remains the same for any permutation of
the x.
If
De Finetti regarded the notion of an exchangeability as provid-
ing a solution to what he saw as two outstanding problems: one is
the problem of induction, and the other is that of explaining why
a theory of objective probability is redundant. His solution of the
first is based on the second, and that stems from a celebrated the-
orem de Finetti proved about sequences of exchangeable vari-
ables. He showed that if a sequence like the X above is
I
exchangeable then the P-probability of any sequence of n of them
taking the value I r times and 0 the remaining n - r, in some given
order, is independent of that order and equal to
Jz '(1 - z)" 'dF(z) (I)
where the integration is from minus infinity to plus infinity,
and F(z) is a distribution function, uniquely determined by the
THE LAWS OF PROBABILITY 89
P-values on the of a random variable Z equal to the limit,
where that is defined, of the random variables Y = m -1 SX.
m I
But as we saw earlier, (l) is also the expression you get for the
value of the (epistemic) probability r of the taking the value 1
if you believe (i) that there is an objective, but unknown, proba-
bility p of a 1 at any point in the sequence, where F defines the
epistemic probability distribution over p, over the closed unit
interval [0, I], (ii) that the Xi are independent relative to this
unknown probability, and (iii) you evaluate the epistemic proba-
bility of r 1 sand n-r Os to be equal to the unknown probability of
the same event. The significance of de Finetti's result (1) is that
we obtain formally the same explanation of the apparent conver-
gence of the relative frequencies without needing to appeal either
to the existence of an objective probability (which he repudiated
for reasons based on a personal positivistic philosophy) or to the
additional hypothesis h of independence with constant p. As we
observed, de Finetti believed that his theorem reveals the hypoth-
esis of independence and constant probability with respect to an
objective probability to be merely redundant metaphysical bag-
gage, the phenomena they purport to explain being merely a con-
sequence (for consistent reasoners) of a prior judgment that
certain events are exchangeable-a judgment that amounts to no
more than saying that in your opinion they betray no obvious
causal dependence, a very mild judgment indeed.
Or is it? We claim that it is not, and that the independence
assumption apparently eschewed in (1) is nonetheless implicitly
present. Consider two finite sequences of length 2n: one, Sf' has
the form
01010101010 ... 01
wh ile the other, s" is some fairly disorderly sequence of Os and Is,
also having n Os -and n I s, and ending with a 1. Let S3 be a third
sequence, obtained by eliminating the terminal 1 of s ,. Assume
again that all the sequences, i.e. all the are exchange-
able. By the probability calculus
P(Sf) = P(l I 010101 ... O)P(OlOlOl ... 0)
90 CHAPTER 3
And
By exchangeability
P(OIOIOI .. . I) = P(s)
And hence
P (l I s) = P( II 0 I 0 I 0 I ... 0) (3)
But (3) holds for all 11 , however large. Were we to deny that the Xi
are independent there could surely be no reason for accepting (3)
for large values of 11 ; we should on the contrary expect that the
right hand side would move to the neighbourhood of I and the left
hand side to stay in the region of 112 or thereabout s. We should
certainly not expect equality (thi s example is in Good 1969, p.
21 ).
This is not a proof that exchangeability deductively implies
independence (whi ch is certainly not true), but it is a powerful
reason supposing that exchangeability assumpti ons would not be
made relative to repeated trials unless there was already a belief
that the variables were independent. It is noteworthy that there is
a proof, due to Spielman (1976) , that if the Xi are all exchangeable
according to your personal probability measure (assumed count-
ably additi ve) then you must beli eve with probability one with
respect to that same measure that they constitute a von M ises
Ko/lecfi v.
CHAPTER 4
Bayesian Induction:
Deterministic Theories
Philosophers of sci ence have traditionally concentrated mainly on
deterministic hypotheses, leaving statisticians to di scuss how sta-
tistical , or non-deterministic theories should be assessed.
Accordingly, a large part of what naturall y belongs to philosophy
of science is normall y treated as a branch of statistics, going
under the heading ' stati stical inference'. It is not surprising there-
fore, that philosophers and statisticians havc developed distinct
methods for their di fferent purposes. We shall follow the tradition
of dealing separately with deterministic and stati stical theories.
As will become apparent however, we regard thi s separation as
artificial and shall in due course explain how Bayesian principles
provide a unified scientific method.
4.a Bayesian Confirmation
Information gathered in the course of observati on is often con-
sidered to have a bearing on the merits of a theory or hypothesis
(we use the terms interchangeably), either by conf irming or dis-
confirming it. Such information may derive from casual obser-
vation or, morc commonly, from experiment s deliberately
contrived with a view to obtaining rel evant evidence. The idea
that observati ons may count as evidence either for or against a
theory, or be neutral towards it, is at the heart of scientific rea-
soning, and the Bayesian approach must start with a suitable
understanding of these concepts.
As we have described. a very natural one is at hand, for if P(h)
measures your beli ef in a hypothesis when you do not know the
evidence, and P(h I e) is the corresponding measure when you do,
e strengthens your belief in h or, we may say, confirms it, just in
92 CHAPTER 4
case the second probability exceeds the first. We refer in the usual
way to P(h) as 'the prior probability' of h, and to P(h I e) as the
' posterior probability' of h relative to, or in the light of e, and we
adopt the following definitions:
e confirms or supports h just in case P(h I e) > P(h)
e disconfit'ms h just in case P(h I e) < P(h)
e is neutral towards h just in case P(h I e) = P(h) .
One might reasonably take P(h I e) - P(h) as measuring the
degree of e's support for h, though other measures, involving for
example the ratios of these terms, have also been suggested, 1 but
disagreements on this score need not be settled in this book. We
shall, however, say that when P(h I e) > P(h I e') > P(h), the first
piece of evidence confirms the hypothesis more than the second
does.
According to Bayes 's theorem, the posterior probability of a
hypothesis depends on the three factors: pre I h) , pre) and P(h).
Hence, if you know these, you can determine whether or not e
confirms h, and more importantly, calculate P(h I e). Tn practice,
the various probabilities may be known only imprecisely, but as
we shall show in due course, this does not undermine Bayes's the-
orem as a basi s for scienti f ic inference.
The dependence of the posterior probability on these three
terms is refl ected in three principal aspects of scienti f ic inference.
First, other things being equal , the more probable the evidence,
relative to the hypothesis, the more that hypothesis is confirmed.
At one extreme, if e refutes h, then pre I h) = 0 and so disconfir-
mation is at a maximum, while the greatest confirmation is given
when pre I h) = 1, which will be met in practice when h logically
implies e. Statistical hypotheses admit intermediate values for
pre I h); as we show in later chapters, the higher the value, the
greater the confirmation, other things being equal.
I For discussions of various other measures sec, for example, Good 1950, and
Jeffrcy 2004. pp. 29-32.
BAYESIAN INDUCTION: DETERMINISTIC THEORIES 93
Secondly, the power of e to confirm h depends on Pre), that is, on
the probability of e when h is not assumed to be true. This, of
course, is not the same as the probability of e when h is assumed
to be fal se; in fact Pre) is related to the latter by the formula: Pre) =
p re I h)P(h) + pre I -h)P(-h) , as we showed in Chapter 2
(Theorem 12). Thi s inverse dependence of P(h I e) on pre) corre-
sponds to the familiar intuition that the more surprising the evi-
dence, the more confirmation it provides.
Thirdly, the posterior probability of a hypothesis depends on
its prior probability, a dependence that is sometimes discernible in
attitudes to so-called 'ad hoc' hypotheses and in the frequently
expressed preference for the simpler of two hypotheses. As we
shall see, scienti sts always discriminate in advance of any experi-
mentation between theories they regard as more or less credible
(and, so, worthy of attention) and others.
We shall, in the course of this chapter, examine each of these
facets of inductive reasoning.
4.b Checking a Consequence
A characteristic pattern of scientific inference occurs when a log-
ical consequence of a theory is shown to be false and the theory
thereby refuted. As we saw, this sort of inference, with its unim-
peachable logic, impressed Popper so much that he made it the
centrepiece and guiding principle of his scientific philosophy.
Bayesian philosophy readily accommodates the crucial features of
a theory's refutati on by empirical evidence. For if a hypothesis
h entails a consequence e, then, as is easily shown, provided
P(h) > 0, pre I h) = I and P(h I -e) = 0. Interpreted in the Bayesian
fashion, this means that h is maximally di sconfirmed when it is
refuted. Moreover, as we should expect, once a theory has been
refuted, no further evidence can ever confirm it, unless the refut-
ing evidence be revoked. For if e' is any other observation that
is logically consi stent with e, and if P(h I - e) is zero, then so is
P(h I -e & e').
Another characteristic pattern of scientific inference occurs
when a logical conseq uence of a theory is shown to be true and
the theory then regarded as confirmed. Bayes's theorem shows
94 CHAPTER 4
why and under what circumstances a theory is confirmed by its
consequences. First, it follows from the theorem that a theory is
always confirmed by a logical consequence, provided neither the
evidence nor the theory takes either of the extreme probability
P(h)
values. For if h entails e, pre I h) = I , so that P(h I e) = - - .
Pee)
Hence, provided < Pre) < I and P(h) > 0, P(h I e) > P(h) , which
means that e confirms h.
Secondly, the probability axioms tell us, correctly, that suc-
ceeding confirmations by logical consequences eventually dimin-
ish in force (Jeffrcys 1961 , pp. 43-44). For let e I ' e
2
, .. , en be a
succession of logical consequences of h, then
P(h I e
l
& ... & e
n
.
l
) = P(h & ell I e
l
& ... & en_I) =
P(h I e
l
& ... & ell )P(ell l e
l
& ... & ell I) '
As we showed earlier, if h entails all the e
i
, then P(h I e
l
& ... &
en) P(h l ei & ... & en I)' It follows from the Bolzano-
Weierstrass theorem that the non-decreasing sequence of postcri-
or probabilities has a limit. Clearly, the limits, as n tends to
infinity, of the two posterior probabiliti es in this equation are the
same, viz, limP(h I e
l
& ... & ell) = IimP(h I e
l
& ... & en_I)' Hence,
provided that P(h) > 0, P(elll e
l
& ... & ell _I) must tend to 1. Thi s
explains why it is not sensible to test a hypothesis indefinitely.
The result does not however tell us the precise point beyond
which further predictions of the hypothesis are sufficiently
probable not to be worth examining, for that would require a
knowledge of individuals' belief structures which logic does not
supply.
A third sali ent feature of confirmation by a theory's conse-
quences is that in many instances, specific categories of those
consequences each have their own, limited capacity to confirm.
This is an aspect of the familiar phenomenon that however often
a particular experiment is repeated, its results can confirm a gen-
eral theory only to a limited extent; and when an experiment's
capacity to generate significant confirming evidence for the the-
ory has been exhausted through repetition, further support is
BAYESIAN INDUCTION DETERMINISTIC THEORIES 95
sought from other experiments, whose outcomes are predicted by
other parts of the theory.
2
This phenomenon has a Bayesian explanation (Urbach 1981).
Consider a general hypothesis h and let hr be a substantial restric-
tion of that hypothesis. A substantial restriction of Newton's the-
ory might, for example, express the idea that freely falling bodies
near the Earth's surface descend with constant acceleration, or
that the period and length of a pendulum are related by the famil-
iar formula. Since h entails hr' P(h) :5 P(h) , as we showed in
Chapter 2, and if h,. is much less speculative than its progenitor, it
will often be much more probable.
Now consider a series of predictions that are implied by h, and
which also follow from h,.. If the predictions are verified, they
may confirm both theories, whose posterior probabilities are
given by Bayes's theorem thus:
P(h)
P(h Ie & e) & ... & e ) = _...... . ... _ -
I - /I P( e I & e 2 & ... & e)
and
P(h,)
P(h,. 1 e
l
& e) & ... & e) = --- -- . -c"-. - - -
- P( e I & e 2 & ... & e)
Combining these two equations to eliminate the common
denominator yields
P(h)
P(h I e
l
& e) & ... & e ) = P(h _ I e
l
& e 1 & ... & e ).
- n P(h,.) I - /I
Since the maximum value of the last probability in this equation
is 1, it follows that however many predictions of h,. have been ver-
ified, the posterior probability of the main theory, h, can never rise
b
P( h) . b'l- f h d .
a ove - - . Therefore, the pnor proba 1 Ity 0 _ etermmes a
P(h,.) I
limit to how far evidence entailed by it can confirm h. And this
expl ains the phenomenon under consideration, for the predictions
verifi ed by means of an experiment (that is, a procedure designed
2 Thi s is related to the phenomenon that the more varied a body of evidence,
the greater its inductive force, which we discuss in section 4.g below.
96 CHAPTER 4
to a specified pattern) do normally follow from and confirm a
much-restricted version of the predicting theory.
The arguments and explanations in this section rely on the pos-
sibility that evidence already accumulated from an experiment can
increase the probability that further performances of the experi-
ment will produce similar results. Such a possibility was denied by
Popper and by his supporters, on the grounds that the probabilities
involved are not objective. How then do they explain the fact,
familiar to every scientist, that repeating some experiment indefi-
nitely (or usually, more than a very few times) is pointless?
Musgrave (1975) attempted an explanation. He argued that after a
certain (unspecified) number of repetitions, the scientist should
form a generalization to the effect that the experiment will always
yield a result that is similar, in ccrtain respects to those results
already obtained, and that thi s generalization should then be
entered into ' background knowledge' . Relative to the ncwly aug-
mented background knowledge, the experiment is certain to pro-
duce the same result when it is next performed as it did before.
Musgrave then appealed to the putative principle, which we dis-
cuss in the next section, that evidcnce confirms a hypothesis in
proportion to the difference between its probability relative to the
hypothesis plus background knowledge and its probability relative
to background knowledge alone, that is, to P(e I h & b) - P(e I b),
and inferred that even if the experiment did produce the expected
result when next conducted, the hypothesis would receive no new
confirmation.
A number of decisive objcctions can be raised against thi s
account. First, as we show in the next section, although it forms
part of the Bayesian account and seems to be a feature of sci-
ence that confirmation depends in its degree upon the probabil-
ity of the evidcnce, that principle has no basis in Popperian
methodology. Popper simply invoked it ad hoc. Secondly,
Musgrave 's suggestion takes no account of the fact that particU-
lar experimental results may be generalized in infinitely many
ways. This is a substantial objection since different generaliza-
tions givc rise to different implications about future experimen-
tal outcomes. So Musgrave's explanation calls for a rule that
would guide the scientist to a particul ar and appropriate gener-
alization; but we cannot see how appropriateness could be
BAYESIAN INDUCTION DETERMINISTIC THEORIES 97
defined or such a rule possibly justified within the limitations of
Popperian philosophy. Finally, the decision to designate the gen-
eralization background knowledge, with the effect that has on
our evaluation of other theories and on our future conduct
regarding for example, whether or not to repeat certain experi-
ments, is comprehensible only if we have invested some confi-
dence in the generalization. But then this Popperian account
tacitly invokes the same kind of inductive notion as it was
designed to avoid. The fact is that the phenomena concerning
the confirming power of experiments and their repetitions are
essentially inductive and are beyond the reach of anti-inductivist
methodologies such as Popper's.
4.c I The Probability of the Evidence
In the Bayesian account, confirmation occurs when the posterior
probability of a hypothesis exceeds its prior probability, and the
greater the difference, the greater the confirmation. Now Bayes's
theorem may be expressed in the following ways:
P(h I e)
P(h)
pre I h)
Pre)
pie I h ) .
P(h) + ~ h ) \'
pre I h)
We see that the evidential force of e is entirely expressed by the
pre I h )
ratio - - , known as the Bayes {actor. The smaller this factor,
P ~ I ~ .
that is to say, the more probable the evidence if the hypothesis is
true than if it is false, the greater is the confirmation. In the deter-
ministic case, where h entails e, so that pre I h) = I, confirmation
depends inversely on Pre) or pre I -h); this fact is reflected in the
everyday experience that information that is particularly unex-
pected or surprising, unless some hypothesis is assumed to be
true, supports that hypothesis with particular force. Thus if a
soothsayer predicts that you will meet a dark stranger some time
and you do, your faith in his powers of precognition would not be
much enhanced: you would probably continue to regard his pre-
dictions as simply guesswork. But if the prediction also gave you
98 CHAPTER 4
the correct number of hairs on the head of that stranger, your pre-
vious scepticism would no doubt be severely shaken.
Cox (1961 , p. 92) illustrated this point nicely with an incident
in Shakespeare's Macbeth. The three witches, using their special
brand of divination, tell Macbeth that he will soon become both
Thane of Cawdor and King of Scotland. Macbeth finds these two
predictions incredible:
By Sinel 's death I know I am Thane of Glamis;
But how of Cawdor? the Thane of Cawdor lives,
A prosperous gentleman; and to be King
Stands not within the prospect of belief
No more than to be Cawdor.
But shortly after making this declaration, he learns that the Thane
of Cawdor prospered no longer, was in fact condemned to death,
and that he, Macbeth, had succeeded to the title, whereupon, his
attitude to the witches' powers of foresight alters entirely, and he
comes to believe their other predictions.
Charles Babbage (1827), the celebrated polymath and 'father
of computing', examined numerous logarithmic tables published
over two centuries in various parts of the world, with a view to
determining whether they derived from a common source or had
been worked out independently. He found the same six errors in
all but two and drew the " irresistible" conclusion that the tables
containing those errors had been copied from a single original. As
Jevons (1874, pp. 278- 79) pointed out, the force of this conclu-
sion springs from the t ~ l t that if the tables originated from the
same source, then it is practically certain that an error in one will
be reproduced in the others, but if they did not, the probability of
errors being duplicated is minuscule. Such reasoning is so COIll-
pelling that compilers of mathematical tabl es regularly protect
their copyrights by purposely incorporating some minor errors
"as a trap for would-be plagiarists" (L.J. Comrie )\ and cartogra-
phers do the same.
The inverse relationship between the probability of evidence
and its confirming power is a simple and direct consequence of
.1 Thi s is quoted in Bowden 1953. p. 4.
BAYESIAN INDUCTION DETERMINISTIC THEORIES 99
Bayesian theory. On the other hand, methodologies that eschew
probabilistic evaluations of hypotheses, in the interests of objec-
tivity, seem constitutionally unable to account for the phenome-
non. Popper (1959a, appendix *ix) recognized the need to provide
such an account and rose to the challenge. First, he conceded that,
in regard to confirmation, the significant quantities are Pre I h)
and pre); he then measured the amount of confirmation or "cor-
roboration" which e confers on h by the difference between those
quantities. But Popper never said explicitly what he meant by the
probability of evidence. He could not allow it a SUbjective conno-
tation without compromising the intended objectivist quality of
his methodology, yet he never worked out what objective signifi-
cance the term could have. His writings suggest he had in mind
some purely logical notion of probability, but neither he nor any-
one else has managed to give an adequate account oflogical prob-
ability. Secondly, Popper never satisfactorily justified his claim
that hypotheses benefit in any epistemie sense from improbable
evidence; indeed, the idea has been closely examined by philoso-
phers and is generally regarded as indefensible within the
Popperian scheme. (Sec Chapter 1, above, and, for example,
Howson 2000, and Grunbaum 1976.)
The Bayesian position has recently been misunderstood to
imply that if some evidence is known, then it cannot support any
hypothesis, on the grounds that known evidence must have unit
probability. That the objection is based on a misunderstanding is
shown in Chapter 9, where some other criticisms of the Bayesian
approach arc rebutted.
4.d The Ravens Paradox
The Bayesian position that confirmation is a matter of degree,
determined by Bayes's theorem, scotches a famous puzzle, first
posed by Hempel (1945), known as the Paradox ol Confirmatiol1
or sometimes as the Ravens Paradox. It was called a paradox
because its premises seemed extremely plausible, despite their
supposedly counter-intuitive consequences, and the reference to
ravens stems from the paradigm hypothesis, 'All ravens are
black', that is frequently used to present the problem. The alleged
100 CHAPTER 4
difficulty arises from the following assumptions about confirma-
tion. (RB will signify the proposition that a certain object is black
and a raven, and RB that it is neither black nor a raven.)
I. Hypotheses of the form ' All Rs are B' are confirmed by
evidence of something that is both Rand B. (Hempel called
this N i c o d ~ Condition, after the philosopher Jean Nicod.)
2. Logically equivalent hypotheses are confirmed by the
same evidence. (This is the Equivalence Condition.)
Now, by the Nieod Condition, 'All non-Bs are non-Rs' is con-
firmed by RB; and by the Equivalence Condition, so is 'All Rs are
B', since the two generalizations are logically equivalent. Many
philosophers regard this consequence as blatantly false, since it
says that you can confirm the hypothesis that all ravens are black
by observing a non-black non-raven, say, a white lie or a red her-
ring. This seems to suggest that you could investigate that and
other similar general izations jllst as lvell by examining objects on
your desk as by studying ravens on the wing. But that would be a
non sequitur. For the fact that RB and RB both confirm a hypoth-
esis does not mean that they do so with equal force. And once it
is recogni zed that confirmation is a matter of degree, the conclu-
sion ceases to be counter-intuitive, because it is compatible with
RB confirming 'All Rs are B', but to a negligible degree. This sim-
ple point constitutes the Bayesian solution to the problem.
But a Bayesian analysis can take the matter further, first of all ,
by demonstrating that in the case of the paradigm hypothesis, data
of the form R B do in fact confirm to a negl igible degree; second-
ly, by showing that Nicod's condition is not valid as a universal
principle of confirmation. Consider the first point. The impact of
the two data on h, 'All ravens are black', is given as follows:
P(h I RB) P(RB I h) P(h I RB) P(RIJ I h)
- --- = .. _-- & -- = ---'---
P(h) P(RB) P(II) P(RB) .
These express ions can be simplified. First, P(RB I /1) =
PCB I h & R)P(R I /1) = P(R I h) = peR). We arrived at the last
equality by assuming that whether some arbitrary object is a raven
is independent of the truth of h, which seems plausible to us, at
3AVESIAN INDUCTI ON: DETERMINISTIC THEORIES 101
my rate as a close approximation, though Horwich (1982, p. 59)
hinks it lacks plausibility.4 By parallel reasoning, P(kB I h) =
P(B I h) = P(B). Also, P(RB) = P(B I R)P(R) , and P(B I R) =
'f.P(B I R & 8)P(8 I R) = LP(B I R & fJ)P(fJ) , where 8 represents
Jossible values of the proportion of ravens that arc black (h says
hat fJ = 1), and assuming independence between 8 and R. Finally,
P(B I R & 8) = 8, for if the proportion of ravens in the universe
hat are black is fJ, the probability of a randomly selected raven
Jeing black is also 8.
Combining all these considerations with Bayes 's theorem
yields:
P( h I RB) 1 P( h I RB )
--- = --_. &-.. =
P(h) L8P(8) P(h) P(RIE)
i\ccording to the first of thcse equations, the ratio of the posteri-
)r to the prior probabilities of h is inversely proportional to
'f.fJP(8). This means, for example, that if it were initially very
Jrobable that all or virtually all ravens are black, then LfJP(fJ)
>Yould be large and RB would confirm h rather little. While if it
>Yere initially relatively probable that most ravens are not black,
he confirmation could be substantial. Intermediate degrees of
lIlcertainty regarding fJ would bring their own levels of confirma-
ion to h.
The second equation refers to the confirmation to be derived
from the observation of a non-black non-raven, and here the cru-
~ i l probability term is peR I B). Now presumably there are vast-
.y morc non-black things in the universe than ravens. So even if
>Ye felt certain that no ravens are black, the probability of some
)bject about which we know nothing, except that it is not
Jlack, being a non-raven must be very high, practically I. Hence,
P(h I RE) = (1 - E)P(h), where E is a very small positive number;
Vranas (2004) interprets the assumption as asserting that whether some arbi-
rary object is a raven "should" be independent or h. and he criticizes this and
lther Bayesian accounts for depending upon a c laim for which, he says. there
:an be no reasoned defence. i:3ut our argument docs not need slich a strong
Isslimption. Ollr position is mere ly that in this particular case, our and, we SIIS-
)ect, most other people's personal probabilities are such that independence
lbtains.
102 CHAPTER 4
therefore, observing that some object is neither a raven nor black
provides correspondingly little confirmation for h.
5
A Bayesian analysis necessarily retains the Equivalence
Condition but gives only qualified backing to thc Nicod
Condition, for it anticipates circumstances in which the condition
fails . For instance, suppose the hypothesis under examination is
'All grasshoppers are located outside the County of Yorkshire'.
One of these creatures appearing just beyond the county border is
an instance of the generalization and, according to Nicod, con-
firms it. But it might be more reasonably argued that since there
are no border control s or other obstacles to restrict the movement
of grasshoppers in that area, the observation of one on the edge of
the county but outside it increases the probability that some oth-
ers have actuall y crossed over, and hence, contrary to Niemi, it
undermines the hypothesis. In Bayesian terms, this is a case
where, relative to background information, the probability of
some datum is reduced by a hypothesis--that is, pre I h) < P(e)-
which is thereby disconfirmed-that is, P(h I e) < P(h).6 This
example is adapted from Swinburne 1971, though the idea seems
to originate with Good 1961.
Another, more striking case where Nicod's Condition breaks
down was invented by Rosenkrantz (1977, p. 35). Three people
leave a party, each with a hat. The hypothesi s that none of the
three has his own hat is confirmed, according to Nicod, by the
observation that person I has person 2's hat and by the observa-
tion that person 2 has person 1 's hat. But since the hypothesis con-
cerns only three, particular people, the second observation must
rejitfe the hypothesis, not confirm it.
Our grasshopper example may also be used to show that
instances of the type RB can sometimes confirm ' All Rs are B' .
Imagine that an object that looks for all the world like a grasshop-
per had been found hopping about just outside Yorkshire and that
it turned out to be some other sort of insect. The discovery that the
object was not a grasshopper after all would be relatively unlikely
unless the grasshopper hypothesis were true (hence, Pre) < pre I h));
'The account given here is substantially similar to Mackic's, 1963.
(, This example is adapted from Swinburne 1971. though the idea secms 10 ori g-
inate with Good 19() I.
BAYESIAN INDUCTION DETERMINISTIC THEORIES 103
so it would confirm that hypothesis. [fthe deceptively grasshopper-
like object were discovered within the county, the same conclusion
would follow for this RB instance.
Horwich (1982, p. 58) has argued that the ravens hypothesis
could be differently confirmed depending on how the black raven
was chosen, either by randomly selecting an object from the pop-
ulation of ravens, or by restricting the selection to the population
of black objects. Korb ( 1994) provides a convincing demonstra-
tion of this, which we discuss in a related context in Chapter 8.
We do not accept Horwich's argument for his conclusion.
Denoting a black raven either R* B or RB*, depending on whether
it was discovered by the first selection process or the second, he
claims that evidence of the former kind always confirms more,
because only it subjects the raven hypothesis to the risk of falsifi-
cation. But this conf1ates the process of collecting evidence,
which may indeed expose the hypothesis to different risks of refu-
tation, with the evidence i t s l f ~ which either does or does not
refute the hypothesis, and in the present case it does not.
Our conclusions are, first, that the so-called paradox of the
ravens is not in fact problematic; secondly, that of the two condi-
tions of confirmation that generated it only the Equivalence
Condition is acceptable; and thirdly, that Bayesian theory explains
why.
4.e The Duhem Problem
The Duhem (sometimes called the Duhem-Quine) problem aris-
es with philosophies of science of the type associated with
Popper, which emphasize the power of certain evidence to refute
a theory. According to Popper, falsifiability is the feature of a
theory which makes it scientific. "Statements or systems of state-
ments," he said, "in order to be ranked as scientific, must be
capable of conflicting with possible, or conceivable. observa-
tions" (1963, p. 39). And claiming to apply this criterion, he
judged Einstein's gravitational theory scientific and Freud's psy-
chology not. The term 'scientific' carries a strong f1avour of
commendation, which is, however, misleading in this context.
For Popper could never demonstrate a link between his concept
104 C HAPTER 4
of sci entifi cness and epistemic or inductive merit: a theory that is
scientific in Popper's sense is not necessaril y true, or probably true,
nor can it be sai d either definitely or even probably to lead to the
truth. There is littl e alternati ve then, in our judgment, to regarding
Popper's demarcati on between scientifi c and unscientific state-
ments as without normati ve si gnif icance, but as a claim about the
content and character of what is ordinarily termed science.
Yet as an attempt to understand the practi ce of science,
Popper's ideas bear little fruit. First of all , the claim that sci en-
tific theori es are fa lsifiable by "poss ibl e, or conceivable. obser-
vations" raises a difficulty, because an observation can onl y
fal sify a theory (in other words conclusively demonstrate its fal-
sity) if it is itself conclusively certain. Yet as Popper himself
appreciated, no observations fall into thi s category; they are all
fallibl e. But unwilling to concede degrees of fallibility or any-
thing of the kind, Popper took the view that observation reports
that are admitted as evidence "are accepted as the result of a
decisi on or agreement ; and to that extent they are conventions"
(1959a, p. 106; our italics). It is unclear to what psychological
attitude such acceptance corresponds, but whatever it is,
Popper's view pull s the rug from under hi s own philosophy,
since it impli es that no theory can reall y be falsified by evi-
dence. Every ' fa lsif ication ' is merely a convention or deci sion:
" From a logical point of view, the testing of a theory depends
upon basic statement s whose acceptance or rej ecti on, in its turn,
depends upon our decisions. Thus it is decisions which settl e the
fate of theori es" (\ 959a, p. 108).
Watkins was one of those who saw that the Popperian positi on
could not rest on this arbitrary basis, and he attempted to shore it
up by arguing that some infallibly true observation statements do
in fact exist. He agreed that a statement like ' the hand on this di al
is pointing to the numeral 6' is fallible, since it is possible, how-
ever unlikely, that the person reporting the observation mistook
the positi on of the hand. But he claimed that introspecti ve percep-
tual reports, such as ' in my visual fi eld there is now a sil very cres-
cent against a dark blue background'. " may ri ghtl y be regarded by
their authors when they make them as infa llibly true" (1984, pp.
79 and 248). But in our opinion Watkins was wrong, and the state-
ments he regarded as infallible are open to the same sceptical
BAYESIAN INDUCTION: DETERMINISTIC THEORIES 105
doubts as any other observational report. We can illustrate this
through the above example: clearly it is possible, though admit-
tedly not very probable, that the introspector has misremembered
and mistaken the shape he usually describes as a crescent, or the
sensation he usually receives on reporting blue and silvery
images. These and other similar sources of error ensure that intro-
spective reports are not exempt from the rule that non-analytic
statements are fallible.
Of course, the kinds of observation statement we have men-
tioned, if asserted under appropriate circumstances, would never
be seriously doubted, for although they could be false, they have
a force and immediacy that carries conviction: in the traditional
phrase, they are 'morally certain'. But if they are merely indu-
bitable, then whether or not a theory is regarded as refuted by
observational data rests ultimately on a subjective feeling of cer-
tainty, a fact that punctures the objectivist pretensions of
Popperian philosophy.
A second objection to Popper's falsifiability criterion, and the
one upon which we shall focus for its more general interest, is that
it deems unscientific most of those theories that are usually judged
science's greatest achievements. This is the chief aspect of the
well-known criticisms advanced by Polanyi (1962), Kuhn (1970),
and Lakatos (1970), amongst others, but based on the arguments
of Duhem ( 1905). They pointed out that notable theories of science
are typically unfalsifiable by observation statements, because they
only make empirical predictions in association with certain auxil-
iary theories. Should any such prediction turn out to be false, logic
does not compel us to regard the principal theory as untrue, since
the error may lie in one or more of the auxiliaries. Indeed, there are
many occasions in the history of science when an important theo-
ry led to a false prediction but was not itself significantly
impugned thereby. The problem that Duhem posed was this: when
several distinct theories are involved in deriving a Ialse prediction.
which olthem should be regarded asfalse?
Lakatos and Kuhn on the Duhem Problem
Lakatos and Kuhn both investigated scientific responses to
anomalies and were impressed by the tendency they observed for
106 CHA PTER 4
the benefit of the doubt persistently to be given to particular,
especially fundamental theori es, and for one or more of the aux-
iliary theories regularly to be blamed for any false prediction.
Lakatos drew from this observation the lesson that science of the
most significant kind usually proceeds in what he called scientif-
ic research programmes, each comprising a central , or ' hard
core', theory, and a so-called ' protective belt' of auxiliary theo-
ri es. During the lifetime of a research programme, these clements
are combined to yield empirical predicti ons, which arc then
experimentally checked; and if they turn out to be false, the aux-
ili ary hypotheses act as a protective shield, as it were, for the hard
core, and take the brunt of the refutation. A research programme
is also characterised by a set of heuristic rules by which it devel-
ops new auxiliary hypotheses and extends into new areas. Lakatos
regarded Newtonian physics as an exampl e of a research pro-
gramme, the three laws of mechanics and the law of gravitation
comprising the hard core, and various optical theories, proposi-
tions about the natures and dispositions of the planets, and so
forth, being the protective belt.
Kuhn's theory is similar to the methodology we have just out-
lined and probabl y inspired it in part. Broadly speaking, Kuhn 's
' paradigm' is the equivalent of a scientific research programme,
though his idea is developed in less detail.
Lakatos, following Popper, also added a normative element,
something that Kuhn deliberat ely avoided. He held that it was per-
fectly all right to treat the hard core systemat ically as the innocent
party in a refutat ion, provided the research programme occasion-
all y leads to successful "novel" predictions or to successful , "non-
ad hoc" expl anat ions of existing data. Lakatos call ed such
programmes "progressive."
The sophisticated falsificationi st [which Lakatos counted himsel f] ...
sees nothing wrong with a group of brilliant scientists conspiring to
pack cvcrything they can into their favourite research programme .
with a sacred hard core. As long as their genius ----and luck- enables
them to expand their programme ' pmgressil 'eh" , whilc sticking to its
hard core, they are allowed to do it. (Lakatos 1970, p. 187)
I f, on the other hand, the research programme persistently pro-
duces false predictions, or if its explanations are habituall y ad
BAYESIAN INDUCTION DETERMINISTI C THEORIES 107
hoc, Lakatos called it "degenerating." The notion of an ad hoc
explanation-briefly, one that does not produce new and verified
predictions-is central to attempts by the Popperi an school to deal
with the Duhem probl em and we discuss it in greater detail below,
in section g. In apprais ing research programmes, Lakatos
employed the tendenti ous terms 'progressive' and 'degenerating' ,
but he never succeeded in substantiating their normative intima-
tions, and in the end he seems to have abandoned the attempt and
settled on the more modest claim that, as a matter of historical
fact, progressive programmes were well regarded by scientists,
while degenerating ones were distrusted and eventually dropped.
This last claim, it seems to us, contains a measure of truth, as
cvidenced by case studies in the history of science, such as those
in Howson 1976. But although Lakatos and Kuhn identified and
described an important aspect of scientific work, they could not
explain it or rationali ze it. So, for example, Lakatos did not say
why a research programme's occasional predictive success could
compensate for numerous failures, nor did he specify how many
such successes arc needed to convert a degenerati ng programme
into a progressive one, beyond remarking that they should occur
"now and then".
Lakatos was also unable to explain why certain theories arc
raised to the privil eged status of hard core in a research pro-
gramme while others are left to their own devices. His writings
give the impression that the scientist is free to decide the question
at will, by "methodological fiat", as he says. Which suggests that
it is perfectly canonical scientific practice to set up any theory
whatever as the hard core of a research programme, or as the cen-
tral pattern of a paradigm, and to attribute all empirical difficul-
ties to auxiliary hypot heses. This is far from being the case. For
these reasons and also because of ditTiculties with the notion of
an ad hoc hypothesi s, to be discussed bclow, neither Kuhn's theo-
ry of paradigms nor Lakatos's so-called 'sophisticated falsifica-
tionism' are in any position to solve the Duhem problem.
The Bayesian Resolution
The questions left unanswered in the Kuhn and Lakatos method-
ologies are addressed and resolved, as Dorling (1979) brilliantly
108 CHAPTER 4
showed, by referring to Bayes's theorem and considering how the
individual probabilities of theories are severally altered when, as
a group, they have been falsified.
We shall illustrate the argument through a historical example
that Lakatos (1970, pp. 138-140; 1968, pp. l74-75) drew heavi-
ly upon. In the early nineteenth century, William Prout (1815,
1816), a medical practitioner and chemist, advanced the idea that
the atomic weight of every element is a whole-number multiple of
the atomic weight of hydrogen, the underlying assumption being
that all matter is built up from different combinations of some
basic element. Prout believed hydrogen to be that fundamental
building block. Now many of the atomic weights recorded at the
time were in fact more or less integral multiples of the atomic
weight of hydrogen, but some deviated markedly from Prout's
expectations. Yet this did not shake the strong belief he had in his
hypothesis, for in such cases he blamed the methods that had been
used to measure those atomic weights. Indeed, he went so far as
to adjust the atomic weight of the element chlorine, relative to that
of hydrogen, from the value 35.83, obtained by experiment, to 36,
the nearest whole number. Thomas Thomson (1818, p. 340)
responded in a similar manner when confronted with 0.829 as the
measured atomic weight (relative to the atomic weight of oxygen)
of the element boron, changing it to 0.87S, "because it is a multi-
ple of 0.125, which all the atoms seem to be". (Thomson erro-
neously took the relative atomic weights of hydrogen and oxygen
as 0.12S.)
Prout's reasoning relative to chlorine and Thomson's, relative
to boron, can be understood in Bayesian terms as follows : Prout's
hypothesis t, together with an appropriate assumption a, asserting
the accuracy (within specified limits) of the measuring tech-
niques, the purity of the chemicals employed, and so forth,
implies that the ratio of the measured atomic weights of chlorine
and hydrogen will approximate (to a specified degree) a whole
number. In 181S that ratio was reported as 3S.83-call this the
evidence e-a value judged to be incompatible with the conjunc-
tion of t and a.
The posterior and prior probabilities of t and of a are related
by Bayes's theorem, as follows:
BAYESIAN INDUCTION DETERMINISTIC THEORIES 109
P(e I t)P(t) P(e I a)P(a)
P(t I e) = and P( a Ie) = -
P(e) P(e)
To evaluate the two posterior probabilities, it is necessary to quan-
tify the various terms on the right-hand sides of these equations.
Consider first the prior probabilities oft and of a. J.S. Stas, a dis-
tinguished Belgian chemist whose careful atomic weight measure-
ments were highly influential, gives us reason to think that chemists
of the period were firmly disposed to believe in t, recalling that "In
England the hypothesis of Dr Prout was almost universally accept-
ed as absolute truth" and that when he started investigating the sub-
ject, he himself had "had an almost absolute confidence in the
exactness of Prout's principle" (1860, pp. 42 and 44).
It is less easy to ascertain how confident Prout and his contem-
poraries were in the methods used to measure atomic weights, but
their confidence was probably not great, in vicw of the many clear
sources of error. For instance, errors were recognised to be inherent
in the careful weighings and manipulations that were required; the
particular chemicals involved in the experiments to measure the
atomic weights were of questionable purity; and, in those pioneer
days, the structures of chemicals were rarely known with certainty.
7
These various uncertainties were reinforced by the fact that inde-
pendent measurements of atomic weights, based on the transforma-
tions of different chemicals, rarely delivered identical results.s On
the other hand, the chemists of the time must have felt that that their
atomic weight measurements were more likely to be accurate than
not, otherwise they would hardly have reported them.
9
, The several sources of error were rehearsed by Mallet ( 1893).
For example, Thomson (1818, p. 340) reported two independent measurc-
ments--2.998 and 2.66-for the weight. relative to the atomic weight of oxygcn.
ofa molecule of boracic (boric) acid. He required this value in order to calculatc
the atomic weight of boron from the weight of the boric acid produced after the
c1cmcnt was combusted.
4 " I am far from flattcring myself that thc numbers which I shall give are all
accurate; on the contrary, I have not the least doubt that many of them are still
erroneous. But they constitute at least a nearer approximation to the truth than
the numbers contained in thc first tablc [which Thomson had published some
years before]" (Thomson 1818. p. 339).
110 CHAPTER 4
For these reasons, we conjecture that P(a) was in thc ncigh-
bourhood of 0.6 and that P(t) was around 0.9, and these are the
figures we shall work with. Wc stress that these figures and those
we shall assign to other probabilities are intended chiefly to show
that hypotheses that are jointly refuted by an observation, may
sometimes be disconfirmed to very different degrees, so illustrat-
ing the Bayesian resolution of Duhem's problem. Nevertheless,
we believe that the figures we have suggested are reasonably
accurate and sufficiently so to throw light on the historical
progress of Prout's hypothesis. As will become apparent, the
results we obtain are not very sensitive to variations in the
assumed prior probabilities.
The posterior probabilities of f and of a depend also on Pre) ,
pre I t), and pre I a). Using the theorem of total probability, the
first two of these terms can be expressed as follows:
Pre) = P( e I t)P(t} + pre I ~ t ) P ~ t )
pre I t) = P( e If & a)P(a I t) + pre I t & -a)P(-a I ().
We will follow Dorling in taking t and a to be independent,
viz, P(a I t) = P(a) and hence, P(-a I t) = P(-a). As Dorling points
out (1996), this independence assumption makes the calculations
simpler but is not crucial to the argument. Nevertheless, that
assumption accords with many historical cases and seems clearly
right here. For we put ourselves in the place of chemists of Prout 's
day and consider how our confidence in his hypothesis would
have been affected by a knowledge that particular chemical sam-
ples were pure, that particular substances had particular molecu-
lar structures, specific gravities, and so on. It seems to us that it
would not be affected at all. Bovens and Hartmann (2003, p. Ill)
take a different view and have objected to the assumption of inde-
pendence in this context. Speaking in general terms, they allege
that "experimental results are determined by a hypothesis and
auxiliary theories that are often hopelessly interconnected with
each other."
And these interconnections raise havoc in assessing the value of
experimental results in testing hypotheses. There is always the fear
that the hypothesis and the auxiliary theory really come out of the
BAYESIAN INDUCTION: DETERMINISTIC THEORIES III
samc deceitful family and that the lics of one reinforce the lies of
the other.
Wc do not assert that theories are never entangled in the way that
Bovcns and Hartmann describe, but for the reasons we have just
cited, it secms to us that thc prcscnt situation is very far from
being a case in point.
Returning to the last equation, if we incorporate the independ-
ence assumption and take account of the fact that since the con-
junction f & a is rcfutcd bye, pre I f & a) must be zero, we obtain:
pre I t) = pre I t & ~ a ) P ~ a ) .
By parallel reasoning, we may derive the results:
pre I a) = P( e I t & a ) P ~ t)
pre I - t) = P( e I - t & a)P(a) + pre I t & ~ a ) P ~ a ) .
So, provided the following terms are fixed, which we havc donc in
a tentative way, to be justified presently, the posterior probabilities
of t and of a can be calculated:
pre I - t & a) = 0.0]
pre I - t &-a) = 0.0]
Pre I t & -a) = 0.02.
The first of these gives the probability ofthc evidence if Prout's
hypothesi s is not true, but if the assumptions madc in calculating
the atomic weight of chlorine are accurate. Certain ninctecnth-
century chemists thought carcfully about such probabilities, and
typicall y took a theory of random distribution of atomic weights as
the alternative to Prout's hypothesis (for instance, Mallet ]880): we
shall follow this. Suppose it had been established for ccrtain that
the atomic weight of chlorine lies between 35 and 36. (The final
results wc obtain respecting the postcrior probabilities of t and of
a are, incidentally, unaffected by the width of this interval.) The
random-distribution theory assigns equal probabilities to the atom-
ic weight of an element lying in any 0.0 I-widc band. Hence, on
112 CHAPTER 4
the assumption that a is true, but t false, the probability that the
atomic weight of chlorine lies in the interval 35.825 to 35.835 is
0.01. We have attributed the same value to pre I -( & -a), on the
grounds that if a were false, because, say, some of the chemicals
were impure, or had been inaccurately weighed, then, still
assuming t to be false, one would not expect atomic weights to
be biased towards any particular part of the interval between
adjacent integers.
We have set the probability pre I t & -a) rather higher, at 0.02.
The reason for this is that although some impurities in the chem-
icals and some degree of inaccuracy in the measurements were
moderately likely at the time, chemists would not have considered
their techniques entirely haphazard. Thus if Prout's hypothesi s
were true and the measurement technique imperfect, the meas-
ured atomic weights would be likely to deviate somewhat from
integral values; but the greater the deviation, the less the likeli-
hood, so the probability distribution of atomic weight measure-
ments falling within the 35-36 interval would not be uniform, but
would be more concentrated around the whole numbers.
Let us proceed with the figures we havc proposed for the cru-
cial probabilities. We note however that the absolute values of
the probabilities are unimportant, for, in fact, only their relative
values count in the calculation. Thus we would arrivc at the
samc results with the weaker assumptions that pre I -t & a) =
Pre I -{ & -a) = ~ P e I t & -a). Wc now obtain:
pre 1- t) = 0.01 x 0.6 + 0.01 x 0.4 = 0.01
pre I t) = 0.02 x 0.4 = 0.008
pre I a) = 0.01 x 0.1 = 0.001
Pre) = 0.008 x 0.9 + 0.01 x 0.1 = 0.0082.
Finally, Bayes's theorem allows us to derive the posterior proba-
bilities in which we are interested:
P(t I e) = 0.878 (Recall that P(t) = 0.9)
P(a I e) = 0.073 (Recall that P(a) = 0.6).
BAYESIAN INDUCTION DETERMINISTIC THEORIES 113
We see then that the evidence provided by the measured atom-
ic weight of chlorinc affects Prout's hypothesis and the set of aux-
iliary hypotheses very differently; for while the probability of the
first is scarcely changed, that of the second is reduced to a point
where it has lost all credibility.
It is true that these results depend upon certain-we have
argued plausible- premises concerning initial probabilities, but
this does not seriously limit their general significance, because
quite substantial variations in the assumed probabilities lead to
quite similar conclusions, as the reader can verify. So for exam-
ple, if the prior probability of Prout's hypothesis were 0.7 rather
than 0.9, the other assignments remaining unchanged, P(t I e)
would equal 0.65, and P(a I e) would be 0.21. Thus, as before,
Prout's hypothesis is still more likely to be true than false in the
light of the adverse evidence, and the auxiliary assumptions are
still much more likely to be false than true.
Successive pieces of adverse evidence may, however, erode the
probability of a hypothesis so that eventually it becomes more
likely to be false than true and loses its high scientific status. Such
a process would correspond to a Lakatosian degcnerating research
programme or be the prelude to a Kuhnian paradigm shift. In the
prescnt case, the atomic weight of chlorine having been repeated
in various, improved ways by Stas, whose laboratory skill was
universally recognized, Mallet (1893, p. 45) concluded that "It
may be reasonably said that probability is against the idea of any
future discovery ... ever making the value of this element agree
with an integer multiple of the atomic weight of hydrogen". And
in the light of this and other atomic weight measurements he
regarded Prout's original idea as having been "shown by the cal-
culus of probability to be a very improbable one". And Stas him-
s l t ~ who started out so very sure of its truth, reported in 1860 that
he had now " reached the complete conviction, the entire certain-
ty, as far as certainty can be attained on such a subject, that Prout's
law ... is nothing but an illusion" (1860, p. 45).
We conclude that Bayes's theorem provides a framework that
resolves the Duhem problem, unlike the various non-probabilistic
methodologies which philosophers have sought to apply to it. And
the example of Prout's hypothesis, as well as others that Dorling
114 CHAPTER 4
( 1979 and 1996) has analysed, show in our view, that the Bayesian
model is essentially correct.
4.f Good Data, Bad Data, and Data Too Good
to Be True
Good Data
The marginal influence that an anomalous observation may
exert on a theory 's probability contrasts with the dramatic effect
of some confirmati ons. For instance, if the measured atomic
weight of chlorine had been a whole number, in line with
Prout's hypothesi s, so that P(e I t & a) = I instead of 0, and if
the other probability assignments remained the same, the prob-
ability of the hypothes is would shoot up from a prior ofO.9 to
a posterior of 0. 998. And even more striking: had thc prior
probability of t been 0.7, its posterior probability would have
risen to 0.99.
This asymmetry between the effects of anomalous and con-
firming instances was emphasized by Lakatos, who regarded it as
highly significant in science, and as a characteri stic feature of a
research programme. He maintained that a scientist invol ved in
such a programme typically "forges ahead with almost complete
disregard of 'refutati ons' ," provided there arc occasional predic-
tive successes (1970, p. 137): the scientist is "encouraged by
nature's YES, but not di scouraged by its NO" (p. 135). As we have
indicated, we beli eve there to be much truth in Lakatos's observa-
tions: the trouble, however, is that these observat ions are merely
absorbed, without justification, into his methodology; the
Bayesian methodology, on the other hand, expl ains why and under
what circumstances the asymmetry effect is present.
Bad Data
An interesting fact that emerges from the Bayes ian analysis is that
a successful prediction derived from a combinati on of two theo-
ries docs not necessarily redound to the credit of both of them,
BAYESIAN INDUCTION DETERMINISTIC THEORIES lIS
indeed one may even be discredited. Consider Prout's hypothesis
again, and suppose the atomic weight of chlorine had been deter-
mined, not in the establi shed way, but by concentrating hard on
the element while selecting a number blindly from a gi ven range
of numbers . And let us suppose that the atomic weight of chlorine
is reported by this method to be a whole number. Thi s is just what
one would predict on the basis of Prout's hypothesis, if the out-
landish measuring tec hnique were accurate. But accuracy is obvi-
ously most unlikely, and it is equally obvious that the results of the
technique could add little or nothing to the credibility of Prout's
hypothesis. This intuiti on is upheld by Bayes's theorem: as before,
let t be Prout 's hypothesis and a the assumpti on that the measur-
ing technique is accurate. Then, set p re I I & ~ a = pre I - ( & - a) =
Pre I-I & a) = 0.0 I, for reasons similar to those stated above. And,
because, as we said, a is extremely implausible, we will set P(a)
at, say 0.0001. It then fo llows that t is not signi f icantly confirmed
bye, for P(t) and P(t I e) are virtually identical.
This example shows that Leibniz was wrong to declare as a
maxim that" It is the greatest commendation of a hypothesi s (next
to truth) if by its help predictions can be made even about phe-
nomena or experiments not [yet] tried". Leibni z, and Lakatos,
who quoted these words with approval ( 1970, p. 123 ), seem to
have overlooked the fact that if a prediction can be deduced from
a hypothesis only with the assistance of highly questionable aux-
iliary claims, then that hypothesis may accrue very little credit
when the predicti on is verified. This explains why the various
sensational predictions that Velikovsky drew from his theory
failed to impress most serious astronomers, even when some of
those predictions were to their amazement fulfilled. For
instance, Velikovsky 's prediction (1950, p. 351) of the existence
of large quantities of petroleum on the planet Venus relied not
only on hi s pet theory that vari ous natural di sasters in the past
had been caused by co lli sions bet ween the Earth and a comet,
but al so on a string of unsupported and implausible assump-
tions, for instance, that the comet in question carried hydrogen
and carbon; that these had been converted to petroleum by elec-
trical discharges supposedly generated in the violent impact
with the Earth; that the comet had later evolved into the planet
Venus; and some others. (More details of Velikovsky's theory
are given in the next section.)
116 CHAPTER 4
Data Too Good to Be True
Data are sometimes said to be 'too good to be true', when they fit
a favoured hypothesis more perfectly than seems reasonable.
Imagine, for instance, that Prout had advanced his hypothesis and
then proceeded to report numerous atomic weights that he had
himself measured, each an exact whole number. Such a result
looks almost as if it was designed to impress, and just for this rea-
son it fails to.
We may analyse this response as follows: chemists in the early
nineteenth century recognized that the measuring techniques
available to them were not absolutely precise in their accuracy but
were subject to experimental error, and so liable to produce a cer-
tain spread of results about the true value. On this assumption,
which we label a
l
, it is extremely unlikely that numerous inde-
pendent atomic weight measurements would all produce exactly
whole numbers, even if Prout's hypothesis were true. So pre I t & a 1 )
is extremely small, and clearly pre I & a I) could be no larger.
Now there are many possible explanations of e, apart from those
involving aI, one being that the experiments were consciously or
unconsciously rigged so as to appear favourable to Prout 's
hypothesis. If this were the only plausible alternative (and so, in
effect, equivalent to pre I t & I) would be very high, as too
P(e I & It follows from the equations in section e, above
that
pre I t) = P(e I t & I) and
P(e I = pre I & I)
and hence,
pre) = P(e I t & + pre I &
Now presumably the rigging of the results to produce exactly
whole numbers would be equally effective whether t was true or
not; in other words,
pre I t & = pre I -t & _al).
BAYESIAN INDUCTION: DETERMINISTIC THEORIES 117
Therefore,
Pee I t)P(t) Pee I t & ~ a ) P ( ~ a )P(t)
P(tle)= = =P(t).
Pee) Pee I t & ~ a ) P ( ~ a )
Thus e does not confirm t significantly, even though, in a mislead-
ing sense, it fits the theory perfectly. This is why it is said to be
too good to be true. A similar calculation shows that the probabil-
ity of a! is diminished, and on the assumptions we have made, this
implies that the idea that the experiments were fabricated is ren-
dered more probable. (The above analysis is essentially due to
Dorling 1996.)
A famous case of data that were alleged to be too good to be
true is that of Mendel's plant-breeding results. Mendel's genetic
theory of inheritance allows one to calculate the probabilities of
different plants producing specific kinds of offspring. For exam-
ple, under certain circumstances, pea plants of a certain strain
may be calculated to yield round and wrinkled seeds with proba-
bilities 0.75 and 0.25, respectively. Mendel obtained seed fre-
quencies that matched the corresponding probabilities in this and
in similar cases remarkably well, suggesting (misleadingly,
Fisher contended) substantial support for the genetic theory.
Fisher did not believe that Mendel had deliberately falsified his
results to appear in better accord with his theory than they really
were. To do so, Fisher said, "would contravene the weight of the
evidence supplied in detail by ... [Mendel's] paper as a whole".
But Fisher thought it a "possibility among others that Mendel
was deceived by some assistant who knew too well what was
expected" (1936, p. 132), an explanation that he backed up with
some, rather meagre, evidence. Dobzhansky (1967, p. 1589), on
the other hand, thought it "at least as plausible" that Mendel had
himself discarded results that deviated much from his ideal, in
the sincere belief that they were contaminated or that some other
accident had befallen them. (For a comprehensive review sec
Edwards 1986.)
The argument put forward earlier to show that too-exactly
whole-number atomic weight measurements would not have sup-
ported Prout's hypothesis depends on the existence of some suffi-
ciently plausible alternative hypothesis that would explain the
data better. We believe that in general, data are too good to be true
118 C HAPTER 4
relative to one hypothes is only if there are such alternatives. This
principle implies that if the method of eliciting atomic weights had
long been establi shed as precise and accurate, and if careful pre-
cautions had been taken against experimenter bi as and deception,
so that all the natural alternatives to Prout 's hypothesis could be
discounted, the inductive force of the data would then no longer be
suspicious. Fi sher, however, did not subscribe to the principle, at
least, not expli citly; he beli eved that Mendel's results told against
the geneti c theory, irrespective of any alternati ve explanations that
might be suggested. But despite this official position, Fisher did in
fact, as we have just indicated, sometimes appeal to such alterna-
ti ves when he formul ated his argument. We refer again to Fisher's
case against Mendel in the next chapter, secti on b.
4.9 Ad Hoc Hypotheses
We have been di scussing the circumstances in whi eh an important
scientific hypothes is, in combination with others, makes a false
prediction and yet emerges with its reputation more or less intact,
while one or more of the auxiliary hypotheses are largely discred-
ited. We argued that this process necessarily call s for alternatives
to the discredited hypotheses to be contemplated. Philosophers,
such as Popper and Lakatos, who deny any inductive role for evi-
dence, and who oppose, in particular, the Bayesian approach take
note of the t ~ l t that scienti sts often do deal with paJ1icuiar
instances of the Duhem problem by proposing alternati ve hypothe-
ses; some of these phil osophers have suggested certain normative
rules that purport to say when such alternatives are acceptable and
when they are not. Their idea is that a theory that was introduced
ad hoc, that is, " for the sole purpose of saving a hypothesis seri-
ously threatened by adverse evidence" (Hempel 1966, p. 29), is in
some way inferi or. The adhocness idea was largely inspired by cer-
tain types of sci enti f ic example, which appeared to endorse it, but
in our view, the examples are misinterpreted and the idea badly
flawed. The foll owing are four such exampl es.
1 Velikovsky, in a daring book called Worlds in Collision that
attracted a great dea l of interest and controversy some years ago,
BAYESIAN INDUCTI ON: DETERMINISTIC THEORIES 119
advanced the theory that the Earth has been subj ect at various
stages in its history to cosmic disasters, through near collisions
with massive comets. He claimed that one such comet passed
close by our planet during the Israelites' captivity in Egypt, caus-
ing many of the remarkable events related in the Bible, such as the
ten plagues and the parting of the Red Sea, before settling down
as the planet Venus. Because the putative cosmic encounter
rocked the entire Earth, Velikovsky expected other peoples to
have recorded its consequences too, if they kept records at all. But
as a matter of fact, many communities around the world failed to
note anything out of the ordinary at the time, an anomaly that
Velikovsky attributed to a "collective amnesia". He argued that
the cataclysms were so terrifying that whole peopl es behaved "as
if [they had] obliterated impressions that should be unforget-
table". There was a need Velikovsky said, to "uncover the ves-
tiges" of these events, "a task not unlike that of overcoming
amnesia in a singlc person" (1 950, p. 288).
Indi vidual amnesia is the issue in the next exampl e.
2 Dianetics is a theory that purports to anal yse the causes of
insanity and mental st ress, which it sees as caused by the 'misfil-
ing' of information in un suitable locations in the brain. By re-fil-
ing these 'engrams', it claims, sanity may be restored, composure
enhanced and, incidentally, the mcmory vastly improved. The
therapy is long and expensive and few people have been through
it and borne out the theory's claims. However, L. Ron Hubbard,
the inventor of Dianeti cs, trumpeted one purported success, and
exhibited thi s person to a large audience, saying that she had a
"full and perfect recall of every moment of her li fc". But ques-
tions from the floor ("What did you have for breakfast on October
3rd, 1942,?", "What colour is Mr Hubbard's tie?", and the likc)
soon demonstrated that the hapless woman had a most imperfect
memory. Hubbard expl ained to the dwindling assembly that when
she first appeared on the stage and was asked to come forward
"now", the word had frozen her in "present time" and paralysed
her ability to recall the past. (See Miller 1987. )
3 Investigations into the IQs of different groups of people
show that the average levels of measured intelligence vary. Some
120 CHAPTER 4
environmentalists, so-called, attribute low scores primarily to
poor social and educational conditions, an explanation that ran
into trouble when a large group of Inuit, leading an aimless, poor
and drunken existence, were found to score very highly on IQ
tests. The distinguished biologist Peter Medawar (1974), in an
effort to deflect the difficulty away from the environmentalist the-
sis, tried to explain this unexpected observation by saying that an
"upbringing in an igloo gives just the right degree of cosiness,
security and mutual contact to conduce to a good performance in
intell igence tests."
In each of these examples, the theory that was proposed in
place of the refuted one seems highly unsatisfactory. It is not like-
ly that any of them would have been advanced, save in response to
particular anomalies and in order to evade the consequent diffi-
culty, hence the label 'ad hoc'. But philosophers who attach
inductive significance to adhocness recognize that the mere fact
that the theory was proposed under such circumstances is not by
itself grounds for condemnation. For there are examples, like the
following, where a theory that was proposed for the sole purpose
of dealing with an anomaly was nevertheless very successful.
4 William Herschel , in 1781 , discovered the planet Uranus.
Astronomers quickly sought to describe the orbit of the new plan-
et in Newtonian terms, taking account of the perturbing influence
of the other known planets, and were able to deduce predictions
concerning its future positions. But discrepancies between pre-
dicted and observed positions of Uranus substantially exceeded
the accepted limits of experimental error, and grew year by year.
A few astronomers mooted the possibility that the fault lay with
Newton's laws but the prevailing opinion was that there must be
some unknown planet acting as an extra source of gravitational
attraction on Uranus, which ought to be included in the
Newtonian calculations. Two astronomers in particular, Adams
and Le Verrier, working independently, were convinced of this and
using all the known sightings of Uranus, they calculated in a
mathematical tOllr dej(Jrce where the hypothetical planet must be.
The hypothesis was ad hoc, yet it was vindicated when careful tel-
escopic observations as well as studies of old astronomical charts
BAYESIAN INDUCTION DETERMINISTIC THEORIES 121
revealed in 1846 the presence of a planet with the anticipated
characteristics. The planet was later called Neptune. Newton's
theory was saved, for the time being. (See Smart 1947.)
The Adhocness Criteria
Examples like the first three above have suggested to some
philosophers that when a theory t, and an auxiliary hypothesis a,
are jointly refuted by some evidence, e I, then any replacement, of
the form t & a', must not only imply e' , but should also have
some new, ' independent' empirical implications. And examples
similar to the fourth have suggested that if the new theory satis-
fies this condition, then it is a particular virtue if some of the new,
independent implications are verified.
These two criteria were anticipated some four hundred years
ago, by the great philosopher Francis Bacon, who objected to any
hypothesis that is "only fitted to and made to the measure of those
particulars from which it is derived". He argued that a hypothesi s
should be " larger or wider" than the observations that gave rise to
it and said that "we must look to see whether it confirms its large-
ness and wideness by indicating new particulars" (1620, I, 106).
Popper ( 1963, p. 241) advanced the same criteria, laying it down
that a "new theory should be independently testable. That is to say,
apart from explaining all the explicanda which the new theory
was designed to explain, it must have new and testable conse-
quences (preferably consequences of a new kind) ." And secondly,
he said, the new theory "should pass the independent tests in ques-
tion". Bacon called hypotheses that did not meet the criteria "friv-
olous distinctions", while Popper termed them "ad hOC".10
10 The first recorded use of the term 'ad hoc' in this context in English was in 1936,
in a review of a psychology book, where the reviewer critici zed some explanations
proffered by the book's author for certain aspect s of childish behaviour:
There s a suspici on of 'ad-hoe-ness' about the 'explanations'. The whole point is that
such an account cannot be satisfactory unt il we can prcdi ct the child's movements
from a knowledge of the tensions, vectors and valences whi ch are operative, inde-
pendent of our knowledge of how the child actuall y behaved. So far we seem reduced
to inventing valences, vectors and tensions from a knowledge of the child's behaviour.
(Sprott, p. 249; our italics)
122 CHAPTER 4
Lakatos (1970, p. 175) refined this terminology, calling a the-
ory that failed the first requirement ad hoc
l
, and one that failed
the second ad hoc" intending these, of course, as terms of disap-
proval. By these criteria, the theories that Velikovsky, Medawar,
and Hubbard advanced in response to anomalous data arc proba-
bly ad hoc I' for they seem to make no independent predictions,
though of course a closer study of those theories might reverse
that assessment. The Adams-Le Verrier hypothesis, on the other
hand, is ad hoc in neither sense, because it did make new predic-
tions, some of which were verified by telescopic sightings of
Neptune. Again, philosophical and intuitive judgment coincides.
Nevertheless, the adhocness criteria are unsound.
This unsoundness is evident both on apriori grounds and
through counter-examples, some of which we consider now. For
instance, suppose one were examining the hypothesis that a par-
ticular urn contains only white counters, and imagine an experi-
ment in which a counter is withdrawn from the urn at random and
then, after its colour has been noted, replaced; and suppose that in
10,000 repetitions of this operation 4,950, say, of the selected
counters were red and the rest white. This evidence clearly refutes
the initial hypothesis taken together with the various necessary
auxiliary hypotheses, and it is then natural to conclude that, con-
trary to the original assumption, the urn contains both red and
white counters in approximately equal numbers. This inference
seems perfectly reasonable, and the revised hypothesis appears
well justified by the evidence, yet there is no independent evi-
dence/hI" it. And if we let the urn vaporize immediately after the
last counter has been inspected, no such independent evidence
would be possible. So the hypothesis about the (late) urn's con-
tents is ad hoc I & 2; but for all that, it seems plausible and satisfac-
tory (Howson 1984; Urbach 1991).
Speculating on the contents of an urn is but a humble form of
enquiry, but there are many instances in the higher sciences which
have the same import. Take the following one from the science of
genetics: suppose it was initially proposed or believed that two
phenotypic characteristics of a certain plant are inherited in accor-
dance with Mendel's principles, through the agency of a pair of
independently acting genes located on different chromosomes.
Imagine now that plant-breeding experiments throw up a surpris-
BAYESIAN INDUCTI ON: DETERMINISTIC THEORIES 123
ing number of plants carrying both phenotypes, so that the origi-
nal hypothesis of independence is rejected in favour of the idea
that the genes are linked on the same chromosome. Again, the
revised theory would be strongly confirmed, and established as
acceptable merel y on the evidence that discredited its predecessor,
without any further, independent evidence. (Fi sher 1970, Chapter
IX, presented an example of this sort.)
The history of the discovery of Neptune, which we have
already discussed, illustrates the same point. Adams estimated
the mass of the hypothetical planet and the elements of its orbit
by the mathematical technique of least squares applied to all the
positional observat ions availabl e on Uranus. Adams's hypot hesis
fitt ed these observations so well that even belore Neptune had
been sighted through the telescope or detected on astronomical
charts, its existence was contemplated with the greatest confi-
dence by the leading astronomers of the day. For instance, in his
retirement address as president of the British Association, Sir
John Herschel , after remarking that the previous year had seen
the discovery of a minor planet, went on: "It has done more. It
has given us the probable prospect of the discovery of another.
We see it as Columbus saw America from the shores of Spain.
Its movements have been felt , trembling along the far-reaching
I ine of our analysis, Hith a certainty hardly il!j(:rior to that ol
ocular demonstration". And the Astronomer Royal, Sir George
Airy, who was initially inclined to believe that the problem with
Uranus would be resolved by introducing a slight adjustment to
the Inverse-Square law, spoke of "the extreme prohahility of now
di scovering a new planet in a very short time" (quoted by Smart,
p. 6 1; our italics). Neptune was indecd discovered within a very
short time.
There is a more general objection to the idea that hypothe-
ses are unacceptable if they are ad hoc. Imagine a scienti st who
is interested in the conjunction of the hypot heses t & a, whose
implication e can be checked in an experiment. The experi ment
is performed with the result e', incompatibl e with e, and the
sci ent ist ventures a new theory t & a', which is consistent with
the observations. And suppose that either no new predi ct ions
foll ow or none has been confirmed, so that the new theory is
ad hoc.
124 CHAPTER 4
Imagine that another scientist, working without knowledge of
his colleague's labours, also wishes to test t & a, but chooses a dif-
ferent experiment for this purpose, an experiment with only two
possible outcomes: either e or -e. Of course, he obtains the latter,
and having done so, must revise the refuted theory, to t & a I, say.
This scientist now notices that e I follows from the new theory and
performs the orthodox experiment to verify the prediction. The
new theory can now count a successful prediction to its credit, so
it is not ad hoc.
But this is strange. We have arrived at opposite valuations of
the very same theory on the basis of the very same observations,
breaching at the same time what we previously called the
Equivalence Condition and showing that the standard adhocness
criteria are inconsistent. Whatever steps might be taken to resolve
the inconsistency, it seems to us that one element ought to be
removed, namely, the significance that the criteria attach to the
order in which the theory and the evidence were thought up by a
particular scientist, for this introduces into the principles of theo-
ry evaluation considerations concerning the state of scientists'
minds that are irrelevant and incongruous in a methodology with
pretensions to No such considerations enter the corre-
sponding Bayesian evaluations.
The Bayesian approach, incidentally, explains why people
often react with instant incredulity, even derision, when certain ad
hoc hypotheses are advanced. Is it likely that their amusement
comes from perceiving, or even thinking they perceive, that the
hypotheses lead to no new predictions? Surely they are simply
struck by the utter implausibility of the claims.
Independent Evidence
The adhocness criteria are formulated in terms that refer to 'inde-
pendent' evidence, yet this notion is always left vague and intu-
itive. How can it be made more precise? Probabilistic
independence cannot fit the case. For suppose theory h was
advanced in response to a refutation bye' and that h both explains
that evidence and makes the novel prediction e". It is the general
opinion, certainly shared by Popperians, and also a consequence
of Bayes's theorem, that e II confirms h, provided it is sufficiently
BAYESIAN INDUCTION: DETERMINISTIC THEORIES 125
improbable, relative to already available information. As dis-
cussed earlier in this chapter, such confirmation occurs, in partic-
ular, when pre II I h & e ') > pre II Ie'). But this inequality can hold
without e!l and e' being independent in the probabilistic sense.
Logical independence is also not the point here, for e!l might
be independent from e' in this sense through some trivial differ-
ence, say, by relating to a slightly different place or moment of
time. And in that case, e" would not necessarily confirm or add
credibility to h. For, as is intuitive, new evidence supports a theo-
ry significantly only when it is significantly different from known
results, not just trivially different in the logical sense described. It
is this intuition that appears to underlie the idea of independence
used in the adhocness criteria.
That 'different' or 'varied' evidence supports a hypothesis
more than a similar volume of homogeneous evidence is an old
and widely held idea. As Hempel (1966, p. 34) put it: "the confir-
mation of a hypothesis depends not only on the quantity of the
favourable evidence available, but also on its variety: the greater
the variety, the stronger the resulting support". So, for example, a
report that a stone fell to the ground from a certain height in such-
and-such time on a Tuesday is similar to that rel ating to the stone's
fall on a Friday; it is very different, however, from evidence of a
planet's trajectory or of a fluid's rise in a particul ar capillary tube.
But although it is often easy enough to classify particular bodies
of evidence as either similar or varied, it is not easy to give the
notions a precise analysis, except, in our view, in probabilistic
terms, in the context of Bayesian induction.
The similar instances in the above list are such that when one
of them is known, any other would be expected with consider-
able confidence. This recalls Francis Bacon's characterisation of
similarity in the context of inductive evidence. He spoke of
observations "with a promiscuous resemblance one to another,
insomuch that if you know one you know all" and was probably
the first to point out that it is superfluous to cite more than a
small, representative sample of such observations in evidence
(see Urbach 1987, pp. 160-64). We are not concerned to give an
exhaustive analysis of the intuitive notion, which is probably too
vague for that to be possible, but are interested in that aspect of
evidential similarity that is pertinent to confirmation. Bacon's
126 CHAPTER 4
observations seem to capture this aspect and we may interpret
his idea in probabilistic terms by saying that if two items of evi-
dence, e
2
and e\, are similar, then P(e
2
I e\) ,., 1; when this con-
dition holds, e
2
provides I ittle support for any hypothesis if e \
has already been cited as evidence. When the pieces of evidence
are dissimilar, then P(e
2
I e \) is significantly less than I, so that
e
2
now does add a useful amount of confirmation to any already
supplied by e\. Clearly this characterization allows for similarity
to be analysed in terms of degree.
To summarize, the non-Bayesian way of appraising hypothe-
ses, and thereby of solving the Duhem problem, through the
notion of adhocness is ungrounded in epistemology, has highly
counter-intuitive consequences, and relies on a concept of inde-
pendence amongst items of evidence that seems unsusceptible to
analysis, except in Bayesian terms. In brief, it is not a success.
4.h Designing Experiments
Why should anyone go to the trouble and expense of performing
a new experiment and of seeking new evidence? The question has
been debated recently. For example, Maher (1990) argues that
since evidence can neither conclusively verify nor conclusively
refute a theory, Popper's scientific aims cannot be served by gath-
ering fresh data. And since a large part of scientific activity is
devoted to that end, if Maher is right, this would constitute yet
another serious criticism of Popper's philosophy. Of more concern
to us is Miller's claim (1991, p. 2) that Bayesian phi losophy
comes up against the same difficulty:
If e is the agent's total evidence, then P(h I e) is thc value of his prob-
ability and that is that. What incentive does he have to change it, for
example by obtaining more evidence than he has already? He might
do so, enabling his total evidence to advance from e to e-; but in no
clear way would P(h Ie') be a better evaluation of probability than
P(h I e) was.
But the purpose of a scientific investigation, in the Bayesian
view, is not to better evaluate inductive probabilities. It is to
diminish uncertainty about a certain aspect of the world.
BAYESIAN INDUCTION DETERMINISTIC THEORIES 127
Suppose the question of interest concerns some parameter. You
might start out fairly uncertain about its value, in the sense that
your probability distribution over its range of possible values is
fairly diffuse. A suitable experiment, if successful, would fur-
nish evidence to lessen that uncertainty by changing the proba-
bility distribution, via Bayes's theorem, making it now more
concentrated in a particular region; the greater the concentration
and the smaller the region the better. This criterion has been
given a precise expression by Lindley (1956), in terms of
Shannon's characterization of information, and is discussed fur-
ther in Howson 2002. Lindley showed that in the case where
knowledge of a parameter (3 is sought, provided the density of x
varies with (3, any experiment in which x is measured has an
expected yield in information. But, of course, this result is com-
patible with a well-designed experiment (with a high expected
information yield) being disappointingly uninformative in a par-
ticular case; and by the same token, a poor experiment may be
surprisingly productive of information.
In deciding whether to perform a particular experiment, at
least three other factors should be taken into account: the cost of
the experiment; the morality of performing it; and the value, both
theoretical and practical, of the hypotheses one is interested in.
Bayes's theorem, of course, cannot help here.
4.i Under-Determination and Prior Probabilities
We pointed out in Chapter 1 that any data are explicable by infi-
nitely many, mutually incompatible theories, a situation that some
philosophers have called the 'under-determination' of theories by
data. For example, Galileo carried out numerous experiments on
freely falling bodies, in which he examined how long they took to
descend various distances. His results led him to propound the
well-known law: s = a + ut + gt
2
, where s is the distance fallen by
the body in time t, and a, 1I and g are constants. Jeffreys (1961, p.
3) pointed out that without contradicting his own experimental
results, Galileo might instead have advanced as his law:
128 CHAPTER 4
where t" t
e
, ... , t" are the elapsed times of fall that Galileo record-
ed in each of his experiments; a, u and g have the same values as
above; and I is any function that is not infinite at any of the val-
ues t
l
, f 2' . . . , tn . Jeffreys's modification therefore represents an
infinity of alternatives to the orthodox theory, all implying
Galileo's data, all mutually contradictory, and all making different
predictions about future experiments.
There is a similar example due to Goodman (1954; for a live-
ly and illuminating discussion, see Jeffrey 1983, pp. 187-190). He
noted that the evidence of many green emeralds, under varied cir-
cumstances, would suggest to most observers that all emeralds are
green; but he pointed out that that hypothesis bears the same rela-
tion to the evidence as does a type of hypothesis that he formulat-
ed as 'All emeralds are grue'. Goodman defined something as
'grue' when it was either observed before the present time (T = 0)
and was green, or was not observed before that time and was blue.
Clearly there are infinitely many grue-type predicates and infi-
nitely many corresponding hypotheses, each associated with a dif-
ferent value of T > O. All the current evidence of green emeralds
is implied by both the green-hypothesis and the grue variants, yet
not more than one of the hypotheses could be true.
As Jeffreys put it, there is always "an infinite number of rules
that have held in all previous cases and cannot possibly all hold in
future ones." This is a problem for those non-Bayesian scientific
methods that regard a theory's scientific value as determined just
by pre I h) and, in some versions, by Pre). Such philosophical
approaches, of which Popper's is one example, and maximulll-
likelihood estimation (Chapter 7, section e) another, would have
to regard the standard law offree fall and Jeffreys's peculiar alter-
natives as equally good scientific theories relative to the evidence
that was available to Galileo, and similarly with Goodman's
strange hypotheses concerning emeralds, although these are judg-
ments with which no scientist would agree.
In the Bayesian scheme, if two theories explain the evidence
equally well, in the sense that pre I hi) = pre I h), this simply
means that their posterior probabi I ities are in the sallle ratio as
their priors. So theories, such as the contrived variants ofGalileo's
law and the Goodman grue-alternatives, which have the same
BAYESIAN INDUCTION: DETERMINISTIC THEORIES 129
relation to the evidence as the orthodox theories and yet are
received with incredulity, must have much lower prior probabili-
ties. The role of prior probabilities also accounts for the important
feature of scientific reasoning that scientists often prefer a theory
that explains the data imperfectly, in the sense that pre I h) < I, to
an alternative that explains them perfectly. This occurs when the
better explanatory power of the alternative is offset by its inferior
prior probability (Jeffreys 1961, p. 4).
This Bayesian account is of course only partial, for we can pro-
vide no general account of the genesis of prior probabilities. In
some situations, the prior may simply be the posterior probability
derived from earlier results and an earlier prior. Sometimes, when
there are no such results, a prior probability may be created
through what we know from other sources. Consider, for instance,
a theory that makes some assertion about a succession of events in
the development of a human society; it might, for example, say
that the elasticity of demand for herring is constant over a particu-
lar period, or that the surnames of all future British prime minis-
ters and American presidents will start with the letter B. These
theories could possibly be true, but are immensely unlikely to be
so. And the reason for this is that the events they describe are the
causal effects of numerous, independent processes, whose separate
outcomes are improbable. The probability that all the processes
will turn out to favour one of the theories in question is therefore
the product of many small probabilities and so is itself very small
indeed (Urbach 1987b). But the question of how the probabilities
of the causal factors are estimated remains. This could be answered
by reference to other probabilities, in which case the question is
just pushed one stage back, or else by some different form of rea-
soning. For instance, the 'simplicity' of a hypothesis has been
thought to have an influence on its initial probability. This and
other possibilities are discussed in Chapter 9.
4.j I Conclusion
The various, mostly familiar aspects of scientific reasoning that
we have examined have all shown themselves to correspond nat-
130 CHAPTER 4
urally to aspects of Bayesian logic, whereas non-Bayesian
accounts fail more or less completely. So far, we have concentrat-
ed chiefly on deterministic theories. We shall see in the next and
following chapters that the Bayesian approach applies equally
well to statistical reasoning.
CHAPTER 5
Classical Inference:
Significance Tests and
Estimation
In the last chapter, we showed how leading aspects of scientific
rcasoning are illuminated by reference to Bayes 's theorem, con-
fining our attention, however, mainly to determini stic theories.
We now consider theori es that are not determini stic but proba-
bilistic, or statistical. From the Bayesian viewpoint the division is
artificial and unnecessary, the two cases differing only in regard
to the probability of the evidence relative to the theory, that is,
pre I h), which figures in the central theorem: when h is determin-
istic, thi s probability is either I or 0, depending on whcther h
entails e or is refuted by it ; when h is statistical , pre I h) typically
takes an intermediate value. The uniform treatment that this
affords is unavailable in non-Bayesian methodologies, whose
advocates have instead developed a specific system, known as
Classical Statistical inference or sometimes as Frequentism, to
deal with statistical theori es.
This system, with its ' significance tests', ' conf idence inter-
vals' , and the rest, swept the board for most of thc twentieth cen-
tury, and its influence is still considerable. The challenge to
Bayesian methodology posed by Frequentism requires an answer,
and this we shall gi ve in the present and succeeding chapters.
5.a Falsificationism in Statistics
The simple and objective mechanism by which a hypothcsis may.
under certain circumstances, be logically refuted by observation-
al evidence could never work with statistical hypotheses, for these
ascribe probabilities to possible events and do not say of any that
they will or will not actually occur. The fact that statistical theo-
ries have a respected place in science and are regul arly tested and
132 CHAPTER 5
evaluated through experiment is therefore an embarrassment to
the methodology of falsificationism. In consequence, defenders
of that methodology have tried to take account of statistical theo-
ries by modifying its central dogma.
The modified idea acknowledges that a statistical hypothesis
is not strictly falsifiable, and what it proposes is that when an
event occurs to which the hypothesis attaches a sufficiently
small probability, it should be deemed false; scientists, Popper
said, should make "a methodological decision to regard highly
improbable events as ruled out- as prohibited" and he talked of
hypotheses then being "practically falsified" (1959a, p. 191).
The mathematician and cconomist Cournot (1843, p. ISS)
expressed the samc idea when he said that events of sufficient
improbability "are rightly regarded as physically impossible".
But is it right? After all, a distinctive feature of statistical
hypotheses is that they do not rule out cvents that they class as
improbable. For example, the Kinetic Theory attaches a tiny prob-
ability to the event of ice spontaneously forming in a hot tub of
water, but does not rule it out; indeed the fact that the theory
reveals so strange an event as a possibility, contrary to previous
opinion, is one of its especially interesting features. And even
though this particular unlikely event may never materialize,
immensely improbable events, which no one would regard as
refuting the Kinetic Theory, do occur all the time, for instance, the
spatial distribution at a particular moment of the molecules in this
jug of water.
Or take the simple statistical theory that we shall frequently
use for the purpose of illustration, which claims of some particu-
lar coin that it has a physical probability, constant from throw to
throw, of 1 of landing heads and the same probability of landing
tails (the coin is said then to be 'fair'). The probability of any par-
ticular sequence of heads and tails in, say, 10,000 tosses of the
coin is 2 - 10.000, a minuscule value, yet it is the probability of every
possible outcome of the experiment, one of which will definitely
occur. The implication of the Cournot-Popper view that this defi-
nite occurrence should be regarded as physically impossible is
clearly untenable.
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 133
S.b Fisherian Significance Tests
Fisher was inspired by both the falsificationist outlook and the ideal
of objectivity when, building on the work of Karl Pearson and WS.
Gossett (the latter, writing under the pen name' Student'), he devel-
oped his system of significance tests for testing statistical theories.
Fisher did not postulate a minimal probability to represent physical
impossibility, and so avoided the problem that destroys the
Cournot-Popper approach. His proposal, roughly speaking, was
that a statistical hypothesis should be rejected by experimental evi-
dence when it is, on the assumption of that hypothesis, contained in
a certain set of outcomes that are unlikely, relative, that is,
to other possible outcomes of the experiment.
Before assessing how well they are suited to their task, let us
set out more precisely the nature of Fisher's significance tests,
which we shall illustrate using, as the hypothesis under test (what
Fisher called the null hypothesis), the fair-coin hypothesis men-
tioned above. To perform the test, an experiment must be devised:
in our example, it will involve flipping the coin a predetermined
number of times, say 20, and noting the result; this result is then
anal ysed in the following four stages.
1 First, specify the outcome ,Ipace, that is, all the results that
the experiment could have produced. In our example, this would
normally be taken to comprise the 2
20
possible sequences of 20
heads or tails. (We examine the assumptions underlying the spec-
ification of any outcome space in the next section when we dis-
cuss 'stopping rules'.) The result of a coin-tossing experiment
would not normally be reported as a point in the outcome space
just described but would be summarized in some numerical form,
and for the purpose of our example, we shall select r, the number
of heads in the outcome. Such a numerical summary when used
in a significance test is known as a test-statistic; it is formally a
random variable, as defined in Chapter 2. (We shall presently dis-
cuss the basis upon which test-statistics are chosen.)
2 Next, calculate the probability, relative to the null hypoth-
esis, of each possible value of the test-statistic-its sampling
distribution. In general, if the probability of getting a head in a
134 CHAPTER 5
coin-tossing experiment is p, and of getting a tail is q, then r heads
will appear in n tosses of the coin with probability "C,p'q"-,. I In
the present case, p = q = and n = 20. The required probabilities
can now be directly calculated; they are shown in Table 5.1 and
also displayed graphically below.
TABLE 5.1
The Probabilities of Obtaining r Heads in a Trial consisting of 20
Tosses of a Fair Coin
Number oj'
Heads (r) Proba/Jilitl'
----------
0 9 X 10
7
1 1.9x10
5
2 2 x 10
4
3 0.0011
4 0.0046
5 0.0148
6 0.0370
7
(J.(l739
8 0.1201
9 0.1602
10 0.1762
0.2
Probability.
given the 0.1
null hypothesis
0.0
o 5
Number oj'
Heads (r)
1 1
12
13
14
15
16
17
18
19
20
10
Number of heads
Probability
0.1602
0.1201
0.0739
0.0370
(l.0148
0.0046
0.0011
2 x 10-
4
1.9 X 10-
5
9 x 10-
7
15
in 20 throws of the coin
20
I This familiar fact is demonstrated in standard statistics textbooks. "C, is equal to
(/I')
(11-1')' 1")
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 135
3 The third stage of Fisher's analysis requires us to look at all
the results which could have occurred and which, relative to the
null hypothesis, are, as Fisher put it, "more extreme than" the result
that did occur. In practice, this vague expression is interpreted prob-
abilistically, the requirement then being that we examine possible
outcomes of the trial which, relative to the null hypothesis, have a
probability less than or cqual to the probability of the actual out-
come. We should then calculate the probability (p*) that the exper-
imental result will fall within this group. (p* is often called the
p-vallle of the result.)
To illustrate, suppose our experiment produced 4 heads and 16
tails, which we see from the table occurs, if the null hypothesis is
true, with probability 0.0046. The results with less or equal prob-
ability to this are r = 4,3,2, 1,0 and r = 16,17,18,19,20 and
the probability of anyone of them occurring is the sum of their
separate probabilities, viz:
p * = 2 x (0.0046 + (>.00 II + 2 x 10 4 + 1. 9 x 10 5 + 9 x 10 7) =
0.012.
4 A convention has grown up, following Fisher, to reject the
null hypothesis just in case p * ,;:; 0.05. However, some statisticians
recommend 0.0 I or even 0.00 I as the critical probability. The crit-
ical probability that is adopted is called the significance level of
the test and is usually labelled n. I f an experimental result is such
that p* ,;:; n , it is said to be significant at the n significance level,
and the null hypothesis is said to be rejecled al Ihe (J. (or I ()()CJ.
percent) level.
In our example. the coin produced 4 heads when flipped 20
times, corresponding to p* = 0.012; since this is below 0.05. the
null hypothesis should be rejected at the 0.05 or 5 percent level.
But a result of6 heads and 14 tails, withp* = 0.115, would not be
significant, and so the null hypothesis should then not be rejected
at that level.
This simple example illustrates the bare bones of Fisher's
approach. It is, however, not always so easy to apply in practice.
Take the task often treated in statistics textbooks of testing
whether two populations have the same means, for instance,
136 CHAPTER 5
whether two groups of children have the same mean IQ. It may not
be feasible to take measurements from every child, in which case,
the recommended procedure is to select children at random from
each of the groups and compare their IQs. But to perform a signif-
icance test on the results of this sampling one needs a test-statistic
with a determinate and known distribution and these are often dif-
ficult to find. A solution was found in the present case by 'Student',
who showed that provided the experimental samples were suffi-
ciently large to ensure approximate normality, the so-called t-statis-
tic
2
has the appropriate properties for use in a significance test.
Which Test-Statistic?
Fisher's theory as so far expounded is apparently logically incon-
sistent. This is because different random variables may be defined
on any given outcome space, not all of them leading to the same
conclusion when used as the test-statistic in a significance test;
one test-statistic may instruct you to reject some hypothesi s when
another tells you not to.
We can illustrate this very simply in relation to our coin-toss-
ing experiment. We there chose the number of heads in the out-
come as the test-statistic, which, with 20 throws of the coin, takes
values from 0 to 20. Now define a new statistic, r', with values
from 0 to 18, derived from the earlier statistic by grouping the
results as indicated in Table 5.2. In this slight modification, the
outcome 5 heads and the outcome J 0 heads are counted as a si n-
gle result whose probability is that of obtaining either one of
these; similarly, for the results J 4 and J 5 heads. This new statistic
is artificial, having no natural meaning or appeal , but according to
the definition, it is a perfectly proper test-statistic.
It will be recalled that previously, with the number of heads as
the test-statistic, the result 6 heads. J 4 tails was not significant at
the 0.05 level. It is easy to see that using the modified statistic,
this result now is significant at that level (p* = 0.049). Hence
Fisher's principles as so far described tell us both to reject and not
to reject the null hypothesis, which is surely impossible. Clearly
2 See Section 6.c for morc dctails of the I-statistic.
:::LASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 137
fABLE 5.2
fhe Probability Distribution of the " -Statistic
-----
Value ~ l Value of
Statistic (r') Probability Statistic (r') Probability
------------ .-
0(0 heads)
9 x 10 7
10 (I I heads) 0.1602
1 (I heads) 1.9 x lO S
II (12 heads) 0.1201
2 (2 heads)
2 x 10-4
12 (13 heads) 0.0739
3 (3 heads) 0.0011 13 (14 or IS heads) 0.0518
4 (4 heads) 0.0046 14 (16 heads) 0.0046
5 (6 heads) 0.0370 15 (17 heads) 0.0011
6 (7 heads) 0.0739 16 (18 heads) 2 x 10
4
7 (8 heads) 0.1201 I 7 (19 heads) 1.9xlO
s
8 (9 heads) 0.1602 18 (20 heads)
9 x 10-
7
9 (5 or 10 heads) 0.1910
----- - -
est-statistics need some restriction that wi ll ensure that all per-
nissiblc ones lead to simi lar conclusions. And any such restri c-
ion must be recommended by more than the consistency it
)rings; it must produce the right consi stent result, if there is one,
or the right reasons, if there are any.
'he Chi-Square Test
'\ striking illustration of the difficulties posed by the multiplic-
ty of possible test-statistics is the chi-square (or X 2) goodncss-
If-fit test, whi ch bulks large in the literature and is widel y used
o test hypotheses that ascribe probabiliti es to several different
ypes of event, for example, to the outcomes of rolling a partic-
dar die. Suppose the die were rolled n times and landed with a
ix, fi ve, etc. showing uppermost with frequencies 06' 0
S
, ,
) ,. Ifp, is the probability that the null hypothesis ascribes to the
,utcome i, then np, is the expected Fequency (E) of that out-
ome. The null hypothesis is tested by the following so-called
hi-square statistic:
138 CHAPTER 5
the sum being taken over all the possible outcomes of the trial.
Karl Pearson discovered the remarkable fact, very helpful for
its application to significance tests, that thc probability distribu-
tion of this statistic is practi call y independent of the unknown
probabilities and of 11 and is dependent just on the number, v, of
the test's so-called degrees of Fee do Ill, where v = J - J, and J is
the number of separate cells into which the outcome was divided
when calculating X2. The probability density distributions of X2,
for various values of v , are roughl y as follows:
v = 5
o 2 4 6 8 10 12 14 16
We may illustrate the x
2
-test with a simpl e example. Let the
null hypothesis assert that a particular dic is ' true', that is, has
equal probabiliti es, of t, constant from throw to throw, of fa lling
with each of its sides uppermost. Now consider an experiment
involvi ng 600 roll s of the die, giving the results, say: six (90),flve
(91 ),jhur (125), Three (85), two (116), one (93).
To perform a ch i-square test, we must calculate X" for these
data, as follows (E = 600 X _ (1 = 100, for each i):
I 1
I
X
2
= 100 [(100 - 90)2 + (100 - 91)2+(100-125)2 +
(100 - 85)2 + (100 - 11 6)2 + (100 - 93)2] = 13.36.
Since the outcome involves six cells, the number of degrees of
freedom is five and, as can be roughly gauged from the above
sketches and more precisely established by consulting the appro-
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 139
priate tables, the probability of obtaining a value of X2 as large or
larger than 13.36 is less than 0.05, so the result is significant, and
the null hypothesis must therefore be rej ected at the correspon-
ding significance level.
Chi-square tests are also used to test theories asserting that
some population has a particular, continuous probability distribu-
tion, such as the normal distribution. To test such a theory, the
range of possible results of some sampl ing trial would be divided
into several intervals and the numbers of subj ects falling into each
would be compared with the 'expected' number, proceeding then
as with the examplc of the die.
Although the test has been much further developed and with
great technical ingenuity, it is, we believe, vitiated by the absence
of any principled rule for partitioning the outcomes into separate
intervals or cells, for not all partitions lead to the same inferences
when the significance test is applied. For instance, in our die-
rolling example, if we had based X" on just three cells, formed,
say, by combining thc pairs of outcomes [sLt,five], [four, three],
and (two, one], the result would not now be significant at the 5
percent level.
This problem is rarely taken up in expos itions of the chi-
square test. When it is, it is resolved by considerations of conven-
ience, not epistemology. For instance, Kendall and Stuart ( 1979,
p. 457) argued that the class boundaries should be drawn so that
each cell has the same probability (relative to the null hypothesis)
of containing the experimental outcome, and they defended this
rule on the epi stemicall y irrelevant grounds that it is "perfectly
definite and unique". But it is not even true that the equal-proba-
bility rule leads to a unique result, as we can see from our last
example. We there considered partitioning the outcomes of the
die-rolling experiment into three pairs: there are in fact fifteen
distinct ways of doing this, all satisfying the equal-probability
rule, and only two of them render the results significant at the 5
percent level.
The complacency of statisticians in the face of this difficul-
ty is remarkable. Although Hays and Winkler (1970, p. 195)
warn readers repeatedly and emphatically that "the arrangement
into population cLass intervals is arbitrary", their exposition
proceeds without recognizing that thi s renders the conclusions
140 CHAPTER 5
of a chi-square test equalZv arbitrary. Cochran (1952, p. 335)
claimed that the problem is a "minor" one, which merely calls for
"more standardization in the application of the test". But stan-
dardization would only institute the universal application of arbi-
trary principles and would not address the central problem of the
chi-square test, which is how to set it on a firm epistemic basis.
No such basi s appears to exist, and in view of this, the test should
be abandoned.
It might be argued that, despite its epistemic difficulties,
there are strong indications that the chi-square test is sound,
because the conclusions that are in practice drawn from it gen-
erally fit so well with intuition. In many cases, intuition is
indeed satisfied but the test can also produce quite counter-intu-
itive inferences, and once one such inference has been seen, it is
easy to generate more. Suppose, for instance, that the above trial
with the die had given the results: six (123), five (100), jintr
(100), three (100), two (100), one (77). In a test of the hypothe-
sis that the die was true, X
2
takes the value 10.58, which is not
significant at the 5 percent level, and any statistician who adopt-
ed this as the critical value would not be obliged to reject the
null hypothesis, even though the results of the trial tell us pretty
clearly that it is quite wrong.
3
In the examples we have so far considered, only large values
of X2 were taken as grounds for rejecting a hypothesis. But for
Fisher both extremities of any test-statistic 's distribution were
critical. In his view, a null hypothesis is "as definitely disproved"
when the observed and the expected frequencies are very similar,
leading to a very small X2, as it is when the frequencies are
sharply discrepant and X2 is large (Fisher 1970, Section 20). This
forms the basis of Fisher's famous criticism of Mendel's experi-
mcntal results, which we discussed above in 4.e. Those results, he
said, were "too good to be true", that is to say, although they
seemed to be in close accord with Mendelian theory, and were
usually taken to be so, they corresponded to X2 values that were
sufficiently small to imply its rejcction in a significance test. For
Fisher the chi-square test had to override intuitions in this case.
But this is not the universal opinion amongst classical statisti-
.> Good 1981 , p.161 . makes this point.
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATI ON 141
cians. For example, Stuart (1954) maintained that a small X
2
is
critical only if all "irregular" alternatives to the null hypothesis
have been ruled out, where the irregularity might involve "varia-
tions due to the observer himself", such as "all voluntary and
involuntary forms offalsification". Indeed, the Fisherian idea that
a null hypothesis can be tested in isolation, without considering
rival hypotheses, is not now widely shared and the predominant
form of the significance test, that of Neyman and Pearson, which
we discuss shortly, requires hypotheses to be tested against, or in
the context of, alternat ive hypotheses.
Sufficient Statistics
It is sometimes claimed that consistency may be satisfactorily
restored to Fisher's significance tests by restricting test-statistics
to so-called minimal-sl!lficient statistics, because of their standard
interpretation as containing all the information that is relevant to
the null hypothesi s and none that is irrelevant. We shall argue,
however, that this interpretation is unavailable to Fisher, that there
are no grounds for excluding irrelevant informat ion from a test,
and that the difficult y confronting Fisherian principles is uncon-
nected with the amount of information in the test-statistic, but lies
elsewhere.
Let us first examine the concept of a sufficient statistic.
Some statistics clcarly abstract more information from the out-
comes than others. For instance, tossing a coin four times will
result in one of the sixtecn sequences of heads and tails
(HHHH) , (THHH) , . . . , (TTTT), and a statistic that assigns dis-
tinct numbers to each element of this outcome space preserves
all the information produced by the experiment. But a statistic
that records only the number of heads thereby discards informa-
tion, so if you knew only that it took the value 3, say, you could
not determine from which of the four different outcomes con-
taining 3 heads it was derived. Whether some of the discarded
information is rel evant to an inference is a question addressed
by the theory of sufficiency.
A sample stati stic, t, is said to be sufficient, relative to a
parameter of interest, 8, if the probabi lity of any particular mem-
ber of the outcome space, given t, is independent of 8. In our
142 CHAPTER 5
example, the statistic representing the number of heads in the out-
come is in fact sufficient for 8, the physical probability ofthe coin
to land heads, as can be simply shown. The outcome space of the
coin-tossing experiment consists of sequences x = x I' ... , x"'
where each Xi denotes the outcome either heads or tails, and
P(r: I t) is given as follows, remembering that, since the value of t
is logically implied by x, P("( & t) = P(r:):
P(x & t) P(r:) W (1 - 8)" - ,
P(x I t) =-P(t)-- = P(t) = ;'-C,-() , (I -- fJ)" - , "C
,
Since the binomial term, "C" is independent of 8, so is P(r: I t) ;
hence, t is sufficient for 8.
It seems natural to say that if P(r: I t) is the same whatever the
parameter value, then x "can give us no information about 8 that
the sufficient statistic has not already given us" (Mood and
Graybill 1963, p. I 68). Certainly Fisher (1922, p. 3 16) understood
sufficiency that way: "The Criterion of Sufficiency", he wrote, is
the rule that "the statistic chosen should summarize the whole of
the relevant information supplied by the sample". But natural as it
seems, this interpretation is unavailable to Fisher, for a hypothe-
sis subjected to one of his significance tests may be rejected by
one sufficient statistic and not by another. Our coin-tossing exam-
ple illustrates this, for the statistic that summarizes the outcome
as the number of heads in the sample, and the statistic that assigns
separate numbers to each member of the outcome space are both
sufficient, as is the artificial statistic r', described above, though
these statistics do not generally yield the same conclusion when
used in a Fisherian test of significance.
Since the sufficiency condition does not ensure a unique con-
clusion, the further restriction is sometimes argued for (for exam-
ple by Seidenfeld 1979, p. 83) that the test-statistic should be
minima/-sufficient; that is, it should be such that any further
reduction in its content would destroy its sufficiency. A minimal-
sufficient statistic is thought of as containing all the information
supplied by the sample that is relevant, and none that is irrelevant.
But this second restriction has received no adequate defence;
indeed it would be surprising if a case could be made for it, for if
information is irrelevant, it should make no difference to a test, so
there should be no need to exclude it. It is curious that, despite the
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATI ON 143
almost universal lip service paid to the sufficiency condition, the
principal statistics that are in practice used in significance tests-
the X
2
, t and F stati stics- are none of them suffi cient, let alone
minimal-sufficient (Pratt 1965, pp. 169-1 70).
The idea of restricting admissible statistics according to their
information content seems in any case misconceived as a way of
saving Fisherian significance tests. For Neyman (1952, pp.
45-46) has shown that where the null hypothesis describes a con-
tinuous probability density distribution over the space, there may
be pairs of statistics that are related by a I-I transformation, such
that only one ofthem leads to the rejection (at a specified signif-
icance level) of the null hypothesis. Since these statistics neces-
saril y carry the same information, there must be some other
source of the trouble.
S.c Neyman-Pearson Significance Tests
Fisher's significance tests were designed to provide for the statis-
tical case something akin to the falsification avail able in the deter-
ministic case; hence hi s insistence that the tests should operate on
isolated hypotheses. But as we indicated earlier, stati stical
hypotheses cannot be refuted and, as we show later (Section 5.d),
Fi sher 's own analysis of and arguments for a quasi-refutation are
quite unsatisfactory. For this reason, Neyman felt that a different
epistemic basis was required for statistical tests, in particular, one
that introduces rival hypotheses into the testing process. The ver-
sion of significance tests that he and Pearson developed resem-
bled Fisher's however, in according no role to prior or posterior
probabilities of theories, for they were similarly opposed to
Bayesian methodol o!:,'Y.
In setting out the Neyman-Pearson method, we shall first
consider the simplest cases, where only two hypot heses, h I and
h2' are in competition. Neyman-Pearson tests permit two kinds
of inference: either a hypothesis is rejected or it is accepted. And
such inferences are subject to two sorts of error: you could
regard h I as false when in fact it is true, or accept hi (and, hence,
reject h
2
) when it is fa lse. When these errors can be distin-
guished by their gravity, the more serious is called a type 1 error
144 CHAPTER 5
and the less serious a type II error. The seriousness of the two
types of error is judged by the practical consequences of acting
on the assumption that the rejected hypothesis is false and the
accepted one true. For example, suppose two alternative
hypotheses concerning a food additive were admitted, one that
the substance is safe, the other that it is highly toxic. Under a
variety of circumstances, it would be less dangerous to assume
that a safe additive was toxic than that a toxic one was safe.
Neyman and Pearson, adapting Fisher's terminology, called the
hypothesis whose mistaken rejection is the more serious error
the null hypothesis, and where the errors seem equally serious,
either hypothesis may be so designated.
The possibilities for error are summed up in Table 5.3.
TABLE 5.3
Decision True Hypothesis
Reject hi Error
Accept hi / Error
The Neyman-Peason approach aims to minimize the chance of
committing both types of error. We will examine the Neyman-
Pearson approach through an example borrowed from Kyburg
( 1974, pp. 26- 35). The label on a particular consignment of tulip
bulbs has been lost and the purchaser cannot remember whether it
was the one that contained 40 percent of the rcd- and 60 percent
of the yellow-flowering sort, or 40 percent of the yellow and 60
percent of the red. We shall designate these possibilities h I and h:,
respectively, and treat the former as the null hypothesis. An exper-
iment to test these hypotheses might invol ve planting a predeter-
mined number of bulbs, say 10, that have been randomly selected
from the consignment, and observing which grow red and which
yellow. The testing procedure is similar to Fisher's and involves
the following steps.
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 145
First, specify the outcome space, which in the present case
may be considered to comprise 2
10
sequences, each sequence indi-
cating the flower-colour of the tulip bulb that might be selected
first, second, and so on, down to the tenth. Next, a test-statistic
that summarizes the outcome in numerical form needs to be stip-
ulated, and in this example we shall take the number of reds
appearing in the sample for this purpose. (We discuss the basis for
these arbitrary-seeming stipulations below.) Thirdly, we must
compute the probabil ities of each possible value of the test-statis-
tic, relative to each of the two rival hypotheses. If, as we shall
assume, the consignment of tulip bulbs is large, the probabil ity of
selecting r red-flowering bulbs in a random sample of n is approx-
imated by the familiar binomial function 11 Crpl"q 11 - r. We are
assuming here that the probability, p, of selecting a red-flowering
bulb is constant, an assumption that is more approximately true,
the larger the population of bulbs. In the present case, h I corre-
sponds to p = 0.40, and h] to p = 0.60. The sampling distributions
for the imagined trial, relative to the two hypotheses, are given in
Table 5.4 and displayed graphically, below.
TABLE 5.4
The Probabilities of Selecting r Red- and (10 - r) Yellow-flowering
Tulips
Outcome
hi
h,
(Red, Ye/hJH) (p = 0.40) (p = 0.60)
0,10 0.0060 0.0001
1,9 0.0403 0.0016
2, 8 0.1209 0.0106
3, 7 0.2150 0.0425
4,6 0.2508 0.1115
5, 5 0.2006 0.2006
6,4 0.1115 0.2508
7, 3 0.0425 0.2150
8,2 0.0106 0.1209
9,1 0.0016 0.0403
10, 0 0.0001 0.0060
--
146 CHAPTER 5
0.25
0.20
Probability 0.15
relative to
h, and h, 0.10
0.05
0.00 b ~ : : : : : : : : ~ - - - - r - ' - - - - r - - - - r - ~ ~ ~ ~
o
2 3 4 5 6 7 8 9 10
The number of red tulips in a sample of 10
Finally, the Neyman-Pearson method calls for a rule that will
determine when to reject the null hypothesis. Consider the possi-
bility of rejecting the hypothesis just in case 6 or more red-flow-
ering plants appear in the sample. Then, if hi is true, the
probability of a rejection may be seen from the table to be: 0.111 5
+ 0.0425 + 0.0106 + 0.0016 + 0.0001 = 0.1663, and thi s is there-
fore the probability of a type I error associated with the postul at-
ed rejection rule. This probability is called, as before, the
significance level of the test, or its size. The probability of a type
II error is that of accepting h I when it is false; on our assumption
that one of the two hypotheses is true, this is identical to the prob-
abil ity of rejecting h.' when it is true, which may be calculated
from the tablc as 0.3664.
The povver of a test is defined as I - P(type II error ) and is
regarded by advocates of this approach as a measure of how far
the test 'discriminates' between the two hypotheses. It is also the
probability of rejecting the null hypothesis when it is fal se, and in
the present case has the value 0.6336.
In selecting the size and power of any test, a natural ideal
might seem to be to try to minimize the former and maximize the
latter, in order to reduce as far as possible the chances of both
types of error. We shall, in due course, consider whether an ideal
couched in terms of type I and type II errors is suited to inducti ve
reasoning. We have to note straightaway, though, that the ideal is
incoherent as it stands, for its twin aims are incompatibl e: in most
cases, a diminution in size brings with it a contraction in power,
and vice versa. Thus, in our example, if the rejection rule were
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 147
changed and hi rejected in the event of at least 7 red tulips in the
sample, the size of the test would be reduced from 0.1663 to
0.0548, but its power would also be lower - 0.3823, compared
with 0.6336. We see then that while the revised test has a smaller
size, this advantage (as it is judged) is offset by its smaller power.
For this reason, Neyman and Pcarson proposed instead that one
first fix the size of a test at an appropriate level and, thus con-
strained, then maximize its power.
Randomized Tests
It is generally held amongst classical statisticians that the size of
a significance test should not exceed 0.05 and, for a reason we
shall describe later, practitioners are often exhorted always to
employ roughly the same significance levels. But with the meth-
ods introduced so far the size of a test cannot always be chosen at
will. For this purpose, randomized tests have been devised, which
Kyburg has lucidly explained in the context of the example we
have been discussing. Suppose a test of size 0.10 were desired.
Let the two tests considercd abovc bc labelled I and 2. As wc
showed, they have thc following characteristics:
TABLE 5.5
Probability ola Pmbability ola
Ope I error type 11 ermr P01,l'er
Test I 0.1663 0.3664 0.6336
Test 2 0.0548 0.6177 0.3823
Imagine, now, a third test which is carried out in the following
manner: a pack of 200 cards, of which 119 are red and 81 black,
is well shuffled, and one of these cards is then randomly select-
ed. If the selected card is black, test I is applied, and if red, test
2. This mixed or randomized test has the required size of 0.10,
given by
148 CHAPTER 5
81 119
x 0.1663 + - x 0.0548 = 0.100.
200 200
The corresponding probability of a type II error is similarly cal-
culated to be 0.5159; so the power of the mixed test is 0.484l.
Readers might be surprised by the implication that inspecting
a piece of coloured card, whose causal connexion to the tulip con-
signment is nil , can nevertheless provide an insight into its com-
position. Randomized tests are rarely if ever used, but they form
a proper part of the Neyman-Pearson theory, so any critici sm that
they merit can quite correctly be re-directed to the Neyman-
Pearson theory in general.
The Choice of Critical Region
An advantage that Neyman-Pearson significance tests enjoy over
Fisher's is that they incorporate in a quite natural way a feature that
Fi sher seems to have adopted arbitrarily and in deference merely
to apparent scientific practice, namel y, to concentrate the critical
region in (one or both of) the tails of the sampling di stribution of
outcomes. Fisher's reasoning seems to have been that evidence
capabl e of rejecting a hypothesis must be very improbable and
should lie in a region of very low probability (see Secti on 5.d). But
Neyman pointed out that by this reasoning, Fi sher could equally
well have chosen for the rejection regi on a narrow band in the cen-
tre of a bell-shaped distribution as a broader band in its tails.
By contrast, in the Neyman-Pearson approach, the critical
region is uniquely determined, according to a theorem known as the
Fundamental Lemma. Thi s states that the critical region of maxi-
mum power in a test of a null hypothesis, h I' against a rival , h2' is
the set of points in the outcome space that satisfies the inequality:
P(x I h
1
)
-- _. ~ k
PV .. I h)
where k is a constant that depends on the hypotheses and on
the signifi cance level.
4
The probabiliti es may also be densities.
4 Strictly speak ing. the likeli hoods P{y I h I) . ?(x I h cJ should not be expressed
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTI MATION 149
The lemma embraces randomi zed tests, the critical region then
comprising those of the component non-randomized tests, which
are selected at random, as already described.
Neyman-Pearson Tests and Sufficient Statistics
N eyman- Pearson tests have another fortunate consequence,
namely, that for them sufficient statistics do contain all the rele-
vant information. For if hI and h 2 ascribe di fferent values to a
parameter 8, and if t is a sufficient statistic relative to the out-
comes x = XI' ... , x
n
' then, by definition, P(, I t) is independent of
8, and it follows almost directly that
P(X I hi)
P(x I h
2
)
Pet I hi)
pet I h
2
)
The above lemma tells us that the left-hand ratio does not
exceed some number k; hence, the same holds also for the ri ght-
hand ratio. So if the outcome were summari zed in terms of t,
rather than x, the region of maximum power would comprise
the same outcomes, and consequently, none of the information
in X that is omitted from t is relevant to the significance test
inference.
S.d Significance and Inductive Significance
'The null hypothesis was rejected at such-and-such a significance
level' is a technical expression that simply records that an experi-
mental result fell in a certain designated 'rej ection region' of the
outcome space. But what does it mean as an inductive conclusion
about the hypothesis? There are three principal views on this
amongst advocates of the significance test. None, we shall argue,
is in the least satisfactory.
here as conditional probabi lities. for thcse presuppose that the hypotheses them-
selves have a probabili ty. something thai classical stati sticians strenuously deny.
Hence, they are sometimes written P( x: Ii) or L(x I Ii). Bayes ians. of course. need
have no such qualms.
ISO CHAPTER 5
Fisher's View
Fisher took the process of logical refutation as the model for his
significance tests. This is apparcnt in his frequently voiced claim
that such tests could "disprove" a theory (for example, 1947, p.
16), and that "when used accurately, [they] are capable of reject-
ing or invalidating hypotheses, in so far as these are contradicted
by the data" (1935; our italics).
Fisher seems to be saying here that statistical theories may
actually be falsified, though, of course, he knew full well that this
was impossible, and in his more careful accounts he took a differ-
ent line.
5
The force of a test of significance, he said (1956, p. 39),
"is logically that of the simple disjunction: Either an exceptional-
ly rare chance has occurred, or the theory of random distribution
[i.e., the null hypothesis] is not true". But in thus avoiding an
unreasonably strong interpretation, Fisher fell back on one that is
unhelpfully weak, for the significant or critical results in a test of
significance are by definition improbable, relative to the null
hypothesis. Inevitably, therefore, a significant result is either a
"rare chance" (an improbable event) or the null hypothesis is
false, or both. And Fisher's claim amounts to no more than this
empty truism.
6
Significant Results and Decisions to Act
Neyman and Pearson proposed what is now the most widely
adopted view, namely, that on 'accepting' a hypothesis after a sig-
nificance test, one should act as if one believed it to be true, and
if one 'rejects' it, one's actions should be guided by the assump-
tion that it is false, "without", as Lindgren (i 976, p. 306) put it,
"necessarily being convinced one way or the other". Neyman and
Pearson (1933, p. 142) defended their rule by saying that although
; The careless way that Fisher sometimes dcscribed the inductive meaning of a
significant result is often encountered in statistics texts. For example, Bland
(1987, p. 158) concludes from one such result "that the data are not cOllsistent
with the null hypothesis"; he then wanders to the further conclusion that the
alternative hypothesis is "morc likely".
(, Hacking 1965, p.81, pointed this out.
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 151
a significance test "tells us nothing as to whether in a particular
case h is true"7, nevertheless
it may often be proved that if we behave according to ... [the rule] ,
then in the long run we shall reject h when it is true not more, say,
than once in a hundred times [when the signif icance level is 0.0 I],
and in addition we may have evidence that we shall reject h sufficient-
ly often when it is false [i.e. , when the test's power is suffi ciently
large ].
This is a surpnsIng argument to encounter in this context.
After ali, the signi f icance test idea was born out of the recogni-
tion that events with probability p cannot be proved to occur with
any particular frequency, let alone with frequency p; indeed, they
may never occur at all. This is acknowledged tacitly in the above
argument, through the proviso " in the long run", a prevarication
that suggests some largish but practicaliy accessible number, yet
at the same time also hints at the indefinite and infinite. The for-
mer suggestion is, as we have said, unsustainabl e; the latter would
turn the argument into the unhelpful truism that with a signifi-
cance level of 0.01, we would reject a true null hypothesi s with
probability 0.01. Either way, the argument does not uphold the
Neyman-Pearson rejection rule.
There are also objections to the idea of acti ng as if a hypoth-
esis were definit ely true or defi nitely false when one is not con-
vinced one way or the other. If, to go back to our earlier
example, one were to reject the hypothesis that the tulip con-
signment contained 40 percent of the red variety and then act as
if it were definitely false, there would be no incentive to repeat
the experiment and every incent ive to stake all one's worldly
goods, and whatever other goods one might possess, on a wager
offered at any odds on the hypothesis being true. The idea is clear-
lyabsurd.
Neyman (1941 , p. 380), on the other hand, argued that there
are in fact occasions when it is reasonable to behave as if what one
beli eved to be fal se were actually true, and vice versa, citing the
7 These are our itali cs. Neyman's view that no inductive inferences arc lice nsed
by sampling information is discussed in Section S.f.2 below.
152 CHAPTER 5
purchase of holiday insurance as a case in point. In making such
a purchase, he said, "we surely act against our firm belief that
there wi II be no accident; otherwise, we would probably stay at
home". This, however, seems a perverse analysis of the typical
decision to take out insurance. We surely do not firmly believe
that there will be no accident when we go away, but regard the
eventuality as more or less unlikely, depending on the nature of
the holiday, its location, and so forth; and the degree of risk per-
ceived is reflected in, for example, the sum we are prepared to lay
out on the insurance premium.
Another example that is sometimes used to defend the idea of
acting as if some uncertain hypothesis were true is industrial qual-
ity control.
g
The argument is this. Suppose an industrialist would lose
money by marketing a product-run that included more than a
certain percentage of defective items. And suppose product-runs
were successively sampled, with a view to testing whether they
were of the loss-making type. In such cases, there could be no
graduated response, it is claimed, since the product-run can
cither be marketed or not; but, the argument goes, the industri-
alist could be comforted by the thought that "in the long run" of
repeatedly applying the same significance test and the same
decision rule, only about, say, S percent of the batches marketed
will be defective, and that may be a financially sustainable fail-
ure rate.
But this argument does not succeed, for the fact that only two
actions are possible does not imply that only two beliefs can be
entertained about the success of those actions. The industrialist
might attach probabilities to the various hypotheses and then
decide whether or not to market thc batch by balancing those
probabilities against the utilities of the possible consequences of
the actions, in the manner described by a branch of learning
known as Decision Theory. Indeed, this is surely the more plausi-
ble account.
X Even some vigorous opponents of the Neyman-Pearson method, such as,
A. W. F. Edwards ( 1972, p. 176) accept this defence.
CLASSICAL INFERENCE: SIGNIFI CANCE TESTS AND ESTIMATI ON 153
Significance Levels and Inductive Support
The fact that theories are not generally assessed in the black-and-
white terms of acceptance and rej ection is acknowledged by many
classical stati sticians, as we see from attempts that have been
made to find in significance tests some graduated measure of evi-
dential support. For example, Cramer ( 1946, p. 421- 23) wrote of
results being "almost significant", "significant" and "highly sig-
nificant", depending on the value ofthe test-stati stic. Although he
cautiously added that such terminology is "purely conventional ",
it is cl ear that he intended to suggest an inverse relationship
between the strength of evidence against a null hypothesis and the
significance level that would just lead to its rej ection.
9
Indeed, he implies that when this signi ficance level exceeds
some (unspecified) value, the evidence ceases to have a negative
impact on the null hypothesis and starts to support it; thus, when
the X
2
value arising from some of Mendel's experiments on pea
plants was a good way from rejecting Mendel's theory at the 5
percent level, Cramer concluded that "the agreement must be
regarded as good", and, in another exampl e, when the hypothesis
would only be rej ected if the signifi cance level were as hi gh as
0.9, Cramer said that "the agreement is very good".
Classical stati sticians commonly try to superimpose this sort
of notion of strength of evidence or inductive support on their
analyses. For instance, Weinberg and Goldberg (1990, p. 291):
"The test result was significant, indicating that HI ... was a more
plausible statement about the true value of the population mean
. . . than [the null hypothesi s] H
r
/' . The words we have italici sed
would, of course, be expected in a Bayesian analysis, but they
have no legitimacy or meaning within classical philosophy. And
the gloss whi ch the authors then add is not any cl earer or better
founded: "all we have shown is that there is reason to believe that
[HI is true]". And, of the same result, which was very improbabl e
according to the null hypothesis and signi ficant at the 0.0070
q The signifi cance level that. for a given resul t, would j ust lead to the rej ecti on
of a null hypothesis is also called the p-value of that result , as we stated earli er.
"The lower the p-value, the less plausibl e thi s null hypothesis . .. and the more
plausibl e are the alternati ves" (Wood 2003, p. 134).
154 CHAPTER 5
level, they say that it is "quite inconsistent with the null hypothe-
sis" (ihid., p. 282).
But the result is not "inconsistent" with the null hypothesis, in
the logical sense of the term. And as no useful alternative sense
seems to exist-certainly none has been suggested-the term in
this context is quite misleading. And the project of linking signif-
icance levels with strength of evidence has no prospect of success.
To prove such a link, you would need to start with an appropriate
concept of evidential or inductive support; in fact, no such con-
cept has been formulated in significance test terms, nor is one
likely to be. This, for two compelling reasons. First, the conclu-
sions of significance tests often flatly contradict those that an
impartial scientist or ordinary observer would draw. Secondly,
significance tests depend on factors that it is reasonable to regard
as extraneous to judgments of evidential support. We deal with
these objections in the next three subsections.
A Well-Supported Hypothesis Rejected in a
Significance Test
The first objection was developed in considerable generality by
Lindley, 1957, and is sometimes referred to as Lindley's Paradox.
We illustrate it with our tulip example. Table 5.6 lists the numbers
of red tulips in random samples of size n that would just be suffi-
cient to reject the null hypothesis at the 0.05 level.
It will be noticed that as 11 increases, the critical proportion of
red tulips in the sample that would reject h / at the 0.05 level
approaches more closely to 40 percent, that is, to the proportion
hypothesized in hI" Bearing in mind that the only alternati ve to h /
that the example allows is that the consignment contains red tulips
in the proportion of 60 percent, an unprejudiced consideration
would clearly lead to the conclusion that as 11 increases, the sup-
posedly critical values support h / more and more.
The table also includes information about the power of each
test, and shows that the classical thesis that a null hypothesis may
be rejected with greater confidence, the greater the power of the
test is not borne out; indeed, the reverse trend is signalled.
Freeman (1993, pp. 1446-48) is one of the few to have pro-
posed a way OLlt of these difficulties, without abandoning the
CLASSICAL INFERENCE: SIGNI FI CANCE TESTS AND ESTIMATION 155
TABLE 5.6
The sample size.n
10
20
50
100
1,000
10,000
100,000
The numher
tulips (expressed as a
proportion o/"n) that
wouldjust reject hi
at the 5% level.
0.70
0.60
0.50
0.480
0.426
0.4080
0.4026
------,_ .. . ---------
The power of the
test against h2
0.37
0.50
0.93
0.99
1.0
1.0
1.0
-------
basic idea of the significance test. He argued that Neyman and
Pearson should not have for mulated their tests as they did, by
first fixing a significance level and then selecting the rejection
region that maximizes power. It is this that renders them vulner-
abl e to the Lindley Paradox, because it means that the inducti ve
import of a rejection at a given significance level is the same
whatever the size of the sa mple. Instead, Freeman proposes that
the primary role should go to the likelihood ratio- that is, the
ratio of the probabilities of the data relat ive to the null and an
alternati ve hypothesis. And he argued that in a signi f icance test,
the rule should be to rej ect the null hypothesis if the likelihood
ratio is less than some f ixed value, on the grounds that this
ensures that the probabil it ies of hoth the type I and the type II
errors dimini sh as the sample size increases.
Freeman's rule is a version of the so-called Likelihood
Principle, according to which the inductive force of evidence is
contained ent irely in the likelihood rati os of the hypotheses under
consideration. This principl e, in fact, foll ows directly from Bayes's
theorem (see Section 4.c) and is unavoidable in Bayesian inductive
inference. Freeman (1993, p. 1444) too regards thi s principl e as
essential-"the one secure foundation for all of stati sti cs"-but
156 CHAPTER 5
neither he nor any other non-Bayesian has proved it. And this is not
surprising, for they strenuously deny that hypotheses have proba-
bilities, and it is precisely upon this idea that the Bayesian proof
depends. The likelihood principle therefore cannot save signifi-
cance tests from the impact of Lindley's Paradox, which, it seems
to us, shows unanswerably and decisively that inferences drawn
from significance tests have no inductive significance whatever.
We now consider a couple more aspects of significance tests
which reinforce this same point.
The Choice of Null Hypothesis
In a Neyman-Pearson test you need to choose which of the com-
peting hypotheses to treat as the null hypothesis, and the result <;If
that choice has a bearing on which is finally accepted and which
rejected. Take the tulip example again: if an experiment showed 50
red-flowering plants in a random sample of 100, then h, (40 per-
cent red) would be rejected at the 0.05 level if it were the null
hypothesis, and h2 (60 percent red) would be accepted. But with h]
as null hypothesis, the opposite judgment would be delivered! It
will be recalled that the role of null hypothesis was filled by con-
sidering the desirability, according to a personal scale of values, of
certain practical consequences of rejecting a true hypothesis; and
where the hypotheses were indistinguishable by this practical yard-
stick, the null hypothesis could be designated arbitrarily. But prag-
matic and arbitrary decisions such as these have no epistemic
meaning and cannot form the basis of inductive support.
Another sort of influence on significance tests that is also at
odds with their putative role in inductive reasoning arises through
the stopping rule.
The Stopping Rule
Significance tests are performed by comparing the probability of
the outcome obtained with the probabilities of other possible out-
comes, in the ways we have described. Now the space of possible
outcomes is created, in part, by what is called the stopping rule;
this is the rule that fixes in advance the circumstances under
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 157
which the experiment should stop. Our trial to test the fair-coin
hypothesis, for example, was designed to stop after the coin had
been flipped 20 times. Another stopping rule for that experiment
might have instructed the experimenter to end it as soon as 6
heads appeared, which would exclude many of the outcomes that
were previously possible and introduce an infinity of new ones.
Expressed as the number of heads and tails in the outcome, the
possibilities for the two stopping rules are: (20,0), (19,1), ... ,
(0,20), in the first case, and (6,0), (6,1), (6,2), ... , and so on, in
the second. The two stopping rules have surprisingly and pro-
foundly different effects.
Consider, for example, the result (6,14), which could have
arisen with either stopping rule. When the rule was to stop after 6
heads, the null hypothesis would be rejected at the 0.05 level. This
is shown as follows: the assumed stopping rule produces the result
(6, i) whenever (5, i), appearing in any order, is then succeeded by
a head. Thus, relative to the fair-coin hypothesis, the probability
of the result (6, i) is given by it5CS( )5( Y X . Table 5.7 shows
the sampling distribution.
TABLE 5.7
The Probabilities of Obtaining i Tails with a Fair Coin in a Trial of
Designed to Stop after 6 Heads Appear.
--------
Outcome Outcome
(H,T) Probability (H,T) Probabi/i(v
6,0 0.0156 6, II 0.0333
6,1 0.0469 6,12 0.0236
6,2 0.0820 6,13 0.0163
6,3 0.1094 6,14 0.0 III
6,4 0.1230 6,15 0.0074
6,5 0.1230 6,16 0.0048
6,6 0.1128 6,17 0.0031
6,7 0.0967 6,18 0.0020
6,8 0.0786 6,19 0.0013
6,9 0.0611 16,20 0.0008
6,10 0.0458 6,21 0.0005
etc. etc.
158 CHAPTER 5
We see from the table that the results which are at least as
improbable as the actual one are (6,14), (6, J 5), ' . . , and so on,
whose combined probability is 0.0319. Since this is below the
critical value of 0.05, the result (6,14) is significant at this level
and the null hypothesis should therefore be rejected. It will be
recalled that when the stopping rule predetermined a sample size
of 20, the very same result was not significant. 10 So in calculating
the significance of the outcome of any trial , it is necessary to
know the stopping rule that informed it.
We have considered just two stopping rules that could have
produced some particular result, but any number of others have
that same property. And not all of these other possibilities rest the
decision to stop on the outcomes themselves, which some statis-
ticians regard as not quite legitimate. For instance, suppose that
after each toss of the coin, you drew a playing card at random
from an ordinary pack, with the idea of calling the trial off as soon
as the Queen of Spades has been drawn. This stopping rule intro-
duces a new outcome space, which will lead to different conclu-
sions in certain cases. Or suppose the experimenter intends to
continue the trial until lunch is ready: in this case, the sampling
distribution could only be worked out with complex additional
information about the chance, at each stage of the trial, that prepa-
rations for the meal are complete.
The following example brings out clearly how inappropriate it
is to involve the stopping rule in the inductive process: two scien-
tists collaborate in a trial , but are privately intent on different stop-
ping rules; by chance, no conflict arises, as the result satisfies
both. What then are the outcome space and the sampling distribu-
tion for the trial? To know these you would need to discover how
each of the scientists would have reacted in the event of a dis-
agreement. Would they have conceded or insisted, and if they had
put up a fight, which of them would have prevailed? We suggest
that such information about experimenters' subjective intentions,
their physical strengths and their personal qualities has no induc-
tive relevance whatever in this context, and that in practice i! is
never sough! or even contemplated. The fact that significance
10 This illustration of the stopping-rule clfeet is adapted from Lindley and
Phillips 1976.
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 159
tests and, indeed, all classical inlerence models require it is a
decisive objection to the whole approach.
Whitehead (1993) is one of the few to have defended the stop-
ping rule as an essential component of the inductive process. He
denies that the subjective intention underlying the stopping rule is
irrelevant, illustrating his point with a football match, of all
things, in which the captain of one side is allowed to decide when
the game should finish, and in fact blows the whistle when his
team is 1-0 ahead. Whitehead remarks that learning the stopping
rule here would reduce his high opinion of the winning side. To
revert to a case where classical statistics can more obviously be
applied, this is analogous to an experimenter, who is predisposed
in favour of one of the hypotheses, deciding to stop sampling as
soon as more red than yellow tulips have flowered. If the final
count were, say, I red and yellow, we would indeed not be much
swayed in favour of the experimenter's preferred hypothesis, but
not because of the known bias, or the stopping rule, rather, we
suggest, because of the smallness of the sample. To believe other-
wise runs into the objection we raised earlier, namely, that if the
biased experimenter were working with an impartial, or different-
ly biased colleague, who was actuated by a different stopping rule,
you would have to delve into the personal qualities of the experi-
menters in order to discover the outcome space of the experiment,
and hence the inductive significance of the result.
Experimenters' prejudices can only have inductive signifi-
cance for us if we believe them to have clairvoyant knowledge
about future samples; but this is just what a random sampling
experiment effectively precludes. On the other hand, the captain
in charge of the stopping rule in the hypothetical football match
does have information about the likely course of the game, since
he may know the teams' recent form and can observe how well
each side is presently playing. But a football game is not a random
sampling experiment, and is therefore an unsuitable example in
this context.
Gillies (1990, p. 94) also argued that the stopping rule is an
essential part of a scientific inference. He claimed that "to those
who adopt falsificationism (or a testing methodology)" it "seems
natural and only to be expected" that the stopping rule should in
general affect a theory's empirical support, because "wherever
160 CHAPTER 5
possible the experimental method should be applied, and this con-
sists in designing and carrying out a repeatable experiment .. .
whose result might refute h [the null hypothesis]". This, he
claims, means that the stopping rule is evidentially relevant.
Tn response, we certainly concede that it can do no harm and
might do good to repeat an experiment. But why should it be
repeatable? Many useful and informative tests cannot be repeat-
ed: for example, pre-election opinion polls and certain astronom-
ical observations. Would our confidence in the age of the Turin
Shroud be any different if the entire cloth had been consumed in
the testing process, so precluding further tests? Surely not.
Moreover, in an important sense, no experiment is repeatable,
for none could ever be done in exactly the same way again.
Indefinitely many factors alter between one performance of an
experiment and another. Of course, not all such changes matter.
For instance, the person who tossed the coin might have worn yel-
low shoes or sported a middle parting; but these are irrelevant,
and if you called for the experiment to be repeated, you would
issue no instructions as to footwear or hairstyle. On the other
hand, whether or not the coin had a piece of chewing gum
attached to one side, or a strong breeze was blowing when it was
tossed should be taken into account. The question then is whether
the stopping rule falls into the first category of irrelevant factors
or into the second of relevant ones. Gillies (ibid., p. 94) simply
presumes the latter, arguing, with reference to the coin trial, that
the "test of h in this case consists of the whole carefully designed
experimental procedure", and suggesting thereby that this proce-
dure must include reference to the stopping rule. But Gillies nei-
ther states this explicitly nor provides any reason why it should be
so-unavoidably, in our view.
We show in Chapter 8 that in the Bayesian scheme thc poste-
rior probabilities in each case are unaffected by the subjective
intentions implicit in the stopping rules and depend on the result
alone. Thus, if the experimental result is, for instance, 6 heads, 14
tails, it does not matter whether the experimenter had intended to
stop the trial after 20 tosses of the coin, or after 6 heads, or after
lunch, or after the Queen of Spades has made her entrance, or
whatever.
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 161
S.e Testing Composite Hypotheses
We have, so far, restricted our account of the Neyman-Pearson
method to cases where just two, specific hypotheses are assumed
to exhaust the possibilities. But such cases are atypical in practice,
and we need to look at how Neyman and Pearson extended and
modified their approach to deal with a wider range of alternative
hypotheses. They considered, for instance, how to test a hypothe-
sis, h J' that some population parameter, (), has a specific value, say
(-}J' against the unspecific, composite hypothesis, h2' that () > (}r
The principle of maximizing the power of a test for a given
significance level cannot be applied where these are the compet-
ing hypotheses. For, although the situation allows one to deter-
mine a critical region corresponding to any designated
significance level, as before, the probability of a type II error is
indeterminate. Neyman and Pearson responded by varying their
central principle.
They first of all proposed that in such cases one should choose
for the critical region one that has maximum power for each com-
ponent of h]. Tests sati sfying this new criterion are called
Uniformly Most Power/it! (UMP). But they ran into the problem
that, relative to some elements of h 2' the power of UMP tests
might be very low, indeed, lower than the significance level , in
which case there would be a greater chance of rejecting the null
hypothesis when it is true than when it is false. This possibility
was unacceptable to Neyman and Pearson and to avoid it, they
imposed the further restriction that the test should be unbiased,
that is, its power relative to each element of h;} should be at least
as great as its significance level.
We can illustrate the idea of tests that are both UMP and unbi-
ased (UMPU) with a simple example. The test will be based on
the mean, X, of a random sample drawn from a normal population
with known standard deviation and unknown mean, (). The dia-
gram shows the sampling distributions relative to the null hypoth-
esis h J: (-) = (-}J' and relative to an arbitrary element, hi' of the
composite alternative hypothesis h
2
: (-) > (}r
162 CHAPTER 5
AN UMPU TEST
Power 0' the test
h. h
[Dotted plJS hatc hed area)
Consider a critical region for rejecting hi consisting of points
to the right of the critical value, xc' The area of the hatched por-
tion is proportional to the significance level. The area under the
hi-curve to the right of Xc represents the probability of rejecting
hi if h; is true (and hence, hi false) . Clearly, the closer are the
means specified by hi and hi' the smaller this probability will be.
But it could never be less than the significance level. In other
words, the test is U M pu.
Suppose now that the range of alternatives to the null hypoth-
esis is greater still and that h / A = A, is to be tested against
h.': H ;;e HI' the standard deviation again being known. A critical
region located in one tail of the sampling distribution associated
with h I would not now constitute an unbiased test. But an UMPU
test can bc constructed by dividing the critical region equally
between the two tails . This can be appreciated diagrammatically
and also rigorously shown (see for instance Lehmann 1986). An
UMPU test thus provides some basis for the two-tailed tests
implied by Fisher's theory, for which he offered no rationale.
But UMPU tests are in fact rather academic, since they exist
in very few situations. And they depart somewhat from the
Neyman-Pearson ideal of maximum power for a givcn signifi-
cance level, in that the power of such a test can never be deter-
mined; so in particul ar cases, it may, for all we know, be only
infinitesimally different from the significance level. More seri-
ously, the modifications introduced to meet the challenge of com-
posite hypotheses are equally afflicted by the various difficulties
we have shown to di scredit even the most uncomplicated form of
the significance test.
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 163
5.f Classical Estimation Theory
Scientists often estimate a physical quantity and come thereby to
regard a certain number, or range of numbers, as a more or less
good approximation to the true valuc. Significance tests do not in
general deliver such estimates and the need tor them has prompt-
ed classical statisticians to develop a distinct body of doctrine
known as Estimation Theory. The theory is classical, in that it pur-
ports to provide objective, non-probabilistic conclusions. It has
two aspects, namely, point estimation and interval estimation,
both of which we regard as fallacious, as wc explain in the follow-
. .
mg reView.
S.t.l . Point Estimation
Point estimation differs from interval estimation, which we deal
with below, in offering a single number as the so-called 'best esti-
mate' of a parameter. Suppose a population parameter, such as its
mean, is in question. The technique for estimating this is to draw a
random sample of predetermined size, 11, from the population, and
to measure each element drawn. Then, letting x = x l' ... ,x/1 denote
the measurements thus derived, the next step is to pick an estimat-
ing statistic, f, this taking the form of a calculable function t =j(.r) .
Finally, the best estimate of the unknown parameter is inferred to
be to' the value of t yielded by the experiment. But not every sta-
tistic is accepted as an estimator. The authors of this approach to
estimation have specified certain conditions that any estimator
must meet; the most frequently mentioned being the conditions of
sutficieJ1(Y, ul1iJiasedness, and e.Uiciency. We discuss
these in turn.
Sufficient Estimators
It will be recalled from Section S.b, that a statistic I is sufficient
for f} when P(:r I t) is independent of (1. The present requirement
is that any estimating statistic should be sufficient in this sense.
The mean of a random sample satisfies the requirement when it is
164 CHAPTER 5
used to estimate a population mean, but the sample range, for
example, is not. Nor is the sample medi an. I I
Sufficiency is a Bayesian requirement too. Expressed in
Bayesian terms a statistic, t, is sufficient for e, just in case P(x I t) =
P(x I t & e) , for all e. It follows strai ghtforwardly from Bayes's
theorem that t is sufficient for e if, and only if, P(8 I t) = p(el x).
Hence, when a sample statistic is sufficient, it makes no differ-
ence whether you calculate the posteri or di stributi on using it or
using the full experimental informati on in x; the results will be the
same. In other words, a sufficient stati sti c contains all the infor-
mation that in Bayes ian terms is relcvant to 8.
A compelling intuition tells us that, in evaluating a parameter,
we should not neglect any relevant informati on. There is a sati s-
factory rati onale for this. Suppose you have two bits of informa-
tion, a and h. There arc then three posterior distributions to
consider: p(e I (I), P(8 I b) and pre I a & h). If these differ, which
should describe your current beli ef state? The Bayesian has no
choice, for P(8 I a) is your distributi on of beli efs were you to learn
a and nothing else. But you have in fact learned a & h and noth-
ing else. Therefore, your current beli ef state must be described by
p(el (I & b), rather than by the other mathemati cal possibiliti es.
The injuncti on to use all the rel evant evidence in an inductive
inference, whi ch Carnap (1947) call ed thc Total Evidence
Requirement, is often considered to be an independent postulate.
This is true, at any ratc, within cl assical estimation theory, which
also has to rely on the intuition, whi ch it cannot prove either, that
a suffi cient stati sti c captures all the relevant evidence. The intu-
itions are we ll founded, but their source, in our opinion, is Bayes's
theorem, appli cd unconsciously.
Unbiased Estimators
These are defined in terms of the expectati on, or expected val ue,
of a random vari abl e, which is givcn by E(.\) = LXjP(X), the sum
I I The sampl e !tIl1ge is the difference between the highest and lowest measure-
ments: if the sample measurements are arra nged in increasi ng order, and if 11 is
odd, the ir mediall is the middle element of t he seri es: if II is even, the median is
the hi gher of t he midd le two elements.
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 165
or in the continuous case, the integral, bcing taken ovcr all possi-
ble values of Xi' We mentioned in Chapter 2 that the expectation
of a random variable is also callcd the mean of its probability or
density distribution; when the distribution is symmetrical, its
mean is also its geometric centre. A statistic is defined as an unbi-
ased estimator of () just in case its expectation equals the parame-
tcr's true value. The idea is often glossed by saying that the value
of an unbiascd statistic, averaged over repeated samplings, will
"in the long run", be equal to the parameter being estimated. 12
Many intuitively satisfactory estimators arc unbiased, for
instance the proportion of red counters in a random sample is
unbiased for the corresponding proportion in the urn from which
it was drawn, and the mean of a random sample is an unbiased
estimator of the population mean. However, sample variance is
not an unbiased estimator of population varianee and is generally
n
"corrected" by the factor -- .
n - 1
But unbiasedness is neither a necessary nor a sufficient condi-
tion for a satisfactory estimation. We may see this through an
example. Suppose you draw a sample, of predetermined size, from
a population and note the proportion of individuals in the sample
with a certain trait, and at the same time, you toss a standard coin.
We now posit an estimating statistic which is calculated as the
sample proportion plus k (> 0), if the coin lands heads, and plus k'
. if it lands tails. Then, if k = - k', the resulting estimator is unbi-
ased no less than the sample proportion itself, but its estimates are
very different and are clearly no good. If, on the other hand, k' =
0, the estimator is biased, yet, on the occasions when the coin lands
tails, the estimates it gives seem perfectly fine.
Not surprisingly, then, one finds the criterion defended, if at
all, in terms which have nothing to do with epistemology. The
usual defence is concerned rather with pragmatics. For example,
Barnett claimed that, "within the classical approach unbiased-
ness is often introduced as a practical requirement to limit the
class of estimators" (1973, p. 120; our italics). Even Kendall and
Stuart, who wrote so confidently of the need to correct biased
12 See for example Hays 1963, p. 196.
166 CHAPTER 5
estimators, conceded that they had no epi stemi e basis for this
censorious attitude:
There is nothing except convenience to exalt the arithmetic mean
above other measures of location as a criteri on of bi as. We mi ght
equal/v well have chosen the median of the di stributi on of t or its
mode as determining the "unbiased" estimator. The mean value is
used, as al ways, fbr its mathematical convenience. ( 1979, p. 4; our
itali cs)
These authors went on to warn their readers that "the term
' unbiased' should not be allowed to convey overtones of a non-
technical nature". But the tendentious nature of the terminology
makes such misl eading overtones hard to avoid. The next criterion
is also named in a way that promises more than can be deli vered.
Consistent Estimators
An estimator is defined to be consistent when, as the sample size
increases, its probability distribution shows a diminishing scatter
about the parameter's true value. More precisely, a statistic
derived from a random sample of size n is a consistent estimator
for e if, for any positive number, E, P( I ( -- e I :s E) tends to I, as
n tends to infinity. Thi s is sometimes described as f tending prob-
abilistically to e.
There is a probl em with the consistency criterion as described,
because it admits estimators that are clearl y inadmissible. For
example, if Til is a consistent estimator, so is the estimator, Tn',
defined as equal to zero for n:s IOili and equal to 7'" for n > 1010.
Fisher therefore added the further restriction that an admissible
estimator should, in Rao's words, be "an expli cit function of the
observed proporti ons only". So, if the task is to estimate a popu-
lation proportion, e, the estimator should be a consistent functi on
just of the corresponding sample proportion, and it should be such
that when the observed and the population proportions happen to
coincide, the estimator gives a true estimate ( Rao 1965, p. 283).
Thi s adjustment appears to eliminate the anomalous estimators.
Fi sher believed that consistency was the "fundamental criteri-
on of estimation" ( 1956, p. 141) and that non-consistent estima-
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 167
tors "should be regarded as outside the pale of decent usage
(1970, p. 11). In this, Neyman (1952, P 188) agreed "perfectly"
with Fisher, and added his opinion that "it is definitely not prof-
itable to use an inconsistent estimate." 13 Fisher defended his
emphatic view in the following way:
as the samples arc made larger without limit, the statistic will usual-
ly tend to some fixed value characteristic of the population, and
therefore, expressible in terms of the parameters of the population.
f ~ therefore, such a statistic is to be used to estimate these parame-
ters, there is only one parametric function to which it can properly be
equated. If it be equated to some other parametric function, we shall
be using a statistic which even from an infinite sample does not give
a correct value .... (1970, p. 1 1 )
Fisher's claim here is that because a consistent estimator con-
verges to some parameter value, it "can properly be equated" to
that value, and it should be equated to no other value, because,
if the sample were infinite, it would then certainly give the
wrong result. This is more assertion than argument and in fact is
rather implausible in its claims. Firstly, one should not, without
qualification, equate an unknown parameter with the value
taken by a statistic in a particular experiment; for, as is agreed
on all sides, such estimates may, almost certainly will be in
error, a consideration that motivates interval estimation, which
we discuss below. Secondly, the idea that a consistent estimator
becomes more accurate as the sample increases, and perfectly so
in the limit, implies nothing at all about its accuracy on any par-
tiCLdar occasion. Arguing for an estimator with this idea in mind
would be like defending the use of a dirty measuring instrument
on the grounds that if it were cleaner it would be better; in
assessing a result, we need to know how good the instrument
was in the experiment at hand, not how it might have performed
under different conditions.
And (rebutting Fisher's last point) just as estimates made by
consistent estimators may be quite inaccurate and clearly wrong,
those from non-consistent ones might be very accurate and
13 Presumably this should read: 'estimator'.
168 CHAPTER 5
clearly right. For instance, suppose x + (n - 100)x were chosen
to estimate a population mean. This odd statistic is non-consis-
tent, for, as the sample size grows, it diverges ever more sharply
from the population mean. Yet for the special case where n = 100,
the statistic is equivalent to the familiar sample mean, and gives
an intuitively satisfactory estimate.
Efficient Estimators
The above criteria arc clearly incomplete, because they do not
incorporate the obvious desideratum that an estimate should
improve as the sample becomes larger. So, for instance, a sample
mean that is based on a sample of 2 would be 'sufficient', ' unbi-
ased' and 'consistent', yet estimates of the population mean
derived from it would not inspire confidence, certainly not as
much as when there are 100, say, in the sample. This consideration
is addressed by classical statistics through the efficiency criterion:
the smaller an estimator's variance about the parameter value, the
more efficient it is said to be, and the better it is regarded. And
since the variance of a sample statistic is generally inversely
dependent on the size of the sample, the efficiency criterion
reflects the preference for estimates made with larger samples.
But it is not easy to establish, in classical terms, why efficien-
cy should be a measure of quality in an estimator. Fisher (1970, p.
12) stated confidently that the less efficient of two statistics is
"definitely inferior ... in its accuracy"; but since he would have
strayed from classical principles had he asserted that particular
estimates were certainly or probably correct, even within a mar-
gin of error, this claim has no straightforward meaning. Kendall
and Stuart's interpretation is the one that is widely approved. A
more ctTicient statistic, they argued, will "deviate less, on the
average, from the true value" and therefore, "we may reasonabl y
regard it as better" (1979, p. 7). Now it is true that ife'J and e
i
2
are
the estimates delivered by separate estimators on the ith trial and
if (j is the true value of the parameter, then there is a calculable
probability that 1 e'I - f) 1 < 1 e
l
} - (j I, which will be greater the
more efficient the first estimator is than the second. Kendall and
Stuart translate this probability into an average frequency in a
long run of trials, which, as we already remarked, goes beyond
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATI ON 169
logic. But even if the translation were correct, the performance of
an estimator over a hypothetical long run implies nothing about
the closeness of a particular estimate to the true value. And since
estimates are usuall y expensive and troublesome to obtain and
often inform practical actions, what is wanted and needed are just
such evaluations of particular estimates.
5.f.2. Interval Estimation
In practice, this demand is evidently met, for estimates are nor-
mally presented as a range of numbers, for example, in the form
e = a b, not as point values, which, as Neyman observed, "it is
more or less hopeless to expect . .. wi ll ever be equal to the true
value" (1952, p. 159). Bayesians would qualify an interval esti-
mate by the subjective probability that it contains the true value
(see the discussion of 'credible intervals' in Section 8.a) .
Neyman's theory of conf idence intervals, developed around 1930,
and now dominant in the field, was intended to give a classical
expression to this idea.
Confidence Intervals
Consider the task of estimating the mean height, e, of the people
in a large population, whose standard deviation, LT, is known. A
sample of some predetermined size, 11, is randomly selected and
its mean, X, is noted. This mean can take many possible values,
some more probable than others; the distribution representing this
situation is approximately normal (the larger the population, the
closer the approximati on) with a mean equal to that of the popu-
lation and a standard deviation given by ~ , = CJ 11 ~
The sampling distribution plots possible sample means against
probability densities, not probabiliti es, and, as explained earlier,
this signifi es that the probability that x lies between any two
points is proportional to the area encl osed by those points and the
curve. Because the distribution is essentially normal, it follows
that, wi th probability 0.95,
170 CHAPTER 5
~ ~ ~ L = x
The sampling distribution of means from a population
with mean fI and standard deviation (J,
And this implies that, with probability 0,95,
x - l. 96 cr :5 8:5 X + l. 96 (J ,
n /I
Let 111 be the value of x in a particular experimental sample;
since fJ and n are known, the terms 111 - 1.96 (J and m + 1.96 (J
/1 11
can be computed. The interval between these two values is called
a 95 percenl confidence interval for fJ. Clearly there are other con-
fidence intervals relating to different regions of the sampling dis-
tribution, and others, too, associated with different probabilities.
The probability associated with a particular confidence interval is
called its confidence coefficient.
The probability statements given above arc simply deductions
from the assumptions made and arc unquestionably correct ; and
what we have said about confidence intervals, being no more than
a definition of that concept, is also uncontroversial. Controversy
arises only when confidence intervals are assigned inducti ve
meaning and interpreted as estimates of the unknown parameter.
What we shall call the categorical-assertion interpretation and the
slIbjecli\'e-confidence interpretation are the two main proposals
for legitimizing such estimates. We deal with these in turn.
The Categorical-Assertion Interpretation
This interpretation was first proposed by Neyman and has been
widely adopted. Neyman said (1937, p. 263) that the "practical
statistician", when estimating a parameter, should calculate a con-
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTI MATI ON 17 1
fidence interval and then "state" that the true value lies between
the two confidence bounds, in the knowledge that (when the con-
fidence coefficient is 0.99) "in the long run he will be correct in
about 99 percent of all cases". The statistician's statement should
not signify a belief in its truth, however. Indeed, Neyman (1941,
p. 379) rejected the very idea of reasoning inducti vely to a con-
clusion, because he beli eved that "the mental process leading to
knowledge .. . can onl y be deducti ve". Inducti on, for Neyman,
was rather a matter of behaviour, and in the case of interval esti-
mates, the proper outcome was a decision "to behave as if we
actually knew" that the parameter lies within the confidence
bounds.
We have already di scussed this interpretati on (in Section 5.d)
in the context of signi f icance tests and argued that typically and
more reasonably sci entists evaluate theories by degree; they do
not, and, moreover, should not act in the way that Neyman recom-
mended. A further indication that the interpretation is wrong ari s-
es from the fact that conf idence intervals are not unique, as we
explain next.
Competing Intervals
It is obvious from the sampling distribution of means depicted
above that indefinitely many regions of that normal distribution
cover 95 percent of its area. So instead of the usual 95 percent
confidence interval located at the centre of the di stribution, one
could consider asymmetrical confidence intervals or ones that
extend further into the tail s while omitting smaller or larger strips
in the centre. Neyman's categorical-asserti on interpretation
requires one to "assert" and "behave as if one actually knew" that
the parameter lies in each and everyone of thi s multiplicity of
possible 95 percent conf idence intervals, which is clearly unsatis-
factory, and indeed, paradoxical.
Defenders of the interpretation have reacted in two ways, both
we believe unsati sfactory. The first di scriminates between confi-
dence intervals on the basis of their length, and claims that, for a
given confidence coefficient, the shortest interval provides the best
estimate. In the words of Hays (1969, p. 290), there is "naturally
172 CHAPTER 5
... an advantage in pinning the population parameter within the
narrowest possible range with a given probability". By this criteri-
on, the centrally symmetrical interval m 1.960;, is the preferred
95 percent confidence interval for the population mean in the
example cited above. The preference is based on the idea that the
width ofa confidence interval is a measure of the 'precision' of the
corresponding estimate, and that this is a desirable feature. Thus
Mood (1950, p. 222), when comparing two 95 percent confidence
intervals, stated that the longer one was inferior, "for it gives less
precise information about the location" of the parameter.
But it is not true that the length of a confidence interval meas-
ures its precision. For, consider the interval I a, h I as an estimate
of e, and the interval I f(a), I(b) I as an estimate off(8). If I is a
1-1 function, the two estimates are equivalent and must be equal-
ly informative and therefore equally precise. But while the first
may be the shortest 95 percent confidence interval for e, the sec-
ond might not be the shortest such interval forf(e); this would be
the case, for instance, whenf(a) = a- I.
Another difficulty is that different sample statistics may yield
different minimum-length confidence intervals, a fact that has
prompted the proposal to restrict interval estimates to those given
by statistics with the smallest possible variance. It is argued that
although this new criterion does not guarantee the shortest possi-
ble confidence interval in any particular case, it does at least
ensure that such intervals " are the shortest on average in large
samples" (Kendall and Stuart 1979, p. 126). We have already crit-
icized both the long-run justification and the short-length criteri-
on, and since two wrongs don't make a right, we shall leave the
discussion there.
Neyman (1937, p. 282) suggested another way of discriminat-
ing between possible confidence intervals. He argued, in a man-
ner familiar from his theory of testing, that a confidence interval
should not only have a high probability of containing the correct
value but should also be relatively unlikely to include wrong val-
ues. More precisely: a best confidence interval , I", should be such
that for any other interval, J, corresponding to the same confi-
dence coefficient, P(8' E J I 8) p(e' E J I 8); moreover, the
"
inequality must hold whatever the true value of the parameter, and
for every value 8' different from e. But as Neyman himself
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 173
showed, there are no 'best' intervals of this kind for most of the
cases with which he was originally concerned.
The Subjective-Confidence Interpretation
Neyman's categorical-assertion interpretation, contrary to its
main intention, does, in fact, contain an element that seems to
imply a scale by which estimates may be evaluated and qualified,
namely the confidence coefficient. For suppose some experimen-
tally established range of numbers constituted a 90 percent confi-
dence interval for (), rather than the conventionally approved 95 or
99 percent, we would still be enjoined to assert that () is in that
range and to act as if we believed that to be true, though with a
correspondingly modified justification that now referred to a 90
percent frequency of being correct " in the long run". But if the
justification had any force (we have seen that it does not), it would
surely be stronger the lower the frequency of error. So the categor-
ical assertion that () is in some interval must after all be qualified
by an index running from 0 to 100 indicating how well founded it
is, and this is hard to distinguish from the index of confidence that
is explicit in the subjective-confidence interpretation that we deal
with now.
In this widely approved position, a confidence coefficient is
taken to be "a measure of our confidence" in the truth of the state-
ment that the confidence interval contains the true value (for
example, Mood 1950, p. 222). This has some surface plausibility.
For consider again the task of estimating a population mean, ().
We know that in experiments of the type described, () is included
in any 95 percent confidence interval with an objective probabil-
ity of 0.95; and this implies that if the experiment were performed
repeatedly, () would be included in such intervals with a relative
frequency that tends, in the limit, to 0.95. It is tempting, and many
professional statisticians find it irresistible, to infer from this limit
property a probability of 0.95 that the particular interval obtained
in a particular experiment does enclose (). But drawing such an
inference would commit a logical fallacy. 14
14 This fallacy is repeated in many statistics texts. For example, Chiang 2003.
(pp. 138 -39): "The probability that the interval contains II is either zero or one;
174 CHAPTER 5
The subjective-confidence interpretation seems to rely on a
misapplication of a rule of inference known as the Principle of
Direct Probability (see Chapter 3), which is used extensively in
Bayesian statistics. The principle states that if the objective, phys-
ical probability of a random event (in the sense of its limiting rel-
ative frequency in an infinite sequence of trials) is known to be r,
then, in the absence of any other relevant information, the appro-
priate subjective degree of belief that the event will occur on any
particular trial is also r. Expressed formally, if the event in ques-
tion is a, and P*(a) is its objective probability, and if a
r
describes
the occurrence of the event on a particular trial, the Principle of
Direct Probability says that P[a
r
I P*(a) = r] = r, where P is a sub-
jective probability function.
For example, the physical probability of getting a number of
heads, K, greater than 5 in 20 throws of a fair coin is 0.86 (see
Table 5.1 above), that is, P*(K > 5) = 0.86. By the Principle of
Direct Probability,
P[(K > 5)r I P*(K > 5) = 0.86] = 0.86.
That is to say, 0.86 is the confidence you should have that any par-
ticular trial of 20 throws of a fair coin will produce more than 5
heads . Suppose one such trial produced 2 heads. To infer that we
should now be 86 percent confident that 2 is greater than 5 would,
of course, be absurd; it would also be a mi sapplication of the prin-
ciple. For one thing, if it were legitimate to substitute numbers for
K, why would such substitution be restricted to its first occurrence
in the principle? But in fact, no such substitution is allowed. For
the above equation does not assert a general rule for each number
K from 0 to 20; the K-tenn is not a number, but a function that
no intermediate va lues are possible. What then is the initial probability of 0.95'1
Suppose we take a large number of samples, eaeh of size /1. For each sampic we
make a statement that the interval observed from the sa mple eOl1lains .LI. Some
of our statements will be true, others will not be. According to [the equation we
give in the text, above] ... 95 percent of our statements wi ll be true. In reality
wc take only one sample and make only one statement that the interval contains
,LI. Thus [si c] we do have confidence in our statement. The mcasure of our con-
fidence is the initial probability 0.95."
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTI MATI ON 175
takes different values depending on the outcome of the underly-
ing experiment.
Mistaking this appears to be the fallacy implicit in the subjec-
tive-confidence interpretation. It is true that the obj ective proba-
bility of f3 being encl osed by experimentally determined
95 percent confidence intervals is 0.95. If 11 and 12 are variables
representing the boundaries of such confidence interval s, the
Principle of Direct Probability implies that
and this tells us that we should be 95 percent conf ident that any
sampling experiment will produce an interval containing (J
Suppose now that an experiment that was actually performed
yielded 1'1 and 1'2 as the confidence bounds; the subjective-
confidence interpretati on would tell us to be 95 percent confident
that 1'1 :5 (}:5 1'2 . But thi s would commit exactly the same fallacy
as we exposed in the above counter-example. For I I and 1
2
, like K,
are functions of possibl e experimental outcomes, not numbers,
and so the desired substitution is blocked. IS
In response, it mi ght be said that the subjective-confidence
interpretation does not depend on the Principl e of Direct
Probability (a Bayesian notion, anyway), that it is justified on
some other basis. But we know of none, nor do we think any is
possible, because, as we shall now argue, the interpretation is fun-
damentally flawed, since it impl ies that one's conf idence in a
proposition should depend on information that is manifestly irrel-
evant, namely, that concerning the stopping rul e, and should be
independent of prior information that is manifestly re levant. We
address these two points next.
The Stopping Rule
Confidence intervals arise from probability di stributions over
spaces of possible outcomes. Although one of those out comes
I) Howson and Oddi c 1979 point ed out this misapplication of the principle in
another context. See also 3.f above.
176 CHAPTER 5
will be actualized, the space as a whole is imaginary, its contents
depending in part on the experimenters' intentions, embodied in
the adopted stopping rule, as we explained earlier. Estimating sta-
tistics employed in point estimation are also stopping-rule
dependent, because the stopping rule dictates whether or not those
statistics satisfy the various conditions that are imposed on esti-
mators. So, for instance, the sample mean is an unbiased estima-
tor of a population mean if it is based on a fixed, predetermined
sample size, but not necessarily otherwise.
16
The criticism we levelled at this aspect of classical inference
in the context of tests of significance applies here too, and we
refer the reader back to that discussion. Tn brief, the objection is
that having to know the stopping rul e when drawing an inference
from data means that information about the experimenters' private
intentions and personal capacities, as well as other intuitively
extraneous facts, is ascribed an inductive role that is highly inap-
propriate and counter-intuitive. This is, in a way, tacitly acknowl-
edged by most classical statisticians, who in practice almost
always ignore the stopping rule and standardly carry out any clas-
sical analysis as i[the experiment had been designed to produce a
sample of the size that it did, without any evidence that this was
so, and even when it clearly was not.
We take up the discussion of the stopping rule again in Chapter
8, where we show why it plays no rol e in Bayesian induction.
Prior Knowledge
Estimates are usually made against a background of partial
knowledge, not in a state of complete ignorance. Suppose, for
example, you were interested in discovering the average height of
students attending the London School of Economics. Without
being able to point to results from carefully conducted studies, but
on the basis of common sense and what you have learned infor-
mally about students and British universities' admission stan-
dards, you would feel pretty sure that this could not be below four
feet, say, nor above six. Or you might already have made an
1(, Sec, for example, Lee 1989, p. 213.
CLASSICAL INFERENCE SIGNIFICANCE TESTS AND ESTIMATION 177
exhaustive survey of the students' heights, lost the results and be
able to recall with certitude only that the average was over fi ve
feet. Now if a random sampl e, by chance, produced a 95 percent
confidence interval of3' 10" 2", you would be required by clas-
sical principles to repose an equivalent level of confidence in the
proposition that the students' average height really does lie in that
interval. But with all you know, this clearly would not be a credi-
ble or acceptable conclusion.
A classical response to thi s difficulty mi ght take one of two
forms, neither adequate, we beli eve. The first would be to restrict
class ical estimation to cases where no relevant information is
present. But thi s proposal is scarcely practicable, as such cases are
rare; moreover, although a littl e knowledge is certainly a danger-
ous thing, it would be odd, to say the least, if it condemned its pos-
sessor to continue in this condition of ignorance in perpetuity. A
second possibility would be to combine in some way informal
pri or information with the formal estimates based on random
samples. The Bayesian method expresses such information
through the prior distribution, which then contributes to the over-
all conclusion in a regulated way, but there is no comparable
mechanism within the confines of classical methodology.
5.g Sampling
Random Sampling
The classical methods of estimation and testing that we have been
considering purport to be entirely objective, and it is for thi s rea-
son that they call for the sampling di stributi on of the data also to
be objective. To thi s end, classical statisticians require the sample
that is used for the estimate to have been generated by an impar-
tial , physical process that ensures for each element of the popula-
ti on an objectively equal chance of being selected. Here is a
simpl e instance of such a process: a bag containing similar coun-
ters, each corresponding to a separate member of the popul ation,
and marked accordingly, is shaken thoroughly and a counter
selected blindfold; this selecti on is repeated the prescribed num-
ber of times, and the population members picked out by the
178 CHAPTER 5
selected counters then constitute a random sample. There are of
course other, more sophisticated physical mechanisms for creat-
ing random samples.
What we call the Principle ~ l Random Sampling asserts that
satisfactory estimates can only be obtained from samples that are
objectively random in the sense indicated.
Judgment Sampling
The Principle of Random Sampling may be contrasted with
another approach, which is motivated by the wish to obtain a rep-
resentative sample, one that resembles the population in all those
respects that are correlated with the characteristic being meas-
ured. Suppose the aim were to measure the proportion of the pop-
ulation intending to vote Conservative in a forthcoming election.
If, as is generally agreed, voting preference is related to age and
socio-economic status, a representative sample should recapitu-
late the population in its age and social class structure; quite a
number of other factors , such as gender and area of residence,
would, no doubt, also be taken into account in constructing such
a sample. A representative sample successfully put together in this
way will have the same proportion of intending Conservative vot-
ers as the parent population. Samples selected with a view to rep-
resentativeness are also known as purposive, or judgment
samples; thcy are not random. A kind of judgment sampling that
is frequently resorted to in market rcsearch and opinion polling is
known as qllota sampling, where interviewcrs are given target
numbers ofpcople to interview in various categories, such as par-
ticular social classes and geographical regions, and invited to
exercise their own good sense in selecting representative groups
from each specified category.
Some Objections to Judgment Sampling
Judgment sampling is held to be unsatisfactory by many statisti-
cians, particularly those of a classical stripe, who adhere to the
random sampling principlc. Three related objections are encoun-
tered. Thc first is that judgment sampling introduces an undesir-
CLASSICAL INFERENCE: SIGNI FICANCE TESTS AND ESTIMATION 179
able subjectivity into the estimation process. It does have a subjec-
ti ve aspect, to be sure, for when drawing such a sample, a view
needs to be taken on which individual characteristics are correlat-
ed with the population parameter whose value is being sought, and
which are not: a judgment must be made as to whether a person's
social class, age, gender, the condition of his front garden, the age
of her cat, and so forth, are relevant factors for the sampling
process. Without exhaustively surveying the population, you could
not pronounce categorically on the rel evance of the innumerable,
possibly relevant factors; there is, therefore, considerable room for
opinions to vary from one experimenter to another. This may be
contrasted with random sampling, which requires no individual
judgment and is quite impersonal and objective.
The second objection, which is, in truth, an aspect of the first,
is that judgment samples are susceptible to bias, due to the exper-
imenter's ignorancc, or through the exercise of unconscious, or
cven conscious, personal prejudices. Yates (1981, pp. 11-16)
illustrates thi s danger with a number of cases where the experi-
menter's careful efforts to select representative samples were frus-
trated by a failure to appreciate and take into account crucial
variables. Such cases are oftcn held up as a warning against the
bias that can intrude into judgment sampling.
Sampling by means of a physical randomizing process, on
the other hand, cannot be affected by a selector's partiality or
lack of knowledge. On the other hand, it might, by chance, throw
up samples that are as unrepresentati ve as any that could result
from the most ill-informed judgment sampling. Thi s seeming
paradox 17 is typically turncd into a principal advantage in the
standard cl assical response, which says that when sampling is
random, the probabilities of different possible samples can be
acc uratel y computed and then systematically incorporated into
the inference process, using classical estimation methods. But
jUdgment sampling-so thc third objection goes-does not lend
itself to obj ect ive methods of estimation.
There is another strand to the classical response, which invokes
the idea of slratifi'ed rando/11 sampling. This involves partitioning
17 See below, S.d, for a discussion of Stuart's description of this situation as the
"paradox of sampling".
180 CHAPTER 5
the population into separate groups, or strata, and then sampling at
random from each. The classically approved estimate of the popu-
lation parameter is then the weighted average of the corresponding
strata estimates, the weighting coefficients being proportional to
the relative sizes of the population and the strata. Stratified random
sampling seems clearly intended as a way of reducing the chance
of obtaining seriously unrepresentative samples. But the orthodox
classical rationale refers instead to the greater 'efficiency' (in the
sense defined above) of estimates derived from stratified samples.
For, provided the strata are more homogeneous than the popula-
tion, and significantly different from one another in relation to the
quantity being measured, estimation is more 'efficient' using strat-
ified random sampling than ordinary random sampling, and so, by
classical standards, it is better.
This rationale is however questionable; indeed, it seems quite
wrong. The efficiency of an estimator, it will be recalled, is a
measure of its variance. And the more efficient an estimator, the
narrower any confidence interval based on it. So, for instance, a
stratified random sample might deliver the 95 percent confidence
interval 4' 8" 3" as an estimate of the average height of pupils
in some school, while the corresponding interval derived from an
unstratified random sample (which, by chance, is heavily biased
towards younger children) might be, say, 3' 2" 6". Classical stat-
isticians seem committed to saying that the first estimate is the
better one because, being based on a more efficient estimating
method, its interval width is narrower. But this surely misapprais-
es the situation. The fact is that the first estimate is probably right
and the second almost certainly wrong, but these are words that
should not cross the lips of a classical statistician.
Some Advantages of Judgment Sampling
Judgment sampling has certain practical advantages. A pre-elec-
tion opinion poll, for example, needs to be conducted quickly, and
this is feasible with judgment sampling; on the other hand, draw-
ing up a random sample of the population, finding the people
who were selected and then persuading them to be interviewed is
costly, time consuming, and sometimes impossible, and the elec-
tion might well be over before the poll has begun. Practical COI1-
CLASSICAL INFERENCE: SIGNIFICANCE TESTS AND ESTIMATION 181
siderations such as these have established the dominance of quota
sampling in market research. "Probably 90 percent of all market
research uses quota sampling and in most circumstances it is suf-
ficiently reliable to provide consistent results" (Downham 1988,
p. 13).
A second point in favour of judgment and quota samples is
that they are evidently successful in practice. Opinion polls con-
ducted by their means, insofar as they can be checked against the
results of ensuing elections, are mostly more or less accurate, and
market research firms thrive, their services valued by manufactur-
ers, who have a commercial interest in accurately gauging con-
sumers' tastes.
A third practical point is that inferences based on non-random
samples are often confidently made and believed by others;
indeed they seem inevitable when conclusions obtained in one
sphere need to be applied to another, as commonly happens. For
instance, in a study by Peto et al. (1988), a large group of physi-
cians who had regularly taken aspirin and another group who had
not showed similar frequencies of heart attacks over a longish
period. Upon this basis, the authors of the study advised against
adopting aspirin generally as a prophylactic, their implicit and
plausible assumption being that the doctors taking part in the
study typified the wider population in their cardiac responses to
aspirin. Although the rest of the statistical procedures employed in
the study were orthodox, this assumption was not checked by
means of random samples taken from the
We consider the question of sampl ing methods again when we
discuss Bayesian inference in Chapter 8.
S.h Conclusion
Classical estimation theory and significance tests, in their various
forms, are still immensely influential; they are advocated in hun-
dreds of books that are recommended texts in thousands of institu-
tions of higher education, and required reading for hundreds of
IX Smith 1983 makes the same point in relation to another study. On the argu-
ments concerning sampling in this section, see also Urbach 1989.
182 CHAPTER 5
thousands of students. And the classical jargon of 'statistical
significance', 'confidence', and so on, litters the academic jour-
nals and has slipped easily into the educated vernacular. Yet, as we
have shown, classical 'estimates' are not estimates in any normal
or scientific sense, and, like judgments of 'significance' and ' non-
significance', they carry no inductive meaning at all. Therefore,
they cannot be used to arbitrate between rival theories or to deter-
mine practical policy.
A number of other objections that we have explored in this
chapter show, moreover, that classical methods are set altogether
on the wrong lines, and are based on ideas inimical to scientific
method. Principal here is the objection that all classical methods
involve an outcome space and hence a stopping rule, which we
have argued brings to bear on scientific judgment considerations
that are highly counter-intuitive and inappropriate in that context.
And classical methods necessarily introduce arbitrary elements
that are at variance not only with scientific practice and intuition,
but also with the objectivist ideals that motivated them. The
founders of the classical philosophy were seeking an alternative to
the Bayesian philosophy, which they dismissed as unsuited to
inductive method because it was tainted by subjectivity. It is there-
fore particularly curious and telling that classical methods cannot
operate except with their own, hefty subjective input. This was
frankly confessed, in retrospect, by one of the founders of the
classical approach:
Of necessity, as it seemed to us [him and Neyman], we left in our
mathematical model a gap for the exercise of a more intuitive
process of personal judgement in such matters ... as the choice of
the most likely class of admissible hypotheses, the appropriate sig-
nificance level, the magnitude of worthwhile effects and the balance
of utilities. (Pearson 1966, p. 277)
Classical distaste for the subjective element in Bayesian inference
puts one in mind of those who were once accused of taking infi-
nite trouble to strain out a gnat, while cheerfully swallowing a
camel!
CHAPTER 6
Statistical Inference
in Practice: Clinical Trials
We have thus far discussed various methodologies in terms suffi-
ciently abstract to have perhaps created the impression that the
question as to which of them is phil osophically correct has littl e
practical bearing. But this is far from being the case. We illustrate
this point in the present chapter by looking at Classical and
Bayes ian approaches to the scientific investigation of causal con-
nections, particularly in agricultural (or 'field') and medical (or
'clinical') trials. Large numbers of such trials are under way at
anyone time; they are immensely expensive; and their results may
exert profound effects on farming and clinical practice. And a fur-
ther pract ical effect, in the case of clinical trials, is the inconven-
ience to whi ch the parti cipants are put and the risks to which they
may be exposed.
6.0 Clinical Trials: The Central Problem
A clinical trial is designed with a view to discovering whether and
to what extent a particul ar drug or medical procedure alleviates
certain symptoms, or causes adverse side effects. And a typical
goal of an agricultural field tri al would be to invest igate whether
a putative fe rtilizer increases the yield of a certai n crop, or
whether a new, genetically engineered potato has improved
growth qualities.
Clinical trials typically involve two groups of subjects, all of
whom are currentl y sufferi ng from a particular medi cal condition;
one of the groups, the test group, is administered the experimen-
tal therapy, while the other, the control group, is not ; the progress
of each group is then monitored over a period. An agricultural
trial to compare a new variety of potato (A) with an establi shed
184 CHAPTER 6
variety (B) might be conducted in a field that is di vided into
'blocks', and then subdivided into plots, in which a seed of each
variety is sown. The field might look like this:
Plot 1 Plot 2
Block 1 A B
Block 2 A B
Block 3 A B
Block 4 A B
The question of causality that is posed in such trials presents a
special difficulty, for in order to demonstrate, for example, that a
particular treatment cures a particular disease, you need to know
not only that people have recovered after receiving the treatment
but also that those self-same people would not have recovered if
they hadn't received it. This seems to suggest that to establish a
causal link you must examine the results of simultaneously treat-
ing and not treating the same patients, under identical circum-
stances, something that is obviously impossible.
An alternative to this impossible ideal might be to conduct the
trial with groups of patients who are identical , not in eve,]!
respect, but just in those respects that are causally relevant to the
progress of the medical condition under study. Such causally rel-
evant influences are known as prognostic factors. Then, if a test
group and a control group were properly matched on all the prog-
nostic factors (with the possible exception of the experimental
treatment itself), and at the end of the trial they exhibited unequal
recovery rates or differed in some other measure of the symptoms,
then these variations can clearly be attributed to the experimental
treatment. A similar type of inference would also be available,
mutatis mutandis, in an agricultural trial, provided the seeds and
the plants into which they develop were exposed to the same
growth-relevant environments. This sort of inference, in which
every potential causal factor is laid out and all but one excluded
by the experimental information, is a form of what is traditional-
ly called eliminative induction.
STATI STI CAL INFERENCE IN PRACTICE: CLINICAL TRIALS 185
But the conditions for such an induction cannot be straight-
forwardly set up. For if you wished to ensure that every prognos-
tic factor will be equally represented in the experimental groups
of a trial , you apparently need a comprehensive li st of those fac-
tors. And as Fisher (1947, p. 18) pointed out, in every situation,
there are innumerably many possible prognostic fact ors, most of
which have not even been thought of, let alone tested for rele-
vance; and some of those that have been so tested might have
been mistakenly dismissed as causally inactive.
6.b i Control and Randomization
To meet this difficulty, Fisher distingui shed between factors that
are known to affect the course of the disease, or the growth of the
crop, and factors whose influence is unknown and unsuspected.
And he claimed that the misleading effects on the inference
process of these two kinds of potentially interfering influences
could be neutralised by the techniques of control and randomiza-
tion, respectively.
Control
A prognostic factor has been 'controll ed for' in a trial when it is
distributed in equal measure in both the test and the comparison
situations. So, for example, a clinical trial involving a disease
whose progress or intensi ty is known to depend on the patient's
age would be controlled for that prognostic factor when the test
and control groups have si milar age structures. Of particular con-
cern in clinical trials is a prognostic factor called the placebo
effect. Thi s is a beneficial , psychosomatic effect that arises simply
from the reassuring feel and encouraging expectations that are
created by all the paraphernalia of white-coated, medical atten-
tion.l The effect is controlled for in clinical trials by creating the
I Wall (1999) presents many fascinating examples of pl acebo effect s and
argues convincingly that these operate through the subjects ' expectations of
improvement.
186 CHAPTER 6
same expectations for recovery in both groups of patients, by
treating them in superficially similar ways. Where two different
treatments are to be compared, they should be disguised in such a
way that the participants in the trial have no idea which they are
receiving. And when there is no comparison treatment, the control
group would receive a placebo, that is to say, a substance that is
pharmacologically inert, yet looks and seems exactly like the test
drug.]
Agricultural trials are sensitive to variations in soil quality and
growing conditions. Fisher (1947, p. 64) advised controlling for
these factors by planting the seeds in soil "that appears to be uni-
form, as judged by [its] surface and texture ... or by the appear-
ance of a previous crop". And the plots on which the pl anting
takes place should be compact, in view of "the widely verified
fact that patches in close proximity are commonly more alike, as
judged by the yield of crops, than those which are further apart".
And when seeds are sown in plant-pots, the soil should be thor-
oughly mixed before it is distributed to the pots, the watering
should be uniform and the developing pl ants should be exposed to
the same amount of light (ibid. , p. 41 ).
Randomization
So much for the known prognostic factors. What of the unknown
ones, the so-called' nuisance variables ', whose influences on the
course of an experiment have not been controll ed for? These,
Fisher said, can only be dealt with through 'randomization'. A
randomized trial is one in which the experimental units are
assigned at random to the trial treatments. So, for example, in a
randomized agricultural trial designed to compare the perform-
ance of two types of potato, the plots on which each is grown are
selected at random. And when testing the efficacy of a drug in a
randomized clinical trial, patients are allocated randoml y to the
test and control groups.
2 In trials where the test treatment is not a drug. but some other sort of medical
or psychological intervention. the pl acebo takes a correspondingly different
form.
STATISTICAL INFERENCE IN PRACTICE: CLINI CAL TRIALS 187
This is not the place to discuss the thorny question of what
randomness really is. Suffice it to say that advocates of random-
ization in trials regard certain repeatable experiments as sources
of randomness, for example, throwing a die or flipping a coin,
drawing a card blindly from a well-shuffled pack, or observing the
decay or non-decay in a given time-interval of a radi oactive ele-
ment. Random number tables, which are constructed by means of
such random processes, are also regarded as providing effective
ways to perform the randomization. If, on the other hand, experi-
menters formed the experimental groups by allocat ing patients,
say, as the spirit moved them, the randomization rul e would not
have been respected, for the allocati on process would then not
have been objectively random in the sense just indi cated.
Fisher's randomizati on rule is wi dely viewed as a brilliant
solution to the problem of nuisance variables in experimental
design. In the opinion of Kendall and Stuart ( 1983, pp. 120-21),
it is "the most important and the most influential of [his] many
achievements in statistics". Certainly it has been influential, so
much so, that clinical and agricultural trials that are not properly
randomi zed are frequentl y written off as fatall y flawed, and regu-
latory bodi es will usuall y refuse to li cense drugs that have not
been tested in randomi zed trials.
We need to examine the far-reaching claims that are made for
the virtues of randomi zation, particularly in view of the fact,
whi ch we shall shortl y explain, that applying it may be costl y,
inconveni ent and ethicall y challenging. We shall in fact argue that
the standard defences of the procedure as an absolute necessity
are unsuccessful and that the problem of nuisance variables in tri-
al s cannot be solved by its means. In Chapter 8, we argue that
although randomization may sometimes be harml ess and even
helpful, it is not a sine qua non, not absolutely essenti al. We shall
maintain that the essential feature of a trial that permits a satisfac-
tory conclusion as to the causal efficacy of a treatment is the pres-
ence of adequate controls.
Two main arguments for randomi zation are propounded in
the literature. The first , which is based on the class ical statisti-
cal idea that significance tests are the central inferential tool ,
claims that such tests are valid only if the experiment was ran-
domized. The second argument, whil e also advanced by those of
188 CHAPTER 6
a classical outlook, evidently appeals to a modified form of
eliminative induction.
6.c I Significance-Test Defences of Randomization
Fisher was concerned by the fact that the comparison groups in a
clinical trial might be badly mismatched through the unequal dis-
tribution of one or more of the infinitely many possible, unknown
nuisance variables. For, in the event of such a mismatch, you
might be badly misled if, for example, you concluded definitely
that the group having the greater recovery rate had received the
superior treatment. Corresponding concerns arise with field trials.
Fisher's response was to say that although we cannot know for
certain that the groups are perfectly matched in regard to every
prognostic factor, the randomization process furnishes us with
objective and certain knowledge of the probabilities of different
possible mismatches, knowledge that is required for a valid test of
significance. And of course, Fisher favoured such tests as a way
of drawing conclusions from experimental results.
[T]he full procedure of randomisation [i s the method] by which the
validity of the test of significance may be guaranteed against corrup-
tion by the causes of disturbance which have not been eliminated [by
being controlled]. (Fisher 1947, p. 19)
Fisher's reason for regarding randomization as a necessary
feature of clinical trials has been widely accepted amongst classi-
cally minded statisticians. For example, Byar et al. (1976, p. 75):
"It is the process of randomization that generates the significance
test, and this process is independent of prognostic factors, known
or unknown" (our italics). This argument will not cut the mustard
with Bayesians, who do not acknowledge significance tests as
valid forms of inductive reasoning. But as we shall argue, it is
incorrect even in the classical context.
The Problem of the Reference Population
Fisher claimed that significance tests could only be applied to
clinical and agricultural trials if these were randomized. It is
STATISTICAL INFERENCE IN PRACTICE: CLINICAL TRIALS 189
therefore surprising that expositions of the standard signif icance
tests employed in the analysis of trial results, such as the t-test, the
chi-square test, and the Wilcoxon Rank Sum test, barely allude to
the proviso. Take the case of the t-test appl ied to a clinical trial
whose goal was to determine whether or not a particular treatment
has a beneficial effect on some medical disorder. In the trial, a
certain ' population' of people who are suffering from the condi-
tion in question is divided between a test and a control group by
the classically required randomization method. Suppose that the
trial records some quantitative measure of the disease symptoms,
giving means of XI and x
2
' respectively, in the test and control
groups. The difference XI - x
2
is likely to vary from experiment to
experiment, and the associated probability distribution has a cer-
tain standard deviation, or 'standard error', that is given by

0
2
0
2
SEC X - X ) = . '-- + _ 2
'2 n, n
l
'
where 01 and 02 are the popUlation standard deviations,
3
and
where n I and n
2
are, respectively, the sizes of two samples. I f the
null hypothesis affirms that the treatment has no average effect, a
test-statistic that may be employed in a test of significance is
X,- Xl
Z= -_
SE(x, - X) '
The situation is complicated because the two standard deviations
are usually unknown, in which case it is recommended that they
be estimated from the standard deviations in the corresponding
samples. And provided certain further conditions are met, the sta-
ti stic that is then proposed for use in the test of significance of the
null hypothesis is the I-statistic obtained from Z by substituting
the corresponding sample standard deviations for 01 and 02'
This and similar significance tests, as well as related estimat-
ing procedures, are directed to detecting and estimating differ-
ences between population parameters, and their validity therefore
J These are hypothetical standard devi at ions. That is. 01 (respectively, 0 " ) is the
standard deviation that the population would have if all its members were treat-
ed with the test treatment (respectively, the placebo).
190 CHAPTER 6
depends on the experimental samples being drawn at random
from the population in question. But which population is that?
The randomization step provides a possible answer, and a pos-
sible explanation for why it is involved in the testing process. For,
by randomizing the subjects across the experimental groups, each
of the groups is certain to be a random selection from the subjects
participating in the trial. This explanation implies that the refer-
ence population comprises just those patients who are involved in
the trial. But this will not do, for the aim of a trial is to determine
not just whether the treatment was effective for those who happen
to be included in the trial but whether it will benefit others who
were left out, as well as unknown sufferers in faraway places, and
people presently healthy or yet unborn who will contract the dis-
ease in the future. But in no trial can random samples be drawn
from hypothetical populations of notional people.
Bourke, Daly, and McGilvray (1985, p. 188) addressed the
question of the reference population slightly differently but they
too saw it resolved through the randomization step. They argued
that the standard statistical tests may be properly applied to clini-
cal trials without "the need for convoluted arguments concerning
random sampling from larger populations", provided the experi-
mental groups were randomized. To illustrate their point they con-
sidered a case where the random allocation of 25 subjects
delivered 12 to one group and 13 to the other. These particular
groups, they affirmed, "can be viewed as one of many possible
allocations resulting from the randomization of 25 individuals
into two groups of 12 and 13, respectively. They noted the exis-
tence of 5,200,300 different possible outcomes of such a random-
ization and proposed this set of possible allocations as the
reference population; the null hypothesis to which the signifi-
cance test is applied, they suggested, should then be regarded as
referring to that hypothetical population.
But this suggestion is not helpful. First, it is premised on the
fiction of a premeditated plan only to accept groups resulting
from the randomization that contain, respectively, 12 and 13 sub-
jects, with the implication that any other combinations that might
have arisen would have been rejected.
4
Secondly, while the refer-
4 The numbers of subjects, 11 1 and 11
2
, in each of the groups must, strictly, be fixed
STATISTICAL INFERENCE IN PRACTICE: CLINICAL TRIALS 191
ence population consists of five million or so pairs of groups, the
sample is just a single pair. But as is agreed all round, you get very
little information about the mean of a large population from a
sample of 1. Finally, as Bourke et al. themselves admitted, they do
not overcome the problem we identified above, for statistical
inference in their scheme "relates only to the individuals entered
into the study" and may not generalize to any broader category of
people. Achieving such a generalization, they argued, "involves
issues relating to the representativeness of the trial group to the
general body of patients affected with the particular disease being
studied". But the path from "representative sample" to "general
body of patients"- two vague notions-cannot be explored via
significance tests and is left uncharted; yet unless that path is
mapped out, the randomized clinical trial can have nothing at all
to say on the central issue that it is meant to address.
Fisher's Argument
When Fisher first advanced the case for randomi zation as a nec-
essary ingredient of trials he did so in somewhat different terms.
Instead of trying to relate the significance test to populations of
sufferers, he thought of it as testing certain innate qualities of a
medical treatment or, in the agricultural context, where Fisher's
main interest lay, innate potentials of a plant variety.
We may illustrate the argument through our potato example,
which slightly simplifies Fisher's (1947, pp. 41--42) own example.
Consider the possibility that, in reality, the varieties A and B have
the same genetic growth characteristics and let thi s be designated
the null hypothesis. Suppose, too, that one plot in each block is
more fertile than the other, and that this apart, no relevant differ-
ence exists between the conditions that the seeds and the resulting
plants experience. If pairs of different-variety seeds are allocated
in advance for a valid I-test, for reasons we discussed earlier (S.d). One way of
combining this and the randomization requirement would be to select pairs of
subjects at random and then allocate the clements of the pairs at random, one to
the test and the other to the control group. When the number of subjects is even,
11 1 and 112 will be equal , and when odd, they will differ by one. This may be what
Bourke, Daly, and McGil vray had in mind.
192 CHAPTER 6
at random, one to each of a pair of plots, there is a guaranteed,
objective probability of exactly a half that any plant of the first
variety will exceed one of the second in yield, even if no intrinsic
difference exists. The objective probability that r out of n pairs
show an excess yield for A can then be computed, a simple test of
significance, such as we discussed in Chapter 5, applied, and a
conclusion on the acceptability or otherwise of the null hypothe-
sis drawn.
All this depends on the objective probabilities that the ran-
domization step is supposed to guarantee. But this guarantee in
turn depends on certain questionable, suppositions, for example,
that none of the innumerable environmental variations that
emerged after the random allocation introduced a corresponding
variation in plant growth. For if the experiment had been exposed
to some hidden growth factor, the null-hypothesi s probabilities
might be quite different from those assumed above, nor could we
know what they are.
Fisher (1947, p. 20) recognized this as a difficulty, but argued
that it could be overcome: "the random choice of the objects to be
treated in different ways would be a complete guarantee of the
validity of the test of significance, if these treatments were the last
in time of the stages in the physical history of the objects which
might affect their experimental reaction" (italics added). And any
variation that occurred after this, carefully located, randomization
step, he then claimed, "causes no practical inconvenience" (1947,
pp. 20-21),
for subsequent causes of differentiation, if under the experimenter's
control ... can either be predetermined before the treatments have
been randomised, or, if this has not been done, can be randomised on
their own account: and other causes of differentiation will be either
(a) consequences of differences already randomised, or (b) natural
consequences of the differences in treatment to be tested, of which on
the null hypothesis there will be none, by definition, or (c) effects
supervening by chance independently from the treatments applied.
In other words, no post-randomization effects could disturb the
probability calculations needed for the significance test, for such
effects would either be the product of differences already random-
ized and hence would be automatically distributed at random, or
STATI STICAL INFERENCE IN PRACTI CE CLINICAL TRI ALS 193
they would be chance factors, independent of the treatment, and
therefore subject to a spontaneous randomization.
But Fisher is here neglecting the possibility of influences
(operating either before or after the random allocation) that are
not independent of the treatments. For example, a certain insect
might benefit plant varieties to an equal degree when these are
growing separately, but be preferentially attracted to one of them
when they are in the same vicinity. Or the different varieties in the
trial might compete unequally for soil nutrients. There are also
possible disturbing influences that could come into play before
the plants were sown. For instance, the different types of seed
might have been handled by different market gardeners, or stored
in slightly different conditions, and so on. Factors such as these
might, unknown to the experimenters, impart an unfair advantage
to one of the varieties, and their nature is such that they cannot be
distributed at random, either spontaneously or through the trial's
deliberate randomi zation. Fi sher (1947, p. 43) was therefore
wrong in his view that randomization "relieves the experimenter
from the anxiety of considering and estimating the magnitude of
the innumerable causes by which the data may be di sturbed."
The natural response to this objection is to say that while one
or more of the innumerabl e variations occurring during an exper-
iment might possibly affect the result, most are very unlikely to do
so. For example, the market gardener's hair colour and the doc-
tor's collar size are conceivable, but most implausible as influ-
ences on the progress of an agricultural or clinical tri al. ft seems
reasonable therefore to ignore factors like these in the experimen-
tal design, and thi s is also the official position:
A substantial part of the skill of the experimenter li es in the choice
of factors to be randomi sed out of the experiment. If he is careful, he
will randomise out [d istribute at random] all the factors which are
suspected to be causally important but which are not actually part of
the experimental structure. But cvery experimenter necessarily neg-
lects some conceivabl y causal factors; if this were not so, the ran-
domisation procedure would be impossibl y complicated. (Kendall
and Stuart 1983, p. (37)
In accordance with this doctrine, when designing a trial to study
the effect of alcohol on reaction times, Kendall and Stuart
194 CHAPTER 6
explicitly omitted the subjects' eye colour from any randomiza-
tion, on the grounds that the influence of this quality was "almost
certainly negligible". They cited no experimental results in sup-
port of their view, presumably because there are none. But even if
there had been a trial to which they could have appealed, certain
conceivable influences on its outcome must also have been set
aside as negligible. Hence, it would be futile to insist that an influ-
ence must only be considered negligible in the light of a properly
randomized trial designed to determine this, for that would open
up the need for an infinite regress of trials. This is why Kendall
and Stuart (p. 137) concluded that the decision whether to "ran-
domize out a factor" or ignore it "is essentially a matter of judge-
ment".5
What Kendall and Stuart demonstrated is that randomization
has to be confined to factors that the experimental designers judge
to be of importance, and that this judgment is necessarily a per-
sonal one, which cannot be based solely on objective considera-
tions. This conclusion is of course completely at odds with the
classical methodology underlying Fisher's argument. So, far from
rescuing Fisher's defence of randomization, Kendall and Stuart
unwittingly knocked another nail in its coffin.
6.d The Eliminative-Induction Defence of
Randomization
This defence, at its strongest, claims that randomization performs
for the unknown prognostic factors the same service as controls
perform for the known ones; that is to say, the procedure guaran-
tees that both the known and the unknown prognostic factors in a
trial will be equally distributed across the experimental groups. rf
, It should be mentioned in passing that Kendall and Stuart's example is not well
chosen. for in their experiment. the different doses of alcohol are allocated ran-
domly to the subjects. and this means that any characteristic to which the sub-
jects are permanently attached. like the colour of their eyes. is, contrary to their
claim. automatically randomized. But their point stands. for there arc other
examples they could have used-the eye colour of those who administer the
alcohol to the subjects is one.
STATISTICAL INFERENCE IN PRACTICE CLINICAL TRIALS 195
such an assurance were available, it would then be a straightfor-
ward matter to infer causal responsibility for any discrepancies
that arise in the course of thc trial between the experimental
groups.
These are some expressi ons of the defence: "Allocating
patients to treatments A and B by randomization produces two
groups of pati ents which are as alike as possible with respect to
all their [prognostic] characteristics, both known and unknown"
(Schwartz et aI., 1980, p. 7; italics added). Or as Giere (1979, p.
296) emphatically put it: randomized groups "are automatically
controlled for ALL other factors, even those no one suspects."
The argument, however, usually takes a more modest form.
For example, Byar et at. (1976, pp. 74-80; our italics) say that
randomi zation "tends to balance treatment groups in . . . prognos-
tic factors, whether or not these variables are known." Tanur et al.
(1989, p. 10; our italics) maintain that it "usually will do a good
job of evening out all the variables- those we didn't recognize in
advance as well as those we did recognize." And Gore (1981, p.
1559; our italics) expresses the idea more precisely by saying that
randomization is "an insurance in the long run against substantial
accidental bias between treatment groups", indicating that "the
long run" covers large tri als with more than 200 subjects. In other
words, in the view of these commentators, although randomiza-
tion does not give a complete assurance that the experimental
groups will be balanced, it makes such an outcome very probable,
and the larger the groups, the greater that probability.
This latter claim is certainly credible, judging by the frequen-
cy with which it is voiced. Nevertheless, we argue that it is mis-
taken, unless significantly modified. And this modified position,
while compatible with Bayesian thought, is inimical to cl assical
inferenti al theory.
Bearing in mind that the argument under examination makes
claims about the unkm)H11 prognostic factors operating in a trial ,
we may consider a number ofpossibilities regarding their number
and nature. The simplest possibility is that there are no such fac-
tors, in which case, the probability that the trial throws up
unmatched groups is clearly zero. The next possibility is that there
is a single unknown factor-call it X-that is carried by some of
the pati ents. The randomized groups will be substantially
196 CHAPTER 6
unmatched on X with a certain definite probability, XII ' say, and this
probability clearly diminishes as the sample size, 11, increases.
There might, of course, be a second factor, Y, also carried by
some of the patients. The chance of one of the groups created in
the randomization step containing substantially more Y-patients
than the other also has a definite value, Y
II
, say. And the probabil-
ity that the groups are unmatched on at least one of the two fac-
tors might then be as much as XII + Y/7 - XIIY
II
, if they are
independent. And because we are dealing with the unknown, it
must be acknowledged that there might be innumerable other fac-
tors; hence the probability of a substantial imbalance on some
prognostic factor might, for all we know, be quite large, as
Lindley (1982, p. 439) has pointed out. Nor are we ever in a posi-
tion to calculate how large, since, self-evidently, we do not know
what the unknown factors arc.
We have so far considered only unknown factors that are, as it
were, attached to patients. What about those that might be inde-
pendent of the patient but connected to the treatment? We are here
thinking of possible, unknown prognostic factors that are acciden-
tally linked to the treatment in this particular trial , so that they
could not be regarded as an aspect of the treatment itself. For
example, suppose the test and control patients were treated in sep-
arate surgeries whosc different environments, through some hid-
den process, either promoted or hindered recovery; or imagine
that the drug, despite the manufacturers' best efforts, included a
contaminant which compromised its effectiveness; or . . . (one
simply needs a rich enough imagination to extend this list indefi-
nitely). For all we know, one or more such factors are active in the
experimental situation; and if that were in fact so, the probability
that the groups are imbalanced would, of course, be one.
So what can we say with assurance about the objective proba-
bility of the randomized experimental groups in a clinical trial dif-
fering substantially on unknown prognostic factors? Merely this:
it lies somewhere in the range zero to one! It follows that the main
defence of randomization in trials-that it guarantees well-
matched groups, or ensures that such groups are highly proba-
ble-is untrue.
Those who advance either of the claims that we have just
shown to be faulty do so in the belief that drawing conclusions
STATISTICAL INFERENCE IN PRACTICE: CLINICAL TRIALS 197
from the results of a clinical trial would bc facilitated by a guar-
antee that the comparison groups are balanced, or very probably
balanced on every prognostic factor. Take thc stronger of thcse
claims, which is in fact rarely maintained and clearly indefensible.
ft has at least the virtue that ifit were true, then the conditions for
an eliminative induction would be met, so that whatever differ-
ences arose bet ween the groups in the clinical trial could be infal-
libly attributed to the trial treatment. On the other hand, the less
obviously faulty and more popular claim cannot exploit elimina-
tive induction. For, the premise that the experimental groups were
prohahly balanced does not imply that differences that arise in the
clinical trial were probably due to the experimental treatment,
unless Bayes's theorem were brought to bear, but that would
require the input of prior probabi I ities and the abandonment of the
classical approach.
We shall, in Chapter 8, see how Bayesi an inference operates in
clinical research, in the course of which we shall show that ran-
domization, while it may sometimes be valuable, is not absolute-
ly necessarily.
6.e Sequential Clinical Trials
There are two aspects of the controlled, randomized trials we have
been considering that have caused concern, even amongst classi-
cal statisticians. The first is this. Suppose that the test therapy and
the comparison therapy (which may simply be a placebo) are not
equally effective and that thi s becomes apparent at the end of the
trial, through a differential response rate in the experimental
groups. This would mean that some of the patients had been treat-
ed with an inferior therapy throughout the trial. Clearly the fewer
so treated the better. So the question arises whether a particular
trial is suited to extracting the required information in the most
efficient way.
The second concern is prompted by the following considera-
tion: as we showed in an earlier chapter, any significance test that
is applied to the results of a trial requires the experimenter to
establish a stopping rule in advance. The stopping rule that is nor-
mally used, or assumed to have been used, fixes the number of
198 CHAPTER 6
patients who will enter the trial. Then, if for some reason, the trial
were discontinued before the predetermined number was reached,
the results obtained at that stage would, as Jennison and Turnbull
(1990, p. 305) note, have to be reckoned quite uninformative.
Intuitively, however, this is wrong; in many circumstances, quite a
lot of information might seem to be contained in the interim
results of a trial. Suppose, to take an extreme example, that a sam-
ple of 100 had been proposed and that the first 15 patients in the
test group completely recover, while a similar number in the con-
trol group promptly experience a severe relapse. Even though the
trial as originally envisaged is incomplete, this evidence would
incline most people strongly in favour of the test treatment and
would make them reluctant to continue the trial, fearing that a fur-
ther 35 or so patients would be denied the apparently superior
treatment.
It might be imagined that the problem could be dealt with by
performing a new significance test after each new experimental
reading and then halting the trial as soon as a 'significant' result
is obtained. But, in fact, this sort of continuous assessment is not
possible, for each significance test would be based on the assump-
tion that the number of patients treated at that stage of the trial
was the number envisaged in the stopping rule for the entire trial,
which of course is not the case. A kind of clinical trial in which
sequences of such quasi-significance tests may be validly used
has, however, been developed and we examine these now.
Sequential clinical trials were designed to minimize the num-
ber of patients that need to be treated and to ensure that "random
allocation should cease if it becomes reasonably clear that one
treatment has a better therapeutic response than another"
(Armitage 1975, p. 27). Such trials enable tests of significance to
be applied even when the sample size is variable; they allow
results to be monitored as they emerge; and in many cases, they
achieve a conclusion with fewer data than a fixed-sample design
would allow.
The following is a simple sequential trial, which we have
taken from Armitage 1975. Appropriately matched pairs of
patients arc treated in sequence either with substance A or sub-
stance B (one of which might be a placebo). The treatments are
randomized, in the sense that the element of any pair that receives
STATISTICAL INFERENCE IN PRACTICE CLINICAL TRIALS 199
A, respectively B, is determined at random, We shall assume that
A and B are indistinguishable to the patients and that the medical
personnel administering them are also in the dark as to how each
patient was treated (the trial is therefore both 'blind' and 'double-
blind'). For each pair, a record is made of whether, relative to
some agreed criterion, the A - or the B-treated patient did better.
The results are assumed to come in sequentially and to be moni-
tored continuously.
Results are recorded for each pair as either A doing better than
B, or B doing better than A; we label these results simply A and
B, respectively. In the diagrams below, the difference between the
number of As and the number of Bs in a sample is plotted on the
vertical axis; the horizontal axis represents the number of com-
parisons made. The zigzag line, or 'sample path', starting from
the origin, represents a possible outcome of the trial, correspon-
ding to the 14 comparisons, ABAAAABAAAAAAA.
(f)
OJ
20
10
U Number of
11


U
x
l.U
-10
-20
40 50
SEQUENTIAL PLAN a
The wedge-shaped system of lines in sequential plan a was
designed to have the following characteristics: if the drugs are
equally effective (the null hypothesis), then the probability of the
sample path first crossing one of the outer lines is 0.05. If the
200 CHAPTER 6
20
10
(J)
~ Number of
a5
! 1 0 ~ - - __ +--___ -+_-<E-__ +-___ -+p_r_ef_e_re_nc_e_s-l
~ C Q
()
x
UJ
-10
-20
SEQUENTIAL PLAN b
sample path does cross one of these lines, sampling is stopped and
the null hypothesis rejected. Tfthe sample path first crosses one of
the inner lines, the sampling is stopped too, the result declared
non-significant, and the null hypothesis accepted. The inner lines
are constructed, in this example, so that the test has a power of
0.95 against a particular alternative hypothesis, namely that the
true probability of an A-treated patient doing better than a B-treat-
ed patient is 0.8.
The sequential test based on plan a is, in Armitage's terminol-
ogy, 'open', in the sense that the trial might never end, for the
sample path could be confined indefinitely within one of the two
channels formed by the two pairs of parallel lines. Plan b repre-
sents a 'closed' test of the same null and alternative hypotheses.
Tn the latter test, if the sample path meets one of the outer lines
first, the null hypothesis is rejected, and if one of the inner lines,
the hypothesis is accepted. This closed test has a similar signifi-
cance level and power to the earlier, open one, but differs from it
in that at most 40 comparisons are required for a decision to be
reached.
The above sequential plans were calculated by Armitage on
the assumption that as each new datum is registered, a new signif-
STATISTICAL INFERENCE IN PRACTI CE CLI NICAL TRIALS 201
icance test, with an appropriate signifi cance level, is carri ed out.
(As we pointed out, these are quasi-significance tests, since they
pres ume a fi xed sampl e size when, in fact, the sample is growing
at every stage. ) The si gnificance level of the individual tests is set
at a value that makes the overall probability of rej ecting a true null
hypothesis 5 percent. This is just one way of determining sequen-
tial tests, part icularly favoured by Arm i tage (1975), but there are
many other possible methods leading to di fferent sequential plans.
The sequent ial tests might also vary in the frequency with which
data are monitored; the results might be followed continuously as
they accumulate or examined aft er every new batch of, say, 5 or
10 or 50 patients have been treated (these are ' group sequential
trials'). Each such interim-analysis policy corresponds to a differ-
ent sequenti al test.
The exi stence of a multiplicity of tests for analysing trial
results carri es a strange, though by now famili ar implication:
whether a particular result is 'significant ' or not, whether a
hypothesis should be rejected or not, whether or not you would be
well advised to take the medicine in the expectation of a cure,
depend on how frequentl y the experimenter looked at the data as
they accumul ated and which sequential plan he or she foll owed.
Thus a given outcome from treating a particular number of
pati ents might be significant if the experimenter had monitored
the results continuously, but not signifi cant if he had waited to
anal yse the results unt il they were all in, and the conclusion might
be different again if another sequential plan had been used. This
is surely unacceptabl e, and it is no wonder that sequenti al trials
have been condemned as "a hoax" (Anscombe 1963, p. 381). As
Meier (1975, p.525) has put it, "it seems hard indeed to accept the
noti on that J should be influenced in my judgement by how fre-
quently he peeked at the data while he was collecting it."
Neverthel ess, the astonishing idea that the frequency with
whi ch the data have been peeked at provides information on the
effectiveness of a medical treatment and should be taken account
of when evaluating such a treatment is thoroughly entrenched.
Thus the Food and Drug Administrati on, the drugs licensing
authority in the United States, includes in its Guideline (1 988, p.
64) the requi rement that "all interim anal yses, formal or informal,
by any study participant, sponsor staff member, or data monitor-
202 CHAPTER 6
ing group should be described in full .... The need for statistical
adjustment because of such analyses should be addressed.
Minutes of meetings of the data monitoring group may be useful
(and may be requested by the review division)". But the intentions
and calculations made by the various study participants during the
course of a trial are nothing more than disturbances in those peo-
ple's brains. Such brain disturbances seem to us to carry as much
information about the causal powers of the drug in question as the
goings-on in any other sections of their anatomies: none.
6.1 Practical and Ethical Considerations
The principal point at issue in this chapter has been Fisher's the-
sis, so widely adopted as an article of faith, that clinical and agri-
cultural trials are worthless if they are not randomized. But as we
have argued, the alleged absolute need for such trials to be ran-
domized has not been established. In Chapter 8, we shall argue
that a satisfactory approach to the design of trials and the analy-
sis of their results is furnished by Bayes's theorem, which does not
treat randomization as an absolute necessity.
It would not, of course, be correct to infer that randomization
is necessarily harmful, nor that it is never useful-and we would
not wish to draw such a conclusion-but removing the absolute
requirement for randomization is a significant step which lifts
some severe and, in our view, undesirable limitations on accept-
able trials. For example, the requirement to randomize excludes as
illegitimate trials using so-called historical controls. Historical
controls suggest themselves when, for example, one wishes to
find out whether a new therapy raises the chance of recovery from
a particular disease compared with a well-established treatment.
In such cases, since many patients would already have been
observed under the old regime, it seems unnecessary and extrav-
agant to submit a new batch to that same treatment. The control
group could be formed from past records and the new treatment
applied to a fresh set of patients who have been carefully matched
with those in the artificially constructed control group (or histor-
ical control). But the theory of randomization prohibits this kind
of experiment, since patients are not assigned with equal proba-
STATISTICAL INFERENCE IN PRACTICE: CLINICAL TRIALS 203
bilities to the two groups; indeed, subjects finding themselves in
either of the groups would have had no chance at all of being cho-
sen for the other.
There is also sometimes an unattractive ethical aspect to ran-
domization, particularly in medical research. A new treatment,
which is deemed worth the trouble and expense of an investiga-
tion, has often recommended itself in extensive pilot studies and
in informal observations as having a reasonable chance of
improving the condition of patients and of performing better than
established treatments. But if there were evidence that a patient
would suffer less with the new therapy than with the old, it would
surely be unethical to expose randomly selected sufferers to the
established and apparently or probably inferior treatment. Yet this
is just what the theory of randomization insists upon. No such eth-
ical problem arises when patients receiving the new treatment are
compared with a matched set of patients who have already been
treated under the old regime.
6.9 Conclusion
We have reviewed the main features of classically inspired
designs for clinical and agricultural trials, particularly the alleged
requirement for treatments to be randomized over the experimen-
tal units. Our conclusion is that neither of the two standard argu-
ments for the randomization principle is effective and that the
principle is indefensible as an absolute precondition on trials,
even from the classical viewpoint. In Chapter 8, we shall argue in
detail that a Bayesian approach gives a more satisfactory treat-
ment of the problem of nuisance variables and furnishes intuitive-
ly correct principles of experimental design.
CHAPTER 7
Regression Analysis
We are often interested in how some variable quantity depends on
or is related to other variable quantities. Such relationships are
often guessed at from specific values taken by the variables in
experiments. As we pointed out earlier (in 4i), infinitely many dif-
ferent, and conflicting relationships are compatible with any set of
data. And yet, despite the immense multiplicity of candidate the-
ories or relationships, scientists are rarely left as bewildered as
might be imagined; in fact, they often become quite certain about
what the true relationships are and feel able to predict hitherto
unobserved values of the variables in question with considerable
confidence. How they do this and with what justification is the
topic of this chapter (which largely follows Urbach 1992).
7.a I Simple Linear Regression
How to infer a general relationship between variable quantities
from specific observed values is a problem considered in practi-
cally every textbook on statistical inference, under the heading
Regression. Usually, the problem is introduced in a restricted
form, in which just two variables are involved; the methods devel-
oped are then extended to deal with many-variable relationships.
When it comes to examining underlying principles, which is our
aim, the least complicated cases are the best and most revealing;
so it is upon these that we shall concentrate.
Statisticians treat a wider problem than that alluded to earlier,
by allowing for cases where the variables are related, but because
of a random 'error' term, not in a directly functional way. With
just two variables, x and y, such a relationship, or regression, may
be represented as y = f(x) + E, where E is a random variable,
206 C HAPTER 7
called the 'error' term. The simplest and most studied regres-
sions-known as simple linear regressions-are a special case, in
which y = ex + f3x + E, ex and f3 being unspecified constants. In
addition, the errors in a simple linear regression are taken to be
uncorrelated and to have a zero mean and an unspecified vari-
ance, (J 2, whosc valuc is constant over all .T. We shall call regres-
sions mceting these conditions linear, dropping the qualification
'simple', for brevity.
Many different systems have becn cxplored as reasonable can-
didates to satisfy the linear regression hypothesis. The following
are typical examples from the textbook literature: the relationship
between wheat yield, y, and quantity of fertilizer applied, x
(Wonnacott and Wonnacott 1980); the measured boiling point of
water, y, and barometric pressure, x (Wcisberg 1980); and the
level of people's income, y, and the extent of their cducation, x
(Lewis-Beck 1980).
The lincar regression hypothesis is depicted below (Mood and
Graybill 1963, p. 330). The vertical axis represents the probability
densities ofy for given values of x. The bell-shaped eurves in the
y. pry I x)-plane illustrate the probability density distributions ofy
for two particular values of x; the diagonal line in the
x,y-plane is the plot of the mcan ofy against x; and the variance of
the density distribution curves, which is the same as the error vari-
ance, (J2, is constant-a condition known as 'homoscedasticity' .
The linear regression hypothesis leaves open the values of the
three parameters ex, /3, and (J, which need to be estimated from
pry Ix)
r - - - - - - - - - 7 - - - - - 7 ~ - - ~ ~ - - - - ~ - - - - - x
E(y Ix) = ()( + (3x
y
REGRESSION ANALYSIS 207
data, such data normally taking the form of particul ar x, y readings.
These readings may be obtained in one of two ways. In the first,
particular values of x (such as barometric pressures or concentra-
tions of fertilizer) are pre-selected and then the resulting ys (for
instance the boiling point of water at each of the pressure levels or
wheat yields at each fe rtilizer strength) arc read off. Alternatively,
the xs may be selected at random (they could, for example, be the
heights of random members of a population) and the correspon-
ding ys (for exampl e each chosen person's weight) then recorded
as before. In the exampl es discussed here, the data willmostiy be
of the former kind, with the xs fixed in advance; none of the criti-
cisms we shall otfer will be affected by this restri cti on.
We are considering the special case where the underlying
regression has been assumed to take the simple linear form. For
the Bayesian, this means that other possible forms of the regres-
sion all have probability zero, or their probabiliti es are sufficient-
ly close to zero to make no difference. In such special (usually
artificial ) cases, a Bayesian would continue the analysi s by
describing a prior, subj ective probability distributi on over the
range of possible values of the regression parameters ex, /3, and
a, and would then conditionalise on the evidence to obtain a cor-
responding posterior di stribution (for a detailed exposition, sec.
for instance, Broemeling 1985). But classical stati sticians regard
such distributions as irrelevant to science, because of their subjec-
tive component. The classical approach tries to put its objectivist
ideal into effect by seeking the linear regression equati on possess-
ing what is called the ' best fit ' to the data, an aim also expressed
as a search for the ' best estimates ' of the unknown linear regres-
sion parameters. There are also classical procedures for examin-
ing the assumption of I inear regression, which we shall review in
due course.
7.b The Method of Least Squares
The method of least squares is the standard classical way of estimat-
ing the constants in a linear regression equation and "hi storically has
been accepted almost uni versally as the best estimati on technique
for linear regression models" (Gunst and Mason 1980, p. 66).
208 CHAPTER 7
The method is this. Suppose there are n x,y readings, as
depicted in the graph below. The vertical distance of the ith point
from any straight line is labelled e
i
and is termed the 'error' or
'residual' of the point relative to that line. The straight line for
which Le
2
is minimal is called the least squares line. lfthe least
I A A
squares line is y = ex+ f3x, then ex and f3 are said to be the least
squares estimates of the corresponding linear regression coeffi-
cients, (X and {3, and the line is often said to provide the ' best fit'
to the data. The idea is illustrated below.
L--------------------------------------------X
X,
The least squares line, that is, the line for which 'Ie/ is minimum.
The least squares estimates, ex and ~ are given as follows,
where x and yare, respectively, the mean values of the X i and Y
i
in the sample (proving these formulas is straightforward and we
leave this to the reader):
A L(y-V)(x.-X)
3 - I' I
{- L(X_X)2
I
ex = 5' -Fix.
These widely used formulas, it should be noted, make no assump-
tions about the distribution of the errors. Note, too, that the least
squares principle does not apply to (J2, the error variance, which
needs to be estimated in a different way (sec below).
REGRESSION ANALYSIS 209
7.c Why Least Squares?
It is said almost universally by classical statisticians that if the
regression of y on x is linear. then least squares provides the best
estimates of the regression parameters. The term 'best" in this
context intimates some epistemic significance and suggests that
the least squares method is for good reason preferable to the infi-
nitely many other conceivable methods of estimation. Many sta-
tistics textbooks adopt the least squares method uncritically. but
where it is defended. the argument takes three forms. First. the
method is said to be intuitively correct. and secondly and thirdly.
to be justified by the Gauss-Markov theorem and by the
Maximum Likelihood Principle. We shall consider these lines of
defence in turn.
Intuition as a Justification
The least squares method is often recommended for its "intuitive
plausibility" (Belsley et al. 1980. p. I). Yet the intuitions typi-
cally cited do not point unequivocally to least squares as the
right method for estimating the parameters of a I inear regression
equation; at best. they are able to exclude some possible alterna-
tive methods. For example. the idea that one should minimize the
absolute value of the sum of the errors. 1 Lei I. is often dismissed
because. in certain circumstances. it leads to intuitively unsatis-
factory regression lines. In the case illustrated below, the method
would result in two, quite different lines for the same set of data.
Although both lines minimize 1 Lei I, Wonnacott and Wonnacott
(p. 16) judge one of them, namely h. to be "intuitively ... a very
bad one".
The trouble here is that the criterion of minimum aggregate
error can be met by balancing large positive errors against large
negative ones, whereas for an intuitively good fit of data to a line.
all the errors should be as small as possible. This difficulty would
be partly overcome by another suggested method. namely, that of
minimizing the sum of the absolute deviations, L 1 e
i
I. But this too
is often regarded as unsatisfactory. For example, Wonnacott and
Wonnacott rejected the method unequivocally, and signalled their
210 CHAPTER 7
y y
L-------------------x L-------------------x
(0) (b)
Two lines both satisfying the condition that I 'Le,2 I is minimized.
emphatic disapproval of thc mcthod by giving it thc acronym
MAD. But this highlights a weakness of arguments that are based
on intuition, namely, that people often disagree over the intuitions
themselves. Thus, unlike the Wonnacotts, Brook and Arnold
(1985, p. 9) found the MAD method perfectly sane; indeed, they
regarded it as "a sensible approach which works well", their sole
reservation being the practical one that "the actual mathematics is
difficult when the distributions of estimates are sought". (As we
shall see in the next section, statisticians require such distribu-
tions, in order to derive confidence intervals for predictions based
on an estimatcd regression equation.)
On the othcr hand, Brook and Arnold saw in the least squares
method an intuitively unsatisfactory featurc. This is that particu-
larly large deviations, since they must be squared, "may have an
unduly large influence on the fitted curve" ( 1985, p. 9). They refer
with approval to a proposal for mitigating this supposed draw-
back, according to which the estimation should proceed by mini-
mizing the sum of a lower power of the crrors than their squares,
that is, by minimizing L I e
i
I", with p lying somewhere bctween I
and 2. The authors propose p = 1.5 as a "reasonable compromise",
but they do not explain why the mid-point value should accord so
well with reason; to us it seems merely arbitrary, despite its air of
Solomonic wisdom. But against this proposal, they point out the
same practical difficulty that they observed with the MAD
method of estimation, namely that "it is difficult to determine the
exact distributions of the resulting estimates", and so--for this
purely pragmatic reason--they revert to p = 2.
REGRESSION ANALYSIS 211
The deviations whose squares are minimized in the least
squares method appear in a graph as the vertical distances of
points from a line. It might be thought that a close fit between line
and points could be equally well secured by applying the least
squares principle to the perpendicular distances of the points from
the line, or to the horizontal distances, or, indeed, to distances
measured along any other angle. Why should the vertical devia-
tions be privileged above all others?
Kendall and Stuart defended the usual least squares procedure,
based on vertical distances, on the vague and inadequate grounds
that since "we are considering the dependence of y on x, it seems
natural to minimize the sum of squared deviations in the y-direc-
tion" ( 1979, p. 278; ital ics added). But this by itself is insufficient
reason, for many procedures that we find natural are wrong;
indeed, Kendall and Stuart's famous textbook offers numerous
'corrections' for what they view as misguided intuitions.
Brook and Arnold advanced an apparently more substantial
reason for concentrating on vertical deviations from the regres-
sion line; they claimed that "when our major concern is predict-
ing y from x, the vertical distances are more relevant because they
represent the predicted error" (1985, p. 10; italics added). For
most statisticians the main interest of regression equations does
indeed lie in their ability to predict (see Section 7.e below). But
predictions concern previously unexamined values of the vari-
ables; the predicting equation, on the other hand, is built up from
values that have already been examined. It is misleading therefore
to say that "vertical di stances [of data points] are more relevant
because they represent the predicted error"; in fact, they do not
represent the errors in predictions of previously unexamined
points.
The following seems a more promising argument for minimiz-
ing the squares of the vertical distances. Suppose the units in
which y was measured were changed, by applying a linear trans-
formation, y' = my + n, m and 11 being constants. The transformed
regression equation would then bey'= a' + f):'( + E, where a' =
ma + nand (J' = mf3. Ehrenberg (1975) and others (e.g. , Bland,
1987, p.I92) have pointed out that if the best estimate of a is B,
the best estimate of a ' should be ma + n; correspondingly for (3'.
If 'best' is defined in terms of a least squares estimat ion method
21 2 CHAPTER 7
that is based on vertical distances, then thi s expectation is satis-
fi ed, a fact that advocates ri ghtly regard as a merit of that
approach. However, the same argument could be made for a least
squares method based on hori zontal distances, so it is inconclu-
sive. More seriously, the argument is self-defeating, for unl ess lin-
ear transformati ons could be shown to be speci al in some
appropriate way, non-linear transformations of)' should al so lead
to correspondingly transformed least squares curves; but this is
not in general the case. Indeed, it is practically a universal rule in
classical estimati on- though maximum likelihood estimation is
an exception (see below)-that if (1 is its optimal estimate of ct,
then f(a ) is suboptimal rclati ve tof (a): in other words, cl assical
estimators are generally not 'invariant'.
Weisberg ( 1980, p. 214) illustrated this rul e with some data on
the average brain and body weights of different animal species.
The least squares line for the data (expressed logarithmicall y) was
calculated to be log(br) = 0.93 + O.7510g(bo) . Weisberg then uscd
thi s equation to estimate (or predict) the log(brain weight) of a so
far unobserved species whose mean body weight is 10 kg, obtain-
ing the valuc 1. 68 kg (0.93 + O. 7510g(1 0) = 1.68). We isberg
regarded thi s as a satisfactory prediction, for one rcason, because
it is unbiased and so satisfics a classical criterion for 'good' esti-
mat ion (see Chapter 5). Now, from the predi ction, it seems natu-
ral to infer that 47.7 kg (47. 7 being the anti logarithm of 1.68)
would be a satisfactory predicti on for the average brain weight of
the species. Yet Weisberg described that conclusion as "na'ive",
beca use estimates of brain wei ghts derived in this way are "bi ased
and will be, on the average, too small".
But it seems commonsensi cal and not at all nai ve to ca ll for
estimates to be invariant under functi onal transformation ; indeed,
this view is often endorsed in classical texts (for exampl e by
Mood and Graybill, p. 185) when maximum likelihood estimation
is under discussi on and invari ance is offered as one of the proper-
ties of that approach which parti cularly commend it.
So much for intuiti ve arguments for least squares. They are
at best inconclusive, and most classical statisticians beli eve a
more telling case can be made on the basi s of certain obj ective
stati stical properti es of least squares estimates, as we shall
describe.
REGRESSION ANALYSIS 213
The Gauss-Markov Justification
The objective properties we are referring to are those of unbi-
asedness, relative efficiency, and linearity, each of which is pos-
sessed by the standard least squares estimators of the linear
regression parameters. The unbiasedness criterion is well estab-
lished in all areas of classical estimation, though, as we have
already argued, with questionable credentials. The criterion of
relative efficiency as a way of comparing different estimators by
their variances is also standard; least squares estimates are the
most efficient amongst a certain class of estimators, a point we
shall return to. Linearity, by contrast, is a novel criterion that was
specially introduced into the regression context. It is simply
explained: the estimates, a. and /3, of the two linear regression
parameters, are said to be linear when they are weighted linear
combinations of the v, that is, when a = La V and /3 = Lb v. A
,/ {. ~ I V I
comparison with the formulas given above confirms that least
squares estimates arc linear, with coefficients
h =
X,-X
I
c
and
(
\' 2 1
"'" . I -
~ -xx
n I
...... , where c = I(, -:x )2.
c . I
a.
I
We now come to the Gauss-Markov theorem, which is the
most frequently cited "theoretical justification" (Wonnacott and
Wonnacott, p. 17) for the method of least squares. The theorem
was first proved by Gauss in 182 I, Markov's name becoming
attached to it because, in 1912, he published a version of the proof
which Neyman believed to be an original theorem (Seal 1967, p.
6). The theorem states that within the class ollinem; unbiased
estimators olf3 and ex. least squares estimators have the smallest
variance.
In Chapter 5 we explored the arguments advanced for unbi-
asedness and relative efficiency as criteria for estimators. Even
those who are unconvinced by our objections against them would
still have to satisfy themselves that linearity was a reasonable
214 CHAPTER 7
requirement, before calling on the Gauss-Markov theorem in sup-
port of least squares estimation. Often, thi s need seems not to be
perceived, for instance by Wei sberg ( 1980, p. 14), who said, on the
basis of the Gauss-Markov theorem, that "i f one believes the
assumptions [of linear regression], and is interested in using lin-
ear unbiased estimates, the least squares estimates are the ones to
use"; he did not attend to the question of whether the " interest" in
linear estimators which he assumed to exist amongst his readers
is reasonable. Seber (1977, p. 49), on the other hand, did describe
linear estimators as "reasonabl e", but gave no supporting argu-
ment.
The only justi f ication we have seen for requiring estimators to
be linear turns on their supposed "simplicity". "The property of
linearity is advantageous", say Daniel and Wood (1980, p. 7), "in
its simplicity". Wonnacott and Wonnacott restricted themselves to
linear estimates "because they are easy to compute and analyse"
(p. 31). But simplicity and convenience are not epistemological
categories: the sin/plest and easiest road might well be going in
the 'vvrong direction and lead you astray.
Mood and Graybill did not appeal to simplicity but referred
mysteriousl y to "some good reasons why we would want to
restrict oursel ves to linear functi ons of the observations Y
i
as esti-
mators" ( 1963, p. 349); but they compounded the obscurity of
their position by adding that "there are many times we would not",
again without explanation, though they probably had in mind
cases where some data points show very large deviations from the
presumed regress ion line or in some other way appear unusual
(we shall deal separately with this concern in 7.fbelow).
The limp and inadequate remarks we have reported seem to
constitute the onl y defences whi ch the linearity criterion has
received, and thi s surel y suggests that it lacks any epistemic basis.
Mood and Graybill confirmed this impressi on when they admit-
ted that since "it will not be possible to examine the 'goodness ' of
the [l east squares] estimator g, relative to all functions . . . we
shall have to limit oursel ves to a subset of all function s" (p.
349; italics added). As an example of such a subset they cited
linear function s of the v and then showed how, according to the
I
Gauss-Markov theorem, the method of least squares excel s
within that subset, on account of its minimum variance. But
REGRESSION ANALYSIS 215
how can we tell whether least squares estimation is not just the
best of a bad lot?
The Gauss-Markov argument for using a least squares estima-
tor is that within the set of linear unbiased estimators, it has min-
imal variance. The argument depends, of course, on minimum
variance being a desirable feature. But this presents a further dif-
ficulty. For, whatever that minimum variance is in any particular
case, there would normally be alternative, biased and/or non-lin-
ear estimators of the parameters, whose variance is smaller. And
to establish least squares as superior to such alternative methods,
you would need to show that the benefit of the smaller variance
offered by the alternatives was outweighed by the supposed disad-
vantage of their bias and/or non-linearity. But this has not been
shown, and we would judge the prospects for any such demonstra-
tion to be dim.
We conclude that thc Gauss-Markov theorem provides no
basis for the least-squares method of estimation. Let us move to
the third way that the method is standardly defended.
The Maximum Likelihood Justification
The maximum likelihood estimate of a parameter 0, relative to
data d, is the value of 0 which maximizes the probability P(d I OJ.
Many classical statisticians regard maximum likelihood estimates
as self-evidently worthy, and rarely offer arguments in their
defence. Hays is an exception. He argued that the maximum like-
lihood principle reflects a "general point of view about infer-
ence", namely, that "true population situations should be those
making our empirical results likely; if a theoretic situation makes
our obtained data have very low prior likelihood of occurrence,
then doubt is cast on the truth of the theoretical situation" ( 1969,
p. 214; italics removed). And Mood and Graybill pointed to cer-
tain "optimum properties" of maximum likelihood estimates,
principally that they are i n v ~ r i n t under functional transforma-
tion, so that, for example, if fJ is the maximum likelihood estimate
of 0, thenj(8) is the maximum likelihood estimate off(fJ), provid-
ed the function has a single-valued inverse. These do not consti-
tute adequate defences. Indeed, there can be no adequate defence,
because the maximum likelihood method of estimation cannot be
216 CHAPTER 7
right in general. This is clear from the fact, to which we alluded
before, that any experimental data will be endowed with the max-
imum probability possibl e, namely, probability I, by infinitely
many theories, most of which will seem crazy or blatantly fal se.
In order to apply the maximum likelihood method to the task
of estimating (1 and f3 in a linear regression equation, the form of
the error distribution must be known, for only then can the prob-
ability of the data relative to speci f ic forms of the regression be
calculated and compared. Least squares, by contrast, can be
applied without that knowledge. It turns out that when the regres-
sion errors are normally di stributed, the maximum likelihood esti-
mates of (1 and f3 are precisely those arrived at by least squares, a
fact that is often claimed to provide "another important theoreti-
cal justification of the least squares method" (Wonnacott and
Wonnacott, p. 54).
Those who regard thi s as a justification cl early assume that the
maximum likelihood met hod is correct, and that it furnishes rea-
sonable estimates; they then argue that since least squares and
maximum likelihood coincide in the specific case where thc
errors are distributed normally, the least squares principle gi ves
reasonable estimates in general. Although we have done so, it is
unneccssary to take a view on the merits of maximum likelihood
estimation to appreciate that this argumcnt is a non sequitur; you
might just as well argue that because you get the right answer with
o and I , x
3
is a good way of estimating x
2
.
Summary
The cl assical arguments in favour of least squares estimation of
linear regression parameters are untenable. That based on intu-
ition is inconclusive, vague, and lacking in epistemol ogical force.
The Gauss-Markov justi f ication rests on thc linearity criteri on,
which itself is unsupported. And the maximum likelihood defence
is based on a fall acy.
The two theoretical defences of least squares are standardly
cited together as if they were mutually reinforcing or compl emen-
tary. Tn fact, the reverse is the casco For while the Gauss-Markov
justification presupposes that estimators must be unbiased, the
REGRESSION ANALYSIS 217
maximum likelihood principle does not. it sometimes
del ivers biased estimates, as for example, in the casc at hand: the
I "
maximum likelihood estimate of 0
2
is- L (v. - ex - f3xY But
n I I
when the regressi on of y on x is linear, this estimate is biased and
11
needs to be increased by the factor--
2
to unbi as it. (It is thi s
/1-
modified estimate, labelled [} that is always empl oyed in classical
expositions. )
The two main ' theoretical' defences of least squares estima-
tion are therefore separately defective and mutually destructive.
But all this does not mean that the least- squares method is
entirely wrong. Its great plausibility does have an explanation, a
Bayesian explanation. For it can be shown that when you restrict
the set of possibl e regressions to the I inear, and assume a normal
di stribution of errors and a uniform prior di stributi on of probabil-
ities over parameter values, the least squares and the Bayes solu-
tions coincide, in the sense that the most probable value of the
regression parameters and their least squares estimates are identi-
cal. When the assumpti on of a uniform distribution is the
same result foll ows, though it is reached asymptotically, as the
size of the sampl e increases (Lindley and EI-Sayyad 1968).
7.d Prediction
Prediction Intervals
Fitting a regressi on curve to data is often said to have as its main
practical purpose the prediction of so far unobserved points. We
will consider predicti ons of v values from pre-selected values of
x, which is a predicti on problem widely considered in statistics
texts. Such predictions would be straightforward if one knew the
true regression equat ion, as well as the form of the error distribu-
tion. For then the di stribution of Yo' the .v corresponding to some
particular x O' would also be known, thus enabling one to calculate
the probability with which )'0 would fall within any given range.
Such a range is sometimes called a prediction interval.
218 CHAPTER 7
The problem is that the regression parameters are usually
unknown and have to be estimated from data. To be sure, if those
estimates were qualified by probabilities, as they would be after a
Bayesian analysis, you could still calculate the probability relative
to the data that Yo lay in any specified range. But this option is
closed to classical statistics, which procceds from a denial that
theories (parameter values, in the present case) do have probabil-
ities, except in special circumstances that do not prevail in the
regression problems we are considering.
Prediction by Confidence Intervals
The classical way round this difficulty is to make predictions
using confidence intervals, by a method similar to the one already
discussed (in Chapter 5). First, an experiment is imagined in
which a fixed set ofxs, XI ' ... , x
m
' is chosen. A prearranged num-
ber, n (:2! 111), of x, v readings is then made. A I inear regression of v
on x is assumed ~ n d a and fi are the resulting least squares esti-
mates of the corresponding regression parameters. A new x value,
x
o
' is now considered, with a view to predicting the corresponding
Yo. On the linear regression assumption, Yo is a random variable
given by v = a + /Jx + E, with constant variance, a
2
. The follow-
-" 0 0
ing analysis requires the error terms to be normally distributed. A
new random variable, 1I, with variance 0 2, is now defined as 11 =
II
\' - ex - /3x . The random variable, f, is then considered, wherc
"' 0 0
all _ ~ 1 - 2
f = "
a-a 11
II
(J
The ratio - is independent of a, being determined just by XI' . .. ,
a
II
\", x()' and 11, so f can always be computed from the data. Because
f has a known distribution (the f-distribution with 11 - 2 degrees of
freedom), standard tables can be consulted to find specific values,
fl and f
2
, say, which enclose, say, 95 percent of the area under the
distribution curve. This means that with probability 0.95, fl :s; f:s; f
2
,
from which it follows (Mood and Graybill 1963, pp. 336-37) that
P(o. + nx - Atl :s; y . :s; a. + f3x + At
2
) = 0.95,
,.J () 0 ()
REGRESSION ANALYSIS 219
where A is a complicated expression involving n, x, x (the mean
of x l' ... ,XIII' xo>, and o. The terms lX, p, A, and Yo all random
variables that can assume different values depending on the result
of the experiment described earlier. If Ct.', /'J', and A' are the val-
ues taken by the corresponding variables in a particular trial, the
interval I lX' + P'xo - A' t
J
, a' + /3' Xo + A't21 is a 95 percent confi-
dence interval for Yo' As we noted in our earlier discussion, other
confidence intervals, associated with other probabilities, may bc
described too.
On the classical view, a confidence interval supplies a good
and objective prediction of Yo' independent of subjective prior
probabilities, with the confidence coefficient (95 percent in this
case) measuring how good it is. Support for that view is some-
times seen (for instance by Mood and Graybill 1963, p. 337) in
the fact that the width of any confidence interval for a prediction
increases with Xo - x. So according to the interpretation we are
considering, if you wished a constant degree of confidence for all
predictions, you would have to accept wider, that is, less precise
or less accurate intervals, the further you were from x. This, it is
suggested, explains the "intuition" that "we can estimate most
accurately ncar the 'centre' of the observed val ues of x" (Kendall
and Stuart 1979, p. 365). No doubt this intuition is widely shared
and is reasonable, provided, of course, that we have presupposed
a linear regression, and the explanation given is a point in favour
of the confidence interval approach. However, it is insufficient to
rescue that approach from the radical criticisms already made. As
we explained earlier, the two standard interpretations of confi-
dence categorical-assertion interpretation and the
subjective-confidence interpretation-are both incorrect. Hence,
confidence intervals do not constitute estimates; for the same rea-
sons, they cannot properly function as predictions. That means
that the declared principal goal of regression analysis-- predic-
tion--cannot be achieved by classical means.
Making a Further Prediction
Suppose, having 'predicted' y for a given X , its true value is
disclosed, thus augmenting data by an additional point, and
220 CHAPTER 7
suppose you now wished to make a further predicti on of Yo', cor-
responding to xo'. It is natural to base the second prediction on the
most up-to-date estimates of the linear regression coefficients,
which should therefore be recalculated using the earlier data plus
the newly acquired data point.
Mood and Graybill in fact recommended such a procedure, but
their recommendati on did not ari se, as it would for a Bayesian,
from a concern that predi ctions should be based on all the relevant
evidence. Instead, they said that if the estimated regression equa-
tion were not regul arl y updated in the light of new evidence, and
if it were used repeatedl y to make confidence-interval predic-
tions, then the basis of the confidence intervals would be under-
mined. For those conf idence intervals are calcul ated on the
assumption that a, E and a are variable quantiti es, arising from
the experiment described earlier (in the previous subsection), and
"if the original estimates arc used repeatedl y (not all owed to vary)
the statement may not be accurate" (Mood and Graybill 1963, p.
337).
But Mood and Graybill's idea of simpl y adding the new
datum to the old and re-estimating the regress ion parameters
does not sol ve the probl em. For although it ensures variability
for the parameter estimates, it fixes the ori ginal data points,
which arc supposed to be variable, and it vari es fl , which
should be fixed. Thi s means that the true distribution of t, upon
which the confidence interval is based, is not the one described
above. The questi on of what the proper I-di stribution is has
not , so far as we are awa re, been addressed. But until it is , clas-
sical stati sti cians ought properl y to restrict themselves to sin-
gle predictions, or e lse change their methodology, for this
inconvenient, unintuiti ve, and regularly ignored restriction
does not arise for the Bayesian, who is at libert y to revise esti-
mates with steadily acc umulating data: indeed, there is an obli-
gation to do so.
7.e Examining the Form of a Regression
The classical way of investi gating relationships between variables
is, as we have described, to assume some general form for the
REGRESSION ANALYSIS 221
relationship and then allow the data to dictatc its particular shape
by supplying values for the unspecified parameters. Tn their start-
ing assumption, statisticians show a marked preference for linear
regressions, partly on account of the conceptual simplicity of such
regressions and partly because their properties with respect to
classical estimation techniques have been so thoroughly explored.
But statistical relati onships are not necessarily nor even normally
linear ("we might expect linear relationships to be the exception
rather than the rul e"- Weisberg 1980, p. 126); and any data set
can be fitted equally well (however this success is measured) to
infinitely many different curves; hence, as Cook (1986, p. 393)
has said, "some reassurance [that the model used is "sensible"] is
always necessary".
But how is that reassurance secured, and what is its character?
The latter question is barely addressed by classical statisticians,
but the former, it is widely agreed, can be dealt with in one or
more of three ways . First, certain aspects of possibl e model s, it
is said, can be checked using significance tests. There is, for
example, a commonly employed test of the hypothesis that thc
(3-parameter of a linear regression is zero. Thi s employs a t-test-
/3 ~
statistic with n -- 2 degrees of freedom: t = - ~ - where s(()) is the
s(())
standard error of p, which can be estimated from the data. Another
test is often employed to check the homoscedasticity assumption
of the linear regression hypothesis, that is, the assumption that a
is independent of x. Such tests are of course subj ect to the stric-
tures we made on significance tests in general, and consequently,
in our view, they have no standing as modes of inductive reason-
ing. But even if the tcsts were valid, as their advocates maintain,
they would need to be supplemented by othcr mcthods for evalu-
ating regression models, for the familiar reason that the conclu-
sions that may be drawn from significance tests are too weak for
practical purposes. Learning that some preci se hypothesis, say
that f3 = 0, has been rejected at such-and-such a sign if icance levcl
is to learn very littl e, since in most cases it would already havc
been extremely unlikely, intuitively speaking, that the parameter
was exactly zero. In any case, most practitioners require more
than this meagre, negative information; they wi sh to know what
222
CHAPTER 7
the true model actually is. The methods to be discussed in the next
two subsections are often held to be helpful in this respect. They
deal first with the idea that the preferred regression model or
models should depend on prior knowledge; and secondly wi th
techniques that subject the data to detailed appraisal , with a view
to extracting information about the true model.
Prior Knowledge
Hays (1 969, p. 551) observed that "when an experimenter wants
to look into the question of trend, or form of relationship, he has
some prior ideas about what the population relation should be
like. These hunches about trend often come directly from theory,
or they may come from the extrapolation of established findings
into new areas."
Such hunches or pri or ideas about the true relationship are
often very persuasive. For example, in studying how the breaking
load (b) varies with the di ameter (d) of certain fibre segments,
Cox (1 968, p. 268) affirmed that as the di ameter vanishes, so
should the breaking load; Seber (1 977, p. 178) regarded this as
"obvious". More often, prior beliefs are less strongly held. For
exampl e, Wonnacott and Wonnacott, having submitted their data
on wheat yields at different concentrations of fertiliser to a linear
least-squares analysis, pointed out that the assumpti on of lineari-
ty is probably fal se: "it is likely that the true relation increases ini-
ti ally, but then bends down eventually as a 'burning point' is
approached, and the crop is overdosed" (p. 49; italics added) .
Similarly, Lewis-Beck observed that although a linear relation
between income and number of years of education appears sati s-
factory, judged from the data, "it seems likely that relevant vari-
ables have been excluded, for factors besides educati on
undoubtedly influence income" ( 1980, p. 27; italics added).
Other statisticians express their uncertainty about the general
model in less obviously probabili stic terms, such as "sensible"
and " reasonable on theoretical grounds" (Weisberg 1980, p. 126),
and "pl ausible" (Atkinson 1985, p. 3); whil e Brook and Arnold
talked of "theoretical clues . . . whi ch point to a particular rela-
tionship" (1985, p. 12; italics added); and Sprent (1 969, p. 120),
stretching tentati veness almost to the limit, said that an "experi-
REGRESSION ANALYSIS 223
menter is often in a position ... to decide that it is reasonable to
assume that certain general types of hypothesis ... may hold,
although he is uncertain precisely which".
One further example: Cox (1968, p. 268) argued that if, in the
case of the breaking load and the diameter of fibres, the two lines
b ex. d and log b ex. log d fit the data equally well, then the "second
would in general be preferable because ... it permits easier com-
parison with the theoretical model load ex. (diameter)2". This is a
very circumspect way of recommending a regression model and
would seem to be no recommendation at all unless the "theoreti-
cal model" were regarded as likely to be at least approximately
true. That this is the implicit assumption seems to be confirmed
by Seber's exposition (1977, p. 178) of Cox's view, in which he
commended the model slightly less tentatively, saying that it
"might be" a "reasonable assumption".
One might expect the natural uncertainty attaching to the gen-
eral model to be revised, ideally diminished, by the data, so pro-
ducing an overall conclusion that incorporates both the prior and
posterior information. This is how a Bayesian analysis would
operate. A Bayesian would interpret the various expressions of
uncertainty uniformly as prior probabilities and then use the data
to obtain corresponding posterior probabilities, though if the dis-
tribution of prior beliefs is at all complicated, the mathematics
may become difficult or even intractable.
In the classical case, the difficulty is not merely mathematical
or technical, but arises from a fundamental flaw. For, in the first
place, the prior evaluation of a model in the light of plausible the-
ories and previous data seems not, as a rule, to be objectively
quantifiable, as the classical approach would demand; certainly
none of the authors we have quoted offers any measure, objective
or otherwise, of the strength of their hunches. Secondly, even if
the reasonableness of a theory could be objectively measured,
classical statistics offers no way of combining such measures with
the results of standard inference techniques (significance testing,
confidence-interval estimation, and so forth) to achieve an aggre-
gate index of appraisal.
The difficulty of incorporating uncertainty about the model
into standard classical inferences is apparent from a commonly
encountered discussion concerning predictions. Typical exposi-
224 CHAPTER 7
tions proceed by first ass uming a linear relation. The apparatus of
confidence intervals is next invoked as a way of qualifying pre-
dictions with a degree of confidence. [t is then explained that the
linearity assumption is often doubtful, though perhaps roughly
correct over the relatively narrow experimental range, and hence
that one should not expect predictions from an equation fitted by
least squares to be quite accurate (for example Weisberg 1980, p.
126; Seber 1977, p. 6; Gunst and Mason 1980, pp. 56-63;
Wonnacott and Wonnacott 1980, p. 49).
But instead of measuring this uncertainty about the model and
then amal gamating it with the uncertainty refl ected in the confi-
dence interval , so as to give an overall level of confidence in a
prediction, classical statisticians merely issue warnings to be
"careful not to attempt to predict very far outside the [range cov-
ercd by the data] " (Gunst and Mason, p. 62), and to "be reluctant
to make predictions [ except for] . .. new cases with predictor vari-
ables not too different from [those] . . . in the construction sam-
ple" (Weisberg, p. 2 15). But these vague admonitions do not
signify how such cauti on should be exercised (should one hold off
from predicting altogether, or tremble slightly when hazarding a
forecast, or what?).
There is also the question of how close Xli should be to thc X
values in the data, before the corresponding Y
li
can be predicted
with assurance. Thi s is always given an arbitrary answer. For
example, Weisberg, in dealing with the case where y is linearly
related to two predi ctor variables x and ? , suggested, with no the-
oretical or epi stemological sancti on, that a "range of validity for
prediction" can be determined by plotting the data values of X and
? and drawing "the smallest closcd figure that includes all thc
points" (p. 216). Making such a drawing turns out to be difficult,
but Wei sberg considered that "the smallest volume ellipsoid con-
taining these points" (p. 216) would be a sati sfactory approxima-
tion to the closed f igure. However, he is uneasy with the proposal
since, in the exampl e on which he was working, there was " a sub-
stantial area inside the [ellipsoid] ... with no observed data" (p.
217) and where, presumabl y, he felt predi cti on is unsafe.
Weisberg's demarcati on between regions of safe and ri sky predic-
tions and his suggested approximation to it have some plausibili-
ty, but they seem quite arbitrary from the classical point of view.
REGRESSION ANALYSIS 225
Classical statisticians are caught in a cleft stick. If they take
account of plausible prior beliefs concerning the regression
model. they cannot properly combine those beliefs with the clas-
sical techniques of inference. On the other hanci if they use those
techniques but eschew prior beliefs, they have no means of select-
ing, arbitrary stipulation apart, among the infinitely many regres-
sion models that are compatible with the data.
Data Analysis
A possible way out of this dilemma is to abandon the imprecise,
unsystematic, and subjective appraisals described in the previous
section and rely solely on the seemingly more objective methods of
'data analysis', though most statisticians would not go so far, pre-
ferring to avail themselves of both techniques. Data analysis, or
'case analysis', as it is often calleci is an attempt to discriminate in
a non-Bayesian way between possible regression models, through a
close examination ofthe individual data points. There are three dis-
tinct approaches to data analysis, which we shall consider in turn.
i Impeding scatter plots. The simplest kind of data analysis
involves visually examining ordinary plots of the data (,scatter
plots') in an informal way. The procedure was authoritatively
endorsed by Cox (1968, p. 268 )-"the choice [of relation] will
depend on preliminary plotting and inspection of the data"; and it
was "strongly recommended" by Kendall and Stuart (1979, p.
292), because "it conveys quickly and simply an idea of the ade-
quacy of the fitted regression lines". Visual inspection is widely
employed in practice to augment the formal process of classical
estimation. Weisberg (p. 3), for instance, motivated a linear
regression analysis of certain data on the boiling points of water
at different atmospheric pressures by referring to the "overall
impression of the scatter plot ... that the points generally, but not
exactly, fall on a straight line". And Lewis-Beck (p. IS) claimed
that "visual inspection of [a certain] ... scatter plot suggests the
relationship is essentially linear".
Weisberg (p. 99) argued that the data of Diagram I showed a
pattern that "one might expect to observe if the simple linear
226 CHAPTER 7
regression model were appropriate". On the other hand, Diagram
2 "suggests . .. [to him] that the analysis based on simple linear
regression is incorrect, and that a smooth curve, perhaps a quad-
ratic polynomial , could be fit to the data ... ". Although the data
are different in the two cases, they give the same least squares
lines, as the diagrams below illustrate.
DIAGRAM! DIAGRAM 2
Probably most scientists would agree with the various judgments
that statisticians make on the basis of visual inspections of scatter
diagrams. But how are such judgments arrived at'? It could be,
indeed, it seems likely, that a closc analysis would reveal a
Bayesian mechani sm, but it is hard to imagine how any of the
standard classical techniques could be involved in either explain-
ing or justifying the process, nor do classical statisticians claim
they are. The whole process is impressioni stic, arbitrary, and sub-
jective.
ii Outliers. Another instructive data pattern is illustrated in
Diagram 3. One of the points stands much further apart from the
least squares line than any other and this suggests to Weisberg (p.
99) that "simple lincar regression may be correct for most of the
data, but one of the cases is too far away from the fitted regres-
sion line". Such points are called outliers.
Outl iers are def ined, rather imprecisely, as "data points [with]
... residual s that are large relative to the resi duals for the remain-
der of the observations" (Chatterjee and Price 1977, p. 19). (The
residual of a data point is its vertical di stance from a fitted line or
curve.) As is often noted, a point may be an outlier for three dis-
tinct reasons. First, it could be erroneous, in the sense that it
REGRESSION ANALYSI S 227
outlier
15
./
residual--_.
"
the least squares line
10
5
DIAGRAM 3
resulted from a recording or transcnptlOn error or from an
improperly conducted experiment; an outlier could also arise
because the assumed regression model is incorrect; on the other
hand, the model might be correct and the outlier be simply one of
those relatively improbable cases that is almost bound to occur
sometimes. Suppose that careful checking has more or less
excluded the first possibility, does the outlier throw any light on
whether an assumed regression equation is correct or not? This is
a question to which classical statisticians have applied a variety of
non-Bayesian methods, though, as we shall argue, without any
satisfactory result.
Some authors, for example Chatterjee and Hadi, seem not to
take seriously the possibility that the linear least-squares line is
wrong, when they note that an outlier in their data, if removed,
would hardly affect the fitted line and conclude from this that
"there is little point in agonizing over how deviant it appears"
(1986, p. 381) . But this conclusion is unjustified, and was not
endorsed by Chatterjee when he previously collaborated with
Price.
Chatterjee and Price's (1977) approach may be illustrated with
their own example. Their data consisted of 30 x,y readings, the
nature of which need not concern us. They applied the linear least
squares technique to the readings, obtaining an upwardly sloping
regression line. They next examined a number of data plots, first
y against x, then residuals against x, and finally the residuals
against) (Y
i
is the point on the fitted line corresponding to x).
Four of the points in their data stood out as having particularly
228 CHAPTER 7
large residuals; moreover, a visual inspection of the various plots
made it "clear [to the authors] that the straight line assumption is
not confirmed" (p. 24), though it "looks acceptable" in the mid-
dle of the x range. On this informal basis, Chatterjee and Price
concluded tentatively that the true line has a zero gradient and that
y is independent of x. They checked this conjecture by dropping
the four outliers, computing a new least-squares line and then
examining the revised plots of residuals. This time they found no
discernible pattern; from a casual inspection "the residuals appear
to be randomly distributed around [the line] e[residual] = 0" (p.
25). Unfortunately, the conclusion from all this is disappointingly
weak. It is that the regression with zero gradient "i s a satisfacto-
ry model for analyzing the ... data, after the deletion ofthef(JlIr
points" (p. 25; italics added). But this says nothing about the true
relation between x and y. Indeed, the authors acknowledged that
this question was still open by then asking: "what of the four data
points that were deleted?" They repeated the trui sm that the points
may be "measurement or transcription errors", or else "may pro-
vide more valuable information about the . . . relationships
between y and x". We take the latter to mean that the line derived
from the data minus the outliers might be wrong, perhaps, it
should be added, wi Idly so (of course, it might be right, too).
Whether the fitted line is right or wrong, and if wrong, what the
true line is, are questions that Chatterjee and Price simply left to
further research: "it may be most valuable to try to understand the
special circumstances that generated the extreme responses" (p.
25). In other words, their examination of the data from different
points of view has been quite uninformative about the true regres-
sion and about why some of the points have particularly large
residuals relative to the linear least-squares line.
Chatterjee and Price called theirs an "empirical approach" to
outliers, since it takes account not just of the sizes of the residu-
als but also of their patterns of distribution. They disagreed with
those who use significance tests alone to form judgments from
outliers since, in their view, "[i]t is a combination of the actual
magnitude and the pattern of residuals that suggests problems" (p.
27). But although significance tests take little account of the pat-
tern of the results, compared with the empirical approach, they are
more in keeping with the objectivist ideals of classical statistics
REGRESSION ANALYSIS 229
and offer more definite conclusions about the validity of hypoth-
esized regression equations.
Weisberg is a leading exponent of the application of signifi-
cance tests in this context. His method is to perform a linear least-
squares analysis on the data set minus one point (in practice this
would be an outlying point) and then to use a significance test to
check the resulting regression equation against the removed point.
To see how this 'outlier test' is carried out, suppose the fitted line
derived from the data, minus the ith point, has parameters a . and
A t
/3
i
. The assumption that this is the true line will be the null
hypothesis in what follows. Let i be the y value corresponding to
Xi on the null hypothesis line.
Y
Y
i
1-----------.
Y, 1-- ---------,7('
least squares line
constructed
without Xi 'Y
i
L-_ ______ -L __________ X
Provided the data were properly collected and recorded, it is
natural to expect that the larger the discrepancy between ,Vi and Y
i
,
the less would be one's confidence that the line drawn is correct.
This idea is given classical clothes through a corresponding sig-
nificance test. Weisberg's test uses as test-statistic a particular
function of ,vi - ,Vi and other aspects of the data- this has the
{-distribution with 11 - 3 degrees of freedom. I f the {-value record-
ed for a data set, relative to a particular point (Xi' y) in that set,
exceeds the test's critical value, then the null hypothesis would be
rejected at the corresponding level of significance.
230 CHAPTER 7
But fixing the appropriate critical value is interestingly prob-
lematic and shows up the pseudo-objectivity and ineffectiveness
of the whole procedure. It is required in significance testing to
ensure that the null hypothesis would be falsely rejected with
some pre-designated probability, usually 0.05 or 0.01. Now
Weisberg's test could be conducted in a spectrum of ways. At one
end of the spectrum you could decide in advance to perform just
one significance test, using the single datum corresponding to a
pre-speci fied X i (call this the first testing plan). At the other
extreme, the plan could be to check the significance of every data
point. As Weisberg points out, if the test were, as a matter of pol-
icy, restricted to the datum with the largest t-value, one would in
effect be following the second testing plan.
Suppose you selected a significance level of 0.05. This would
be classically acceptable, provided the first plan was adopted and
the test applied just to a single, pre-selected point. But if the sec-
ond plan was adopted, and the same significance level chosen, the
overall probability of rejecting the null hypothesis, if it wcre true,
would be a multiple of 0.05, that multiple depending on the num-
ber of points in the data set. You would, in this case, then have to
reduce the significance level of the individual tests, in order to
bring the overall significance level to a classically acceptable
level. (In Weisberg's example, involving 65 data points, the sec-
ond testing plan would need to use a significance level of 0.00077
for the individual tests. )
So whether a particular I-value is significant and warrants the
rejection of a null hypothesis is sensitive to how the person per-
forming the significance test planned to select data for testing.
But , as we have stressed before, such private plans have no epis-
temic significance, and without justifiable, public rules to fix the
most appropriate plan, the present approach is subject to person-
al idiosyncrasies that are at odds with its supposed objectivity.
Weisberg (p. 116) docs in fact propose a rule for choosing a
testing plan, namely, that the one involving a single significance
test, should be adopted only "if the investi gator suspects in
advance" that a particul ar case will fail to fit the same regression
line as the others. But what should the experimenter do if, having
decided to adopt thi s plan, the anticipated outlier unexpectedly
fitted the pattern and some of the other points were surprisingly
REGRESSION ANALYSI S 231
prominent outliers') And why should informal and untested "sus-
picions" have any standing in this area?
Weisberg introduced and illustrated his rule with his data on
the average brain (hr) and body weights (ho) of different specics
of mammals, which show man as the most prominent outlier in a
plot oflog(hr) against log(bo). Weisberg (p. 130) argued that since
"interest in Man as a special case is a priori ", the outlier test
should employ the largcr significance level, corresponding to the
first testing plan. On this basis the datum on man was significant
and so Weisberg concluded that "humans would bc dcclared to
havc brain weight that is too large to be consistcnt [sic] with the
[log-linear) model for the data". But if therc had been no prior
"interest in Man", the datum would not have been significant and
Weisberg would have reached the opposite conclusion.
Weisberg's example, morcover, does not even conform to his
own rule: man is not picked out because of an earlier " suspicion"
that he is an exception to the log-linear rule, but because ofa prior
"intcrest" in Man as a special case. No explanation is given for
this change, though we conjecture that it is madc because, first,
from a visual inspection of the data, Man seems clearly to be an
exception to a log-linear rule; secondly, if the smaller significance
level , corresponding to the second testing plan, were employed in
Weisberg's outlier test, Man would not be significant and the lin-
ear rule would, counter-intuitively, have to be accepted; and third-
ly, there is no plausible reason to suspect Man a priori of being an
exception to a log-linear rule-hence, no reason to employ the
larger critical value in the significance test. This may perhaps
explain why Weisberg changed his rule to suit these circulll-
stances, but clearly it provides no justification.
It seems undeniable that the distribution patterns of data,
including the presence of outliers, often tell us a great deal about
the regression, but they seem not to speak in any known classical
language. It is perhaps too early to say for certain that the Illedi-
um of communication is Bayesian, since a thorough Baycsian
analysis of the outlier notion is still awaited. The form that that
analysis will take is perfectly clear, however. As with every other
problem concerning the evaluation of a theory in the light of evi-
dence, it will involve the application of Bayes's theorem. Thus,
for a Bayesian, regression analysis is not divided into separate
232 CHAPTER 7
compartments operating distinct techniques, each requmng its
own justification; there is just one principle of inference, leading
to one kind of conclusion.
iii points. Another data pattern that statistici ans
often find instructive is illustrated in Diagram 4 (Wei sberg, p. 99).
least squares line
j

15
residual
10
influential
5
point
DIAGRAM 4
The isolated point on the right has a nil residual , so the con-
siderati ons of the previous section give no reason to doubt the lin-
ear least squares line. But the point is peculiar in that the
estimated line depends very largely on it alone, much more so
than on any of the other data. Such points are called' influenti al'.
Diagram 4 shows an extreme case of an influential data point ; it
is extreme because without it no least squares solution even
exists. There are less pronounced ki nds of influence which stati s-
ticians also regard as important, where parti cular data points or a
small subset of the data "have a di sproportionate influence on the
estimated [regress ion] parameters" (Belsley et al. 1980, p. 6).
Many statisti cians regard such influential points as deserving
special attention, though they rarely explain why; indeed, large
textbooks are written on how to measure "int1uence", without the
purpose of the project being adequately examined. All too often,
the argument proceeds thus: "these [parameter estimates] . . . can
be substantiall y influenced by one observati on or a few observa-
tions; that is, not all the observations have an equal importance in
least squares regression ... It is, Iherej(Jre, important for an ana-
lyst to be able to identify such observations and assess their
effects on various aspects of the analysis" (Chatterj ee and Hadi
REGRESSION ANALYSIS 233
1986, p. 379; italics added). Atkinson (1986, p. 398) and Cook
(1986, p. 393) give essentially the same argument, which is clear-
ly a non sequitur, without the further and no doubt implicit
assumption that conclusions obtained using influential points are
correspondingly insecure. Weisberg (p. 100) is one of the few who
states this explicitly, when he says that "[ w]e must distrust an
aggregate analysis that is so heavily dependent upon a single
case". Belsley et al. (1980, p. 9) say roughly the same but express
it in a purely descriptive mode: "the researcher is likely to be
highly suspicious of the estimate [of the slope of a regression
line]" that has been obtained using influential data.
But why should we distrust conclusions substantially based
on just a few data points? Intuitively, there are two reasons. The
first is this : the true regression curve passes through the mean, or
expected value, of y at each x. In describing our best guess as to
that curve, and hence our best estimate of those means, we are
guided by the observed points. But we are aware that any of the
points could be quite distant from the true regression line, either
because the experiment was badly conducted or because of an
error in recording or transcribing, or simply because it is one of
those rare and improbable departures from the mean that is
almost bound to occur from time to time. Sharp departures of a
point from its corresponding mean are relati vely improbable, but
if such a discrepancy actually arose and the datum was, more-
over, influential, the conclusion arrived at would be in error by a
corresponding margin. Our intuition is that the more likely a
reading is to be discrepant in the indicated sense, and the more
influential it is, the less certain would one be about the regres-
sion. The second intuitive reason for distrusting conclusions that
are partly derived from influential data applies when they are
separated from the rest of the data by a wide gap, as for example
in Diagram 4; intuitively we feel that in the range where there are
few or no data points, we cannot be at all sure of the shape of the
regression relation.
A programme of research that was initiated in the early 1970s
aims to explicate these intuitions in classical (or at any rate, non-
Bayesian) terms and to develop rules to guide and justify them.
This now flourishing field is called ' influence methodology', its
chief object being to find ways of measuring influence.
234 CHAPTER 7
The idea governing the meas urement of influence is thi s. You
should first estimate a regress ion parameter using all the data,
and then re-estimate it with a selected data point deleted, noting
the difference between the two estimates. Thi s difference is then
plugged into an ' influence function' or, as it is sometimes called,
a 'regression diagnostic' , to produce an index of how influential
the deleted point was. To put this programme into effect, you
need first to decide on the parameters whose estimates are to be
employed in the influence measure. If the regression has been
assumed linear with a single independent variable, you could
choose from among the three parameters, a , {3, and a , as well as
from the infinity of possible functional combinations of these.
Secondly, a measure of influence is required, that is, some func-
tion of the difference noted above and (possibly) other aspects of
the data; the choice here is similarly vast. Finally, there has to be
some way of usi ng particular values of an influence measure to
judge the reliability of the data point or of the regression
assumptions.
Many influence functions have been proposed and suggestions
advanced as to the numerical value such functions should take
before the reading concerned is designated ' influential ' . But these
proposals and suggestions seem highly arbitrary, an impression
confirmed by those working in the field. For instance, the author
of a function known as Cook's Distance admitted that "[f]or the
most part the development of influence methodology for linear
regression is based on ad hoc reasoning and this partially accounts
for the di versity of recommendat ions" (Cook 1986, p. 396).
Arbitrariness and adhocness would not be a feature of thi s area
if a relation had been established between a point's influence as
def ined by a given measure and the reliability or credibility of a
fitted line or curve. No such relation has been establi shed.
Nevertheless, recommendations on how to interpret levels of
influence are sometimes made. For example, Velleman and
Welsch (1981) felt that "val ues [of their preferred function)
greater than I or 2 seem reasonable to nominate ... for special
attention"; but they fai led to say what speci al attention is called
for, nor why.
Many of those most active in this area acknowledge the appar-
ent absence of any objective epistemic constraints when it comes
REGRESSION ANALYSIS 23S
to interpreting influence measures. For example, Welsch (1986, p.
405), joint author of another influence function, conceded that
"[e]ven with a vast arsenal of diagnostics, it is very hard to write
down rules that can be used to guide a data analysis. So much is
really subjective and subtle". And Velleman (1986, p. 413), who
has already been quoted, talked of " the need to combine human
judgment with diagnostics", without indicating how this judgment
operates, is justified, or coheres with the rest of classical statistics.
A full Bayesian analysis of influential data has yet to be under-
taken. It would, we are sure, endorse many of the intuitions that
guide non-Bayesian treatments, but unlike these, the path it
should take and its epistemic goal are clear; using Bayes's theo-
rem, it would have to trace the effects of uncertainty in the accu-
racy of readings and of relatively isolated data points on the
posterior probabilities of possible regression models. The techni-
cal difficulties facing such a programme arc formidable, to be
sure (and these difficulties are often referred to by critics wishing
to cast doubt over the whole Bayesian enterprise), but the difficul-
ties that any Bayesian analysis of regression data faces arise only
from the complexity of the situation and by no means reflect on
the adequacy of the methodology. The three-body problem in
physics provides an instructive analogy; this problem has so far
resisted a complete solution in Newtonian terms and may, for all
we know, be intrinsically insoluble, but nobody thinks that
Newton's laws are in the least discredited because of the mathe-
matical difficulties of applying them in thi s area.
By contrast with a Bayesian approach, the programme of
influence methodology based on classical ideas runs into trouble
not simply because of intractable mathematics but, we suggest,
because it has no epistemically relevant goal. This explains why
the rules and techniques proposed arc arbitrary, unjustified, and
ad hoc; it is hard to see how they could be otherwise.
7.f Conclusion
In earlier chapters, we catalogued various aspects of classical
methods of testing and estimation that show that they are unsuited
to the tasks of inductive inference for which they were invented.
236 CHAPTER 7
Those same shortcomings, not surprisingly, surface too when
regression problems are at issue. However, some new difficulties
are also revealed. For instance, extra criteria for estimation are
required, which have led to the plausible, but cl assically indefensi-
ble, least squares princi ple. There are also extra sources of subjec-
tivity, for instance, in selecting the regression model, both when
taking account of 'pri or knowledge' and when judging models by
informally inspecting scatter plots. They are also present in the
process of checking models against outliers and influential data.
Thi s subjecti vity frequently passes unnoticed. Thus Wonnacott and
Wonnacott ( 1980, p. 15) set themselves firmly against judging the
gradient and intercept of a regression line "by eye", on account of
its subjectivity-"we need to find a method that is obj ective"; but
without batting an eyelid at their inconstancy, they allow personal
judgment and the visual inspection of scatter plots to play a crucial
part in determining the overall conclusion.
Unlike the hotchpotch of ad hoc and unjustif iable rules of
inference that constitute the classical approach, Bayes's theorem
supplies a single, uni versally applicable, well-founded inductive
rule which answers what Brandt (1986, p. 407) call s the "most
important ... need for integration of this [influence methodolo-
gy] and many other aspects of [classical] regression and model
fitting into a coherent whole".
CHAPTER 8
Bayesian Induction:
Statistical Theories
Bayesian induction is the computation via Bayes's theorem of a
posterior probability, or density distribution from a corresponding
prior distribution, on receiving new information. The theorem
does not discriminate between deterministic and statistical theo-
ries, and so affords a uniform treatment for both, in contrast to the
hotchpotch of non-Bayesian methods that have been invented to
suit different circumstances. In an earlier chapter we considered
how non-Bayesian and Bayesian approaches compared when they
were applied to deterministic hypotheses. Here we make a similar
comparison in relation to statistical theories.
8.0 The Question of Subjectivity
The prior distribution from which a Bayesian analysis proceeds
reflects a person's beliefs before the experimental results are
known. Those beliefs are subjective, in the sense that they are
shaped in part by elusive, idiosyncratic influences, so they are
likely to vary from person to person. The subjectivity of the prem-
ises might suggest that the conclusion of a Bayesian induction is
similarly idiosyncratic, subjective and variable, which would con-
flict with a striking feature of science, namely, its substantially
objective character.
A number of attempts have been made to reconcile Bayesian
methodology with scientific objectivity. For example, it has been
argued that the difficulty may be stemmed at source by repudiat-
ing subjective in favour of purely objective prior probabilities on
which all scientists might rationally agree. But the fact is that sci-
entists often take very different views on how credible particular
theories are, especially in the early stages of an investigation. And,
238 CHAPTER 8
more seriously, although the idea that scientific theories possess
unique, objectively correct, inductive probabilities is eminently
appealing, it has so far resisted determined efforts at a satisfacto-
ry analysis, except in the special cases of tautologies and contra-
dictions, and there is now a wide consensus that no such analysis
is possible.
A second approach tries to exploit certain limit theorems of
Bayesian inductive theory. These are theorems (treated in detail in
the next chapter) to the effect that under certain, mild conditions,
for instance, that there is agreement on which hypotheses, if any,
have zero prior probability, different Bayesian agents will, in the
limit, as data accumulate without bounds, agree in their posterior
views, whatever their subjective prior opinions were. But, remark-
able as they are, the limit theorems are of little utility in the present
context. For onc thing, since they deal only with the properties of
the posterior probability function in the limit, as the amount of data
increases to infinity, they say nothing at all about its character after
any actual and hence finite amount of data; they are therefore inca-
pable of acting in either a normative or an explanatory role.
There is a third approach, which is the one we favour. It does
not seck to account for a global identity of opinion, nor for a con-
vergence to such an identity once some specified amount of data
is available, neither of which seem to be universal facts of science.
The approach recognizes that scientific opinions often do con-
verge, often indeed, very quickly, after relatively little data. And it
appeal s to Bayes's theorem to provide an explanatory framework
for this phenomenon, and to give specific explanations for specif-
ic circumstances. We believe that such explanations should be
judged case by case, and that when this is done, the judgments
will usually be favourable.
Two cases which crop up frequently in the literature of statis-
tical inference, as well as in practical research, and which we con-
sidered earlier in connexion with the frequentist approach, are the
estimation firstly, of the mean of a normal population, and sec-
ondly, of a binomial proportion. We will show that in both these
estimation tasks, Bayesian reasoners, even when their prior opin-
ions are very different, are forced into ever-closer agreement in
their posterior beliefs as the data accumulate. And, more signifi-
cantly, we will see that this convergence of opinion is fairly rapid.
BAYESIAN INDUCTION: STATISTICAL THEORIES 239
Estimating the Mean of a Normal Population
We used this customary example in Chapter 5 to present the ideas
of classical interval estimation. In the example, the population
whose mean is to be estimated is already known to be normal and
known to have standard deviation a. The assumption of such
knowledge is more or less realistic in many cases, for instance,
where an instrument is used to measure some physical quantity.
The instrument would, as a rule, deliver a spread of results if used
repeatedly under similar conditions, and experience shows that
this variability, or error distribution, often approximates a normal
curve. Making measurements with such an instrument would then
be practically equivalent to drawing a random sample of observa-
tions from a normal population of possible observations whose
mean is the unknown quantity and whose standard deviation was
established from previous calibrations.
Let e be a variable that ranges over possible values of the pop-
ulation mean. We shall assume for the present that prior opinion is
represented by a density distribution over e that is also normal,
with a mean of f.1o and a standard deviation of a
o
' In virtually no
real case would the prior distribution be strictly normal, for physi-
cal considerations usually impose limits on a parameter's possible
values, while normal distributions assign positive probabilities to
every range. So for instance, the average height of a human popu-
lation could neither be negative, nor above five thousand miles.
Nevertheless, a normal distribution often provides a mathematical-
ly convenient idealization of sufficient accuracy. Assuming a nor-
mal distribution simplifies this illustration of Bayesian induction
at work, but as we show later, the assumption may be considerably
relaxed without substantially affecting the conclusions.
Suppose a random sample of size n and mean x has been
drawn from the population. The posterior di stribution of e, rela-
tive to these data, turns out to be normal , like the prior, and its
mean, ,UII ' and its standard deviation, 0" , are given by:
nx a 2 + {loa
o
2 I n I
u = --- ------- and 0 + -
'II na
2
+ a -
2
a- 0
2
a
0
2
'
I) II
These results, which are proved by Lindley, 1965, for example,
are illustrated below.
240 CHAPTER 8
posterior distribution
prior distribution
flo
The precision of a distribution is defined as the reciprocal of
its variance. So the second of the above equations tells us that the
precision of the posterior distribution increases with the precision,
(J 2, of the population whose mean is being estimated. By the
same token, the precision of a measured value increases with that
of the measuring instrumcnt (whose precision is the reciprocal of
the variance of its error distribution). The more precise an esti-
mate the less uncertainty there is about the parameter's value, and
the natural wish to diminish uncertainty accounts for the appeal of
the efficiency criterion that classical statisticians have imposed,
for inadequate reasons (see S.f above), on their estimators.
The above equations also show that as n increases, ,u", the
mean of the posterior distribution, tends to X, the mean of the
o
sample. Similarly, (J 2 tcnds to CJ--=- , a quantity that depends on the
" n
sample and on the population but not on the prior distribution.
This means that as the sample is enlarged, the contribution of the
prior distribution, and so of the subjective part of the inference,
lessens, eventually dwindling to insignificance. Hence two people
proceeding from different normal prior distributions would, with
sufficient data, converge on posterior distributions that were arbi-
trarily close. Moreover-and this is the crucial point for explain-
ing the objectivity of the inference-the objective information
contained in the sample becomes the dominant factor relatively
quickly.
We can show how quick the convergence of opinion may be by
an example where the aim is to estimate the mean of a normal
popUlation whose standard deviation is already known to be 10.
BAYESIAN INDUCTI ON: STATISTICAL THEORIES 24 1
And we consider the case in which one person's normal prior dis-
tributi on over the population mean is centred on 10, with a stan-
dard deviation of 10, while another's is centred on 100, with a
standard deviation of 20. This difference represents a very sharp
di vergence of initial opinion, because the region that in the view
of the first person almost certainly contains the true mean is prac-
tically certain not to contain it as far as the second is concerned.
But even so profound a disagreement as this is resolved after rel-
atively few observations. The foll owing table shows the means
and standard deviat ions of the posterior distributions of the two
people, relative to random samples of various sizes, each sample
having a mean of 50.
Sample size Person 1 Person 2
n
,un
a jU
n
(J
n n
--- -_ . .
0 10 10 100 20
1 30 7.1 60 8.9
5 43 4.1 52 4.4
10 46 3.0 5 I 3.1
20 48 2.2 5 I 2.2
100 50 1.0 50 1.0
We deliberately chose an extreme exampl e, where the prior
di stributi ons scarcely overlap. The f irst line of the table, corre-
sponding to no data, represents thi s initial position. Yet we see that
a sample of only 20 brings the two posterior distributions very
close, while a sample of 100 renders them indistinguishable. Not
surprisingly, the closer opinions are at the start, the less evidence
is needed to bring the corresponding posterior opinions within
given bounds of simil arity. I Hence, although the Bayesian analy-
sis of the case under consideration must proceed from a largely
subjective prior di stribution, the most powerful influence on its
conclusion is the objective experimental evidence. We shall see
that the same is more generall y true.
I See, for instance, Pratt et al. 1965, Chapter 1 I.
242 CHAPTER 8
Estimating a Binomial Proportion
Another standard problem in inferential statistics is how to esti-
mate a proportion, for instance, of red counters in an urn, or of
Republican sympathisers in the population, or of the physical
probability of a coin turning up heads when it is flipped. Data are
collected by randomly sampling the urn or the population, with
replacement, or by flipping the coin, and then noting for each
counter sampled whether it is red or not, and for each person sam-
pled his or her political sympathies, and for each toss of the coin
whether it landed heads or tails. If, as in these cases, there are just
two possible outcomes, with probabilities 8, and I - 8, constant
from trial to trial , then the data-generating process is known as a
Bernoulli process and 8 and 1 - 8 are called the Bernoulli param-
eters, or the binomial proportions. The outcomes of a Bernoulli
process are conventionally labelled ' success' and 'failure'.
The Bayesian method of estimating a Bernoulli parameter
from given experimental results starts, of course, by describing a
prior distribution, which we shall assume to have the form of a so-
called beta distribution. This restriction has the expository advan-
tage of simplifying the calculation of the corresponding posterior
distribution. The restriction is not in fact a severe one, for beta
distributions take on a wide variety of shapes, depending on the
values of two positive-valued parameters, u and v, enabling you to
choose a beta distribution that best approximates your actual dis-
tribution of beliefs. And as we shall see, with sufficient data, pos-
terior probabilities are not much affected by even quite big
changes in the corresponding priors, so that inaccuracies in the
specification of the prior will then not have a significant effect.
A random variable x is said to have the beta distribution if its
density is given by
P(x)
8(u, v) X,,-I (I - x)' I
o
O<x<1
elsewhere.
The parameters u and v are both greater than 0, and 8(u, v) =
qu + v)
- - -- . We do not need to spell out the gamma function in
qu)qv)
BAYESIAN INDUCTION: STATISTICAL THEORIES 243
detail here, except to note that when w is a positive integer that
r = (w - I)! (with O! defined as I). When VI' is non-integral, the
value of the gamma function can be obtained from tables in math-
ematical handbooks. The following diagram illustrates some beta
distributions for various values of u and v.
6
5
~
. ~ 4
Q)
-0
. ~ 3
:0
ro
.g 2
a:
o 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
SOME BETA DISTRIBUTIONS
The mean and variance of a beta distribution are given by:
u
. (;;';)( ; ,
variance = . mean =
ll+V u+v+!
If the prior distribution over the Bernoulli parameters is of the
beta form, Bayes's theorem is particularly easy to apply. For sup-
pose a random sample of n observations derived from a Bernoulli
process shows s successes andffailures, it then turns out that the
posterior distribution is also of the beta form, with parameters
11' = U + s and v' = v + f Hence, the mean of the posterior
11 + S s
distribution is . This tends to - , as the number of trials
lI+v+n n
increases to infinity; and the variance of the posterior distribution
tends, though more slowly, to zero.
2
Thus, like the earlier exam-
2 See, for example, Pollard 1985, Chapter 8.
244 CHAPTER 8
pic, the influence of the prior distribution upon the posterior di s-
tribution steadily diminishes with the size of the sample, the rate
of diminution being considerable, as simple examples, which the
reader may construct, would show.
Credible Intervals and Confidence Intervals
Parameter estimates are often reported in the form of a range of
possibilities, e.g., 8 = 8* E. This has a natural Bayesian inter-
pretation, namely, as a set of values possessing a high probabil-
ity of containing the true value of 8. In general, if? denotes the
probability that () lies between a and h, then the interval (a, h)
is said to be a 100? percent credible interval for 8. Bayesians
recommend credible intervals as useful summaries of posteri or
distributions.
We may illustrate the idea with our first example, where we
were concerned to estimate the mean of a normal population. We
showed that in the circumstances hypothesized, a suffi cient
amount of data determined a posterior distribution with a mean
equal to the sample mean, X, and a standard deviation, CJIl equal
o CJ
to . Since the distribution is normal, the range x 1.96 . r
vn
contains 8 with probability 0.95, and so constitutes a 95 percent
credible interval.
Of course, there are other 95 percent credible intervals corre-
sponding to different areas of the posterior distribution, for instance,
the infinite range of values defined by () > x 1.640 , and inter-
1/
vals that extend further into the tails of the posterior di stribution
while omitting a more or less narrow band of values around its cen-
tre. There is sometimes a discussion about which of these interva ls
should be "chosen" (for example by Lindley, 1965, Volume 2, pp.
24-25), which we believe to be misconceived, for strictly speaking,
one should not choose an interval, because a choice impli es a com-
mitment in excess of that permitted by the 0.95 probability that all
these intervals share. All 95 percent credible intervals are 011 a par
from the inductive or scienti fic point of view.
Bayesian credible intervals resemble the confidence intervals
of classical statistics. Indeed, ill the particular case before us, the
BAYESIAN INDUCTI ON: STATI STICAL THEORI ES 245
95 percent credible interval depicted in the diagram and the 95
percent confidence interval that is routinely favoured (the short-
est one) coincide. Yet the two types of interval are crucially differ-
ent. The first states the probability, relative to the evidence, that e
lies within the interval. The second says nothing about the proba-
bility of e, nor does it express any degree of uncertainty about e
in non-probabilistic terms, as its defenders, unsuccessfully claim
that it does. Credible intervals provide an intelligible interpreta-
tion and a rational explanation for the intuiti ons underl ying clas-
sical confi dence intervals.
B.b ! The Principle of Stable Estimation
The estimates delivered in our two examples are, as we showed,
very insensitive to variati ons in the pri or distributi ons. However,
since the priors assumed in the examples were restricted, in the
first case, to normal and in the second, to beta di stributions, the
question arises whether thi s insensitivity persi sts when these
restrictions are relaxed. That it does is the burden of the Principl e
of Stable Estimation, due to Edwards, Lindman and Savage 1963,
a practicall y useful aspect of a more general result proved by
Blackwell and Dubins 1962.
The idea is this. Consider a parameter e, with a prior probabil-
ity density distribution u( e), and the corresponding posterior dis-
tribution, relative to some data x. u(e I x). We will denote by
1<v(e I x) the posterior di stribution that would be induced if the
prior were uniform. B is a credible interval based on w(e I x) ,
which is such that

73 /J
where jj is the complement of B, and a is 10-4 or less (that is, B
is a 99.99 percent or higher credible interval). Consider secondly,
the variation of the actual prior distribution within the interval
B, which the second condition says should be small; more specif-
ically, thi s condition stipulates that there be positive numbers cp
(whose actual value is immaterial in thi s context) and (3, the latter
< 0.05, such that for all e in B
246 CHAPTER 8
cp :::; u(e) :::; (1 + (3)cp.
Finally, consider the variation of the prior distribution outside
B. The third condition stipulates that the prior should be
" nowhere astronomically big compared to its nearly constant val-
ues in B". More specifically, for some positive number 6 < 1000
and for all (j
u(e) :::; 6cp.
Edwards, Lindman, and Savage then show that under these
conditions, the true posterior distribution and the one calculated
on the hypothesi s of a uniform prior are approximately the same.
Moreover, the approximation is greater, the smaller a, (3, and 6.
Hence, whatever the prior beliefs of different people, so long as
they meet the said conditions, the principle ensures that their pos-
teri or beliefs are all roughly the same.
In practice, the stated conditions of the Stable Estimation prin-
cipl e are ordinarily quite accessible, and one can check whether
they hold in any particular case. This is done by first calculating
the posterior distribution that would obtain if the prior were uni-
form,3 then examining a 99.99 percent credible interval (the area
shaded in the diagram) to see whether the range of variation of the
true prior within the interval is as small as the principle requires,
and finally checking outside the interval that the actual prior is
never larger than the average value within it by more than the pre-
scribed amount.
The Principle of Stable Estimation assures us that the relative
independence of Bayesian estimates from the prior distributions,
which we noted in our two examples, is not confined to priors
belonging to particular families of distributions. The principle
also tells us that provided the sample is sufficiently large, you do
not need to describe the prior distribution with great accuracy or
precision in order to arrive at a Bayesian esti mate that is both
3 Sincc a uniform di stri bution is only dcfincd over an interval that is bounded at
both cnds. this will involve setting limits to the values that the parameter could
have.
BAYESIAN INDUCTI ON: STATISTICAL THEORIES 247
Posterior based on
uniform prior
True prior
~ ~ Shaded region
~ covers most of
~ rP_o_st_e_rio_r ______
STABLE ESTIMATION
APPLIES
~ ~ ~ ~ ~ ~ ~ ~
Practical Applicati on of the Principle of Stable Est imation (Phillips 1973)
accurate and preci se. This answers the objection that is sometimes
rai sed against Bayesian estimat ion that it can rarely get off the
ground because of the practical difficulties of accurately ascer-
taining one's own or other peopl e's prior belief distributions.
S.c I Describing the Evidence
An experimental result is a phys ical state; on the other hand, sci-
entific evidence is a linguistic statement, and a question that aris-
es is: which aspects of the physical state should go into the
evidence statement'? A complete description would be infinitely
long and clearly impossible. Nor is it desirabl e even as an abstract
ideal to include every aspect of an experimental result in the evi-
dence, for some aspects are plainly irrelevant. For instance, the
colour of the experimenter's shoes when sowing plant seeds
would not normally be worth recording if the genetic structure of
the plant were the issue. Evidence need not refer to irrel evant
detail s such as these, but, as we showed in 5.f, it should omit no
detai ls that are rel evant.
In this context, a fact is rel evant to a hypothesis if knowing it
affects how the hypothesis is appraised; when the method of
appraisal is Bayesian, this means that the relevance of a fact is
determined by whether it alters the probabilities of any of the
hypotheses of interest. We may illustrate this notion of evidential
248 CHAPTER 8
relevance with the simple, but representative problem of estimat-
ing a coin's physical probabilities oflanding heads and tails, using
evidence obtained from flipping it a number of times. Let us say
the coin was flipped 10 times, producing 6 heads and 4 tai Is. The
table below li sts several possible descriptions of such a result and
several circumstances in which it might have been obtained.
TABLE 8.1
Possible descriptions of the result 6 heads, 4 tails obtained in
various cOin-tossing trials
e
/
: 6 heads
e 2: 6 heads, 4 tails, in a trial designed to end aft er 10 toss-
es of the coin
e
3
: 6 heads, 4 tails, in a trial designed to end after 6 heads
appear
e 4: The sequence TTHTH HHHTH
e,: The sequence TTHTHHHHTH obtained in a trial
designed to terminate when the experimenter is called
for lunch
eo : 6 heads, 4 tails
The first description gives a very thin account of the experi-
ment, not even revealing how many tails it produced, information
that is clearl y relevant, and because of thi s omission, its evidential
value is small and hard to quantify. A Bayesian inference from the
evidence statement would require not only a prior di stribution
over (J (the coin's physical probability of landing heads), but al so
one over 11 , the number of times the coin was tossed. It is unnec-
essary to enter further into this complicated case, save to note that
unlike the cl assical approach, and more plausibl y we think, the
Bayesian does not necessarily find e I uninformative.
The second descripti on, e
1
, tells us the number of heads and
tails in the result and gives the stopping rule that governed the
experiment . We can use thi s directly in Bayes's theorem to calcu-
. d' .. I p(el e) p '(JII S' I
late the postenor Istnbutlon p(e e) = - -- - (,/' Lnee t le
pre)
BAYESIAN INDUCTION STATISTICAL THEORIES 249
number of times the coin was flipped was predetermined accord-
ing to thc stopping rule, the likelihood function and the probabil-
ity of the evidence are given in familiar fashion by thc formulas:
I
P(e21 8) = "C,W( I - 8)" . rand pee) = I"C,W( 1- 8)"- 'P( 8)d8
o
where r denotes the number of heads and n the number of coin
tosses, which in our present example are 6 and 10, respectively.
The binomial factor "C" being a function of rand n only, is inde-
pendent of 8, and so cancels out of Bayes's theorem. Clearly any
other description of the experimental outcome for which P( e
2
I 8) =
KW (l - 8)" , yields the same posterior distribution, provided K
is independent of 8. This is the case with e
l
, which states a differ-
ent stopping rule, and with e
4
, which lists thc precise sequence of
heads and tails in the outcome. In the former case, K = "IC,_ I'
as we showcd in 5.d, and in the latter K = I. Hence, the Bayesian
conclusion is unaffected by whether the experimenter intended to
stop after n tosses of the coin or when r heads appeared in the
sample or, indeed, whether the experiment was performed without
a stopping rule. As we noted in Chapter 5, this is not so in the clas-
sical scheme.
The next experimental description, c" illustrates a case where
the rule to stop the trial depends on some external event rather
than on any feature of the sample. If I states that the experiment
was designed to stop as soon as lunch was ready, then c, is equiv-
alent to the conjunction I & e
4
. So, in applying Bayes's theorem,
we need to consider the probabilities P(l & e 4 I 8), which can also
be cxprcsscd as P(ll e
4
& 8 )P(e41 8). If I is probabilistically inde-
pendent of 8 in the presence of e 4' as wc havc argucd that it is,
then P(e
5
1 8) has the form KW(l - 8)" " where K is a constant,
relative to 8. And so, as we observed above, this constant cancels
from Bayes's theorem, which then delivers the same posterior dis-
tribution for c, as for e 4' implying that the stopping rule is induc-
tively irrelevant. It clearly should be, though the classical
philosophy denies this, as we showed in Chapter 5.
Data arc normally presented without mention of the stopping
rule, as in en' which merely reports the number of heads and tails
produced in the trial. This poses a problem, for the probability of
such a result does depend on the stopping rule, and if pre I 8)
250 CHAPTER 8
cannot be computed neither Bayes's theorem, nor a classical test
of significance can be applied. The difficulty is usually met by
calcul ating the probabiliti es as !fthe sampl e size had been fi xed
in advance. In the present case, thi s means that pre I 8) would be
reckoned as equal to "C/j' (I - BY - '. Thi s seems arbitrary and
wrong, but is in fact justi fied in a Bayesian (though not a classi-
cal ) analysis, for the result r heads, n - r tails must have occurred
as some particular sequence, and whatever that sequence was, its
probability, conditional on B, is 8' ( 1 - e)" ', and we may correct-
ly use this as the input for Bayes's theorem. But as we pointed out
before, we obtain thereby the same posterior distribution for e as
we do when the calculati on is based on the possibly incorrect
assumption of a f ixed-sample stopping rule.
We have argued that the stopping rul e is irrel evant in any
inductive inference, and the Bayesian endorsement of this view
and the classical denial of it seem to us decisi ve arguments in
favour of the one and against the other. But before resting our
case, we should visit a couple of arguments that may appear to
show that in fact the Bayes ian is wrong to take no account of the
stopping rule.
To illustrate the first, consider the stopping rule we mentioned
earlier, in which the trial is planned to stop as soon as lunch is
ready, and suppose the purpose of the trial is to use a random sam-
ple to estimate the mean height of a group of chefs who are
preparing the meal while the trial is progressing. Now if tall chefs
cook faster than short ones, the time taken before the random
sampling stops depends on the unknown parameter and so con-
tains relevant information about it, which should be reflected in
any Bayesian analysis. However, despite a possible initial impres-
sion to the contrary, thi s does not endorse the classical position
concerning the stopping rule, for it does not impl y that the exper-
imenter's subjecti ve intention or pl an to stop the trial at a parti cu-
lar point had any inductive signifi cance; all that was relevant in
this contrived and unusual case was the obj ective coincidence of
lunch being ready at the same moment as the experiment stopped.
A second argument that is made from time to time is thi s: a
Bayesian who fails to announce in advance the circumstances
under which a trial will be stopped could decide to continue to
sample for as long as it takes to reach any desired posterior proh-
BAYESIAN INDUCTION STATISTICAL THEORIES 251
ability for any hypothesis, however remote from the truth it might
be. And according to Mayo (1996, pp. 352-7), for example, the
determined Bayesian will assuredly succeed in this eventually,
that is, in the limit as the sampling is repeated to infinity, or, as
Mayo puts it, by "going on long enough".4 Mayo suggests that this
putative property allows the Bayesian to "mislead", and vitiates
any Bayesian estimate.
But in fact, neither Mayo nor anyone else has demonstrated
that Bayesian conclusions drawn from the results of such try-and-
try-again experiments would be misleading, or spurious, or in any
way wrong. And unless such demonstrations are forthcoming,
there is no case whatever for the Bayesian to answer.
Moreover, the main premise of the objection is wrong, for, as
Savage (1962) proved, the application of Bayes's theorem does not
guarantee supportive evidence for any hypothesis, however long
the sampling is continued, and in fact, as Kadane et al. (1999,
Chapters 3.7 and 3.8) have shown, the probability of eventually
obtaining strongly supportive evidence for a false theory is small.
How small this probability is depends on the prior probability of
the hypothesis and on the degree of confirmation sought for it.
Sufficiency
The Bayesian concept of relevant information is that E is relevant
to 8 if and only if the posterior distribution of 8 given E is differ-
ent from its prior distribution. Of the items of evidence ~ to er, in
the above table, the last contains the least amount of information
and yet, as we showed, it is just as informative about 8 as the oth-
ers. So the information about the various stopping rules in those
pieces of evidence is, in a Bayesian sense, inductively irrelevant.
Classical statistics employs the idea of relevant information too,
approaching it through its concept of sufficient statistics. It will
be recalled from Chapter 5 that a statistic t is defined as sufficient
for 8, relative to data x, when P(x I 1) is independent of 8, this
being interpreted as t containing all the information in x that is
4 Of course, in practical reality, the sampling could never be continued very long
at all, let alone be taken to the limit.
252 CHAPTER 8
relevant to e. Here, x and t are random variables, and the suffi-
ciency definition refers to all the possible values they take in the
experimental outcome space. Since the latter is determined by the
stopping rule, the classical requirement that inferential statistics
be sufficient underlines the centrality of the stopping rule in clas-
sical inference.
We argued that e
4
: the sequence TTHTHHHHTH and en: 6
heads. 4 tails contain the same information about fl. Yet if the
number of coin tosses were 10,000, rather than 10, the same argu-
ment might not hold. For if some pattern were detected in the
sequence, for example, if all the heads preceded all the tails, or if
the first half of the sequence showed a large preponderance of
tails and the second half of heads, there would be evidence that
the coin 's centre of gravity had shifted, which would be missed if
one relied simply on the number of heads and tails. The lesson
here is that the scientist should examine evidence in as much
detail as is feasible, so as not to overlook significant information.
The possibility that the order of the heads and tails might be evi-
dentially significant did not emerge in our earlier discussion,
because we, in effect, treated such variant hypotheses as having
zero probability. But this was a simplification and in practice, the
open-minded scientist would not usually take this most extreme
attitude to unlikely hypotheses.
B.d ! Sampling
In order to derive a posterior distribution for a parameter from
sample data, one needs to compute the likelihood terms that fig-
ure in Bayes's theorem. Likelihoods are also required in classical
inferences. And since these inferences are supposed to be objec-
tive, the likelihoods they employ must also be, to which end clas-
sical statisticians generally call for experimental samples to be
created by means of a physical randomi zing device, in order to
ensure that every element is endowed with precisely the same
objective probability of being included in the sample. This means
that a sample drawn haphazardly, or purposively, in the manner
we described in 5.g, would be uninformative and unacceptable for
the purpose ofa classical estimation, while the very same sample,
BAYESIAN INDUCTION STATISTICAL THEORIES 253
had it been generated by a random process, would have been per-
fectly alright. Thi s contrast sounds paradoxical, and indeed, Stuart
(1962, p. 12) described it as a " paradox of sampling" and as a bit-
ter pi II that must be swallowed, in the interests of what he regard-
ed as the only valid estimation methods, namely those of classical
inference.
In fact, Bayesi an inference is also sensitive to the sampling
method, and as we show, there is nothing paradoxical about this,
since the way that the data were collected may indeed carry use-
ful information. Suppose, for example, that in order to estimate
the proportion fJ of As in a population, a sample of 1 (to cut the
argument to the bare bones) was drawn and that thi s was an A; and
suppose that the selected element al so possesses some other char-
acteri stic, B. The goal might be to discover what proportion of the
population intends to vote for a particular political party, and the
sample might consist of an individual who does so intend, and
who is noted to be above the age of 60. The posterior distribution
in the light of this information is given by
P(fJ lAB)
P( fJ)
P(AB I fJ)
P(AB)
peA I fJB) PCB I fJ )
x
peA I B) P(B) '
The sample could have been gathered in a number of ways. For
example, it might be the result of a random draw from the entire
population, or alternat ively, from just that porti on of the popula-
tion containing Bs. If the latter, then P(B) = P(B I fJ) = 1. If the
former, the equality between P(B) and P(B I fJ) holds only if Band
fJ are probabilistically independent, in which event, the two sam-
pling methods lead to the same Bayesian conclusion. But this is a
particular case, and as a general rule the inductive force of a given
sampl e is not independent of the selection procedure. (See S.d
above, and Korb 1994.)
This is intuitively ri ght, we suggest, because the number of Bs
contained in a random sample that was collected with the pur-
pose of estimating fJ is also a measure of the overall proportion
of Bs. If the latter is probabilistically dependent on fJ, it will con-
vey some informati on about that parameter too. On the other
hand, if you deliberately restricted the sampl e to Bs, the fact that
it contains only elements of that type would carry practically no
254 CHAPTER 8
information about their frequency in the population; hence, that
potential source of knowledge about e would be unavailable.
Bayesian and classical positions agree then that the sampling
method as well as the sample is inductively relevant. But they do
not agree on the role of random samples. The classical position is
that only random samples that are created using a physical ran-
domizing mechanism can be infonnative, making the inconven-
ient demand that other sorts of sample should not be used.
Bayesian induction, on the other hand can operate satisfactorily
using what we described in Chapter 5 as purposive or judgment
sampling.
B.e i Testing Causal Hypotheses
The most widely used statistical methodology for testing and
evaluating causal hypotheses, particularly in medical and agricul-
tural trials, was invented by Fisher and is rooted in classical pro-
cedures of inference. It will be recalled from our discussion in
Chapter 6 that the novelty of Fisher's approach was to require a
process known as randomization. We have argued that despite the
weight of opinion that regards it as a sine qua non, the randomiz-
ing of treatments in a trial does not do the job expected of it and
moreover, that in the medical context, it can be unnecessary,
inconvenient, and unethical. We present here (following Urbach
1993) an outline ofa Bayesian way of testing causal hypotheses,
which is soundly based and we believe, intuitively more satisfac-
tory than the Fisherian method.
Consider the matter through a medical example. Suppose a
new drug were discovered which, because of its structural similar-
ity to an established cure for depression, seems likely also to be
an effective treatment for that condition. Or suppose that the drug
had given encouraging results in a pilot study involving a small
number of patients. (As we said earlier, without some indication
that the drug is likely to be effective, a large-scale trial of an
experimental drug or treatment is indefensible, either economical-
ly or ethically.) A trial to test the drug's efficacy would normally
take the following form. Two groups of sufferers would be consti-
tuted. One of them, the test group, would receive the drug, while
BAYESIAN INDUCTION: STATISTICAL THEORIES 255
the other, the control group would not. In practice, the experiment
would be more elaborate than this; for example, the control group
would receive a placebo, that is, a substance which appears indis-
tinguishable from the test drug but lacks any relevant pharmaco-
logically activity; and patients would not know whether they were
part of the drug group or the placebo group. Moreover, a fastidi-
ously conducted trial would ensure that the doctor too is unaware
of whether he or she is administering the drug or the placebo (tri-
als performed with this restriction are said to be double blind).
Further precautions might also be taken to ensure that any other
factors thought likely by medical experts to influence recovery
were equally represented in each of the groups. It is then said that
these factors have been controlled for.
Why such a complicated experiment? Well, the reason for a
comparison, or control group is obvious. We are interested in the
causal effect of the drug on the chance of recovery; so we need to
know not only how patients react when they are given the drug,
but also how they would respond in its absence, and the condi-
tions in the comparison group are intended to simulate the latter
circumstance. The requirement to match or control the groups, in
certain respects, is also intuitive; but although always insisted
upon, it is never derived from epistemological principles in the
standard, classical expositions. However, selective controls in
clinical trials do have a rational basis. It is provided by Bayes's
theorem, as we shall now explain.
Clinical Trials: a Bayesian Analysis
To simplify our exposition, we will consider a particular trial in
which, for illustration, 80 percent of a specified number of test-
group patients have recovered from the disease in question, while
the recovery rate in the control group is 40 percent: call this the
evidence, e. Ideally, we would like to be able to conclude that
these percentages also approximate the probabilities of recovery
for similar people outside the trial. This hypothesis, which is rep-
resented below as H", says that the physical probability of recov-
ery (R) for a person who has received the drug is around 0.80, and
for someone who has not received the drug it is around 0.40,
256 CHAPTER 8
provided they also satisfy certain conditions, L, M and N, say.
These conditions, or what we earlier called prognostic factors,
might, for example, specify that the patient's age falls in a certain
range, that he or she has reached a certain stage of the illness, and
so forth. (In the formulations below, Drug and denote the
conditions of the drug's presence and absence, respecti vel y.) For
H to explain e, we also need to be able to assert that the condi-
a
tions L, M and N were satisfied by the subjects in both of the
experimental groups, and that the test group received the drug,
while the control group did not; this is the content of So the
hypothesis claiming that the drug caused the observed discrepan-
cy in recovery rates is the combination H" & HI]' which we label
H and call the drug hypothesis:
(H)
H,,: P(R I L, M, N, Drug) = 0.80 & P(R I L, M, N,
= 0.40
HI]: Patients in the experimental groups satisfy conditions L,
M and N.
But the drug hypothesis is not the only one capable of explain-
ing the experimental findings. Another explanation might attrib-
ute them to a psychosomatic effect induced by a greater level of
optimism amongst the test group patients than amongst those in
the control group. Let H;, signify the hypothesis that under con-
ditions L, M and N, the drug has no effect on the course of the dis-
ease, but that an optimistic attitude (0) promotes recovery. By
parallel reasoning, the hypothesis that explains the evidence as a
psychosomatic, confidence effect is the combination of H;, and
H;], whieh we label H':
H ;,: P(R I L, M, N, 0) = 0.80 & P(R I L, M, N, = 0.40
(H')
H;, : Test group patients satisfy L, M, N, 0; control group
patients satisfy L, M, N,
BAYESIAN INDUCTION: STATISTICAL THEORIES 257
The Bayesian method addresses itself to the probabilities of
hypotheses and to how those probabilities are updated via Bayes's
theorem in the light of new information. Let us sec how this
updating works in the present case. To start with, and for the pur-
pose of exposition, we shall suppose that Hand H' are the only
hypotheses with any chance of being true. In that event, Bayes's
theorem takes the following form:
1
P(H I e) = --pre I HjP(H')'
1 + -----
pre I H)P(H)
The question is how to design the experiment so as to maximize
the probability of the drug hypothesis for some given e. We see
from the above equation that the only components that can be
manipulated to this end arc P(H) and P(H'), and that the smaller
the latter and the larger the former, the better. Now, if the two
components of the hypotheses arc independent-a reasonable
premi se, which also simplifies our argument-then P(H) =
P(H,, )P(H
IJ
) and P(H') = The goal of minimizing
P(H ' ) now comes down to that of minimizing And again,
since only P(H I)) can be changed by adjusting the experimental
conditions, it is this component of P(H) that should be maximized.
Now H 't! states, amongst other things, that patients in the test
group were confident of recovery, while those in the control group
were not. We can reduce the probability of this being the case-
reduce that is-by applying a placebo to the control
group; for if the placebo is well designed, patients will have no
idea which experimental group they were in, so that a number of
factors that would otherwise create different expectations of
recovery in the two groups would be absent. In many cases, the
probability of H 'I) could be reduced further by ensuring that even
the doctors involved in the trial cannot distinguish the treatment
from the placebo; such trials are, as we said earlier, called 'dou-
ble blind' . By diminishing P(H'/)) in these various ways, the prob-
ability of H, for a given e, is increased.
The drug hypothesis would, as we pointed out, also be made
more probable by adopting appropriate measures to raise P(H
IJ
).
that is to say, measures that increase the chance that the factors
258 CHAPTER 8
which the drug hypothesis says are relevant to recovery (namely,
L, M and N) are represented equally in the two groups. Now sup-
pose one of those factors was virulence of the disease, or the
strength of the patient, or the like, which may be hard to measure
but which doctors may well be able, through long experience, to
intuit. And suppose we left it to these doctors to construct the
comparison groups. It would not he surprising if the resulting
groups contained unequal numbers of the more vulnerable
patients. For doctors should be and generally are guided by the
wish to secure the best treatment for their patients, in accordance
with the Hippocratic Oath,5 and if they also entertained prior
views on the trial treatment's efficacy, they would be inclined to
distribute patients among the trial groups according to particular
medical needs, rather than with impartiality. And there may be
other unconscious or even conscious motivations amongst exper-
imenters that could lead them to create test groups with an inbuilt
bias either in favour or against the test treatment. This is where
randomized allocation can playa rol e, for it is a mechanism that
excludes the doctor's feelings and thoughts from the process of
forming the experimental groups and thereby makes balanced
groups more probable.
We have simplifi ed this exposition by considering a single
alternative to the drug hypothesis. There will normally be many
alternatives, each contributing an element to the denominator,
which would then include as summands other terms of the form:
Pee I H")P(H") = Pee I H",, &H'{;) P(H",)P(H"I)- again, assum-
ing independence. Each of these terms relates to specific, hypo-
thetical prognostic factors, and each needs to be minimized in
order to maximi ze the posterior probability of H, the drug hypoth-
, Hippocrates (di ed 380 360 H.C.) was the author of the eponymous oath by
which doctors committed themselves to certai n professional standards.
Graduates ill some medical schools today formall y take a modernized version of
that oath. The classical formulation enjoins doctors, amongst other things, to
keep their patients " from harm and injusti ce". A form that is currentl y approved
by the American Medical Association makes the doctor promise "that you will
exercise your profcssion solely for the curc of your patients".
(, Kadane and Seidcnfeld (1990) cite the possibiliti es of bias arising because the
experimenter was thc inventor of the test treatment, with a personal stake in its
success, or was kecn to make a splash in the acadcmic literature by announcing
a surprising result .
BAYESIAN INDUCTION: STATISTICAL THEORIES 259
esis; as we explained, this is done is by matching the experimen-
tal groups in ways that are suited to reducing
But not every conceivable factor can be matched across the
groups; nor is a comprehensive matching in any sense either need-
ed or desirable. For, consider a possible prognostic factor whose
causal influence is described in the hypothesis and suppose
that hypothesis to be very improbable; in that event, the whole of
the corresponding term in Bayes's theorem would already be
extremely small , and so the advantage of reducing it further by
applying appropriate controls would be correspondingly small.
And in most cases, that small advantage would be outweighed by
the extra cost and inconvenience of the more elaborate trial. For
example, though one could introduce a control to reduce the prob-
ability that the clinicians treating the two groups had different
average shoe sizes, such a precaution would only negligibly affect
the posterior probability of the drug hypothesis, since shoe size is
so immensely unlikely to int1uence the outcome of the trial.
In summary, the Bayesian theory explains why the experimen-
tal groups in clinical trials need to be matched in certain respects,
and why they need not be matched in every respect. It agrees with
common sense in affirming that the chielconcern when designing
a clinical trial should he to make it that the experimental
groups differ 011 fClctors that are like Iv to affect the outcome.
Designs that achieve such a balance between groups are termed
"haphazard" by Lindley, 1982a, p. 439, though we prefer to think
of them as adequately matched or controlled.
With this rule in mind, it is evident that a randomized alloca-
tion of subjects to treatments might sometimes be useful in clini-
cal trials as a way of better balancing the experimental groups,
insofar as it stops those making the allocation from doing so on
the basis, for example, of how sick the patient is. But randomized
allocation is not absolutely necessQ/}'; it is no sine qua non; it is
110t the only or even alHYIYs the best way olconstructing the treat-
ment groups in a clinical trial.
Clinical Trials without Randomization
Bayesian and classical prescriptions for clinical trials differ in two
respects of practical importance. First, a Bayesian analysis per-
260 CHAPTER 8
mits the continuous evaluation of results as they accumulate,
thus allowing a trial to be halted as soon as the effectiveness or
otherwise of the experimental treatment becomes apparcnt. It
will be recalled that, by contrast, when classical principles are
strictly observed, a clinical tri al that has been brought to an
unscheduled stop cannot even be interpreted, whatever the
results recordcd at that stage. And sequential clinical trials,
which were speci all y invented in order to allow interim analyses
of trial results using the classical techniques are ineffective, as
we argued in Chapter 6.
The second di fference relates to the formation of the com-
pari son groups in a clinical trial. On the Bayesian view, thc essen-
tial desideratum is that the groups be adequately matched on
likely prognostic factors. Freed from the absolute need to random-
ize, Bayesian principles affirm, what its rival denies, that decisive
information about a medical treatment can be obtained from trials
that use, for example, a histori cal control group, that is to say, a
number of patients who have already received an alternative ther-
apy or no therapy at all. Such groups may be constructed by
means of informati on derived from medical records and, provided
they are adequately matched on prognostic factors with a test
group (which may also be of the historical type), an informative
comparison is allowed.
Historically controlled trial s are widely dismissed as inher-
ently fallacious by classically minded medical statisticians, who
regard concurrent, randomized trials as the "only way to assess
new drugs" (Mc Int yre 1991). But this view is not only indefen-
sible on epistemic grounds, it is daily refuted in medical prac-
tice, where doctors' knowledge and expertise is accumulated in
part by continual , informal comparisons of current and former
pati ents.
Even some classical statisticians (such as Byar et al. 1990)
concede that historical compari son groups may be preferabl e in
certain circumstances, particul arly where a randomized trial
would expose critically ill pati ents in a control group to a useless
placebo, while others are given a promising, experimental treat-
ment. Byar and h is colleagues claimed that the standard trial
structure should be suspended when there is "a justifiable expec-
tation that the potential benefit to the patient will be sufficiently
BAYESIAN INDUCTION STATISTICAL THEORIES 261
large to make interpretation of the results of a non-randomized
trial unambiguous". But they do not say what process of inference
could deliver the unambiguous interpretation of results. No clas-
sical significance test can be intended, since Byar makes the
familiar classical claim, which we discussed in Chapter 6, that "it
is the process of randomization that generates the significance
test". Nor does any other tool of classical inference seem to be
available. It is more likely that Byar et at. are here allowing them-
selves an informal Bayesian interpretation, for they say that a his-
torically controlled tri al is acceptable only if the trial treatment
has a strong prior credibility: "the scientific rationale for the
[tri al] treatment must be sufficiently strong that a positive result
would be widely expected".
The admi ss ibility in principl e of historically controlled trials is
not just an interesting theoretical impli cation of the Bayesian
view, but is of considerable practical significance too. Firstly,
such trials call for fewer new subjects, which is of particular
importance when rare medical conditions arc the subject of study.
And smaller trials are generally cheaper. Secondly, historical
comparison groups do not expose subjects to ineffective pl acebos
or to what clinicians expect will turn out to be inferior compari-
son treatments, considerations that address natural ethical con-
cerns, and mitigate the reluctance commonly found amongst
pati ents to participate in trial s.
Historically controlled trial s are however, not easy to set up.
The compari son groups can only be formed with the aid of thor-
ough medical records of past patients, more detailed than the
records that are routindy kept, and more accessible. To thi s end,
Berry 1989 proposed the establi shment of national databases con-
taining patients ' characteristics, diagnoses, treatments and out-
comes, which could be open to the public and contributed to by
every doctor. Some modest work along these lines has been done.
Unfortunately, the widely held, though erroneous opinion that his-
torical controls are intrinsically unacceptable or impossible has
discouraged efforts to overcome the purely practical obstacles that
stand in the way of effective hi storically controlled trials. Here is
a case where the mistaken principles of classical methodology are
harmful.
262 CHAPTER 8
Summary
Bayes's theorem supplies coherent and intuitive guidelines for
clini cal and similar trials, which contrast signi f icantly with clas-
sical ones. One striking difference between the two approaches is
that the second simply takes the need for controls for granted,
whil e the first explains that need and, moreover, distinguishes in
a plausible way between factors that have to be controlled and
those that do not. Another difference is that the Bayesian approach
does not make the random allocation of subj ects to treatments a
universal , absolute requirement. We regard thi s as a considerable
merit, since we have discovered no good reason for regarding a
random allocation as indispensable and several good reasons for
not so regarding it.
8.f 1 Conclusion
The Bayesian way of estimating parameters associates different
degrees of confidence with different values and ranges of values,
as cl assical stati sti cians sought to do, though as we have seen,
they were unsuccessful in this. It makes such estimates through a
single principle, Bayes's theorem, which appli es to all inferential
probl ems. Hence, the Bayesian treatment of statistical and deter-
mini stic theories is the same and is underwritten by the same
phil osophical idea.
Bayesian estimation also chimes in well with our intuitions. It
accounts for the intuitively plausible suffici ency condition and
for the natural preference for max imum preci sion in estimators,
while finding no pl ace for the criteria of ' unbiasedness' and
'consistency' , whi ch, we have argued, are based on error. The
Bayesian method also avoids a perverse feature of classical meth-
ods, namely, a dependence on the outcome space and hence on
the sUbjective stopping rule.
There is a subj ecti ve element in the Bayesian approach which
offends some, but which, we submit, is wholly realistic. Perfectly
sane scientists with access to the same informat ion often do eval-
uate theories differently. Newton and Leibniz differed sharpl y on
gravitational theory; Einstein's opinion of Quantum theory in its
BAYESIAN INDUCTION: STATISTICAL THEORIES 263
Copenhagen interpretation was at variance with that of most of
his colleagues; for many years, distinguished astronomers defend-
ed the Big Bang theory, while equally distinguished colleagues
defended with equal vigour the competing Steady State theory,
and similar divergences of view continue to occur in every branch
of physical science, in medicine, biology and psychology.
Bayesian theory anticipates that many such divergences will be
resolved as probabilities are revised through new evidence, but it
also allows for the possibility of people whose predispositions
either for or against certain theories arc so pronounced and dis-
tinct from the norm that their opinions remain eccentric, even
after a large mass of relevant data has accumulated. You might
take the view that such eccentrics are pathological, that every the-
ory has a single value relative to a given body of knowledge, and
that responsible scientists ought not to let personal, subjective fac-
tors influence their beliefs. But then you would have to face the
fact that this view is itself a prej udice, because despite an
immense intellectual effort, no one has produced a coherent
defence of it, let alone a proof.
After decades when Bayesian ideas and methods were
despised and rejected, their merits arc now coming to be widely
acknowledged, even by orthodox statisticians, so much so that
many are willing to put Bayesianism and Frequentistism on a par.
But this is not an outcome that we regard as satisfactory. Our
arguments have been that the first is well founded and the second
not, and that the second should give way to the first. Blasco
(2001 , p. 2042) provides an example of the even-handedness that
we reject. Writing on "The Bayesian controversy in animal breed-
ing", he argues that in practice there is no need to take sides:
If the animal breeder is not interested in the philosophical problems
associated with induction. but in tools to solve problems, both
Bayesian and frequentist schools of inference are well established
and it is not necessary to justify why one or the other school is pre-
ferred. Neither of them now has operational difficulties. with the
exception of some complex cases .... To choose one school or the
other should bc related to whether there are solutions in one school
that the other does not offer. to how easily the problems are solved
and to how comfortable the scientist feels with the particular way of
expressing the results.
264 CHAPTER 8
But we have argued that what Blasco call s "the philosophical
problems associated with induction" ought to be of interest to the
practical scientist, for although frequentist and Bayesian tools
give superficially similar results, and recommend superfi cially
similar trials in some cases, they also make crucially different rec-
ommendations in others. We have shown that frequentist tools do
not solve any problems. The conclusions they license ("the best
estimate of thc unknown parameter, e, is such and such"; "such
and such is a 99 percent confidence interval for fJ"; "ho is reject-
ed at the 5 percent level", etc.) have no inductive significance
whatever. True, one can often draw frequenti st conclusions "eas i-
ly", but this is of no account and does not render them sci entifi-
cally meaningful. True, many scientists feel "comfortable" with
frequentist results, but this, we suggest, is because they are misin-
terpreting them and endowing them with a meaning they cannot
possibly bear.
CHAPTER 9
Finale: Some General Issues
9.0 ! The Charge of Subjectivism
The theory of inductive inference, or inductive logic as we feel
entitled to call it, that we have presented here is sometimes called
the subj ecti ve Bayesian theory, 'subjecti ve' primarily because it
imposes no constraints on the form of the prior probabilities in
Bayes's Theorem calculations. Many both inside and outside the
Bayesian camp think this results in a theory inadequate to its pre-
sumed purpose of furnishing an objective account of inductive
inference, since these priors will necessarily have to be 'subj ec-
tive', and subj ectivity should have no pl ace in scicntific inference.
Those who have read the earlier chapters will recall that our
view is that all thi s is incorrect. Firstly, some subjective element
exists in all scientific appraisal and it is a merit of this theory that
where it occurs it is signalled explicitly, not concealed from view.
Secondly, what we have in thi s theory is in fact a perfectly objec-
tive logic of inductive inference, whose ' premi ses' can be just
those pri or probabi lities, with Bayes's Theorem as the inference
engine generating a valid conclusion: the posterior distribution.
As we pointed out, the situation is wholly analogous to deductive
logic, where the logic is the inference engi ne: you choose the
premi ses, and the engine generates the valid conclusions from
them. De Finetti characteristically puts the position very well:
We stri ve to make judgments as di spassionate, refl ecti ve and wisc as
possibl e by a doctrine that shows where and how they intcrvcnc and
lays bare possible inconsistencics bctwcen judgments. There is an
instruct ive analogy bctween [deduct ive] logic, which convinces one
that acceptance of some subjective opini ons as 'certain' entails the
certainty of others, and the theory of subjective probabilities, whi ch
similarly connects uncertain opinions. ( 1972, p. 144)
266 C HAPTER 9
Nevertheless, it cannot be proved that there are no further accept-
able constraints which we have simply missed or ignored which
might substantially reduce or even in suitable cases eliminate the
degrees of freedom in the selection of prior distributions. Indeed,
the history of the Bayesian theory is to a considerable extent the
history of attempts to find such constraints, and we shall end by
looking at the principal ones.
9.0.1 The Principle of Indifference
A constraint additional to those we presented in Chapter 3, and
defined consistency with respect to, has been of great historical
importance, and even today we still find its claim to inclusion
strongly pressed. Called by Keynes the Principle of Indifference,
it is a prima facie highly plausible symmetry principle enjoining
that a symmetric relationship between the members of a partition
should be respected by a correspondingly symmetric a priori dis-
tribution of probabilities. More precisely: equal parts of the pos-
sibili(v space should receive eqlla! probabilities relative to a null
state olbackground inlormation.
Not only did this principle, and to some people it still does,
seem intuitively compelling: it also turned out to have impressive
methodological power (as we shall see, more so even than its
advocates immediately realised). Thomas Bayes was the first to
use it to prove a deep and important result in statistics, the so-
called' inversion' of Bernoulli's Theorem. Recall from Chapter 2
that James Bernoulli had proved a fundamental theorem about , in
modern notation, a possibility space Q consisting of n-fold
sequences s of Os and I s, a class of events defined in Q including
all events of the form 'there is a I at the ith index' (also describ-
able in the language of random variables by a formula 'X[= I ',
where the Xi are 11 {O, Il-valued random variables defined on Q),
and a probability function assigning objective probabilities to
these events such that 'Xi = I' has the same probability p inde-
pendently of i, and the events 'XI = x I', ...... , 'X:, = XII', Xi = 0 or
I, are independent. Bernoulli's result was that for any small E > 0,
the probability that the absolute value of the difference between
the relative frequency (I1
J
)1:X
i
of Is (up to n) andp is less than E
tends to I as n tends to infinity.
FINALE SOME GENERAL ISSUES 267
People, including Bernoulli himself, wanted to interpret this
result as licensing an inference that the probability is very large,
for large enough 11 , that p is close to the observed relative fre-
quency. Unfortunately, no such inference is licensed. Bernoulli's
Theorem is a result about a probability distribution charac-
terised by a real-valued parameter p: hence p is not itself not a
random variable to which probabilites are attached. To 'invert'
Bernoulli's theorem requires defining a more extensive possibil-
ity space Q' , a class C of propositions defined in Q', and a prob-
ability function P defined on C, in which 'p = r S is an event
in C, i.e. in which p is a random variable.
This is just what Bayes attempted to do, and to a great extent
succeeded in doing, and thereby came up with the first rigorous-
ly obtained posterior distribution for a statistical hypothesis, in
this case about the value of p, now explicitly considered as a ran-
dom variable. Here is a brief sketch, in more modern notation than
Bayes used, of how to perform the calculation. In fact, it is a fair-
ly straightforward application of Bayes 's Theorem, in which the
I ikel ihood, i.e. the probabi I ity of the data, r 1 sand 11 - r Os, con-
ditional on p, is given by the function nc, p'(1 - p)" -', and the
prior probability of p determined by the Principle of Indifference
as the uniform distribution with density lover the unit interval
[0,1] (in othcr words, it is simply assumed that the prior probabil-
ity of the Bernoulli model, of constant probability and independ-
ence, is I; Bayes's own example was carefully crafted so that the
assumption is not made explicit). Thus the conditional probabili-
ty density is, by Bayes's Theorem, proportional to
f(p I I) = "C,p'(1 - p)" ,
But since f(p I r) is a continuous probability density it must inte-
grate to 1. Hence we must have
I
)(X
i
= r) = II C) p'(1 - p)" 'dp
"
p' (\ _ 17)" r
f(p I r) = '- 1 - ---
~ p r l _ p)n Idp
268 CHAPTER 9
We have been here before (in Chapter 3). This density is a beta-
density with parameters r, n - r (see Chapter 8), whose mean is
(r + \) / (i1 + 2) , which obviously tends to r/n. The variance is (r
+ \)(/1 - r + 1) / (/1 + 2) 2(/1 + 3), which tends to 0 as n increas-
es, and so the posterior probability of p lying in an arbitrarily
small interval around r / 11 tends to 1. Bayes seemed to have done
what Bernoulli 's Theorem could not do: tell us that it is over-
whelmingly probable that in a large sample p, now explicitly a
random variable over the enlarged poss ibility-space including not
only the possible outcomes of the X; but also the possible values
of the binomial probability, is approximately equal to the
observed relative frequency. As a matter of historical accuracy we
should note that this was not Bayes's own method of derivation:
he used a geometrical construction, in the manner of Newton's
Principia, instead of the analytical calculation (Laplace was the
first to use the integral formulas).
Of course, Bayes 's posterior distribution is itself implictly con-
ditional upon another assumption, namely that the process is a
Bernoulli process, but since Bernoulli himself assumed much the
same thing it is unfair to charge Bayes with making additional mod-
elling assumptions. The important innovation here is the use of the
Principle of Indifference to determine the prior distribution of p,
though even the Principle itself was one that in a general way
Bernoulli also endorsed, under the name of the Principle (if'
Insufficient Reason. At any rate, in this example and in any others
in which the hypothesis-parameter takes values in a bounded inter-
val, the Principle of Indifference appears to yield a fully determi-
nate posteri or distribution. often something else too. Continue with
the Bayes example and consider the probability P(J\, .. I = I / LX; = r),
where the sum is from I to 11. By the rule for conditional probabil-
ities this is equal to the rati o P(X; = ,. + I) / P(LX; = r). where the
first sum is to 11 + 1 and the second to n. Using the reasoning above
we can see that this ratio is equal to
f pi , 1 (I - p )" - 'dp (I)
"
,
f pl( I - p)" rdp
"
which is equal to (r + 1)/ (11 + 2). (I) is historically famous; it is
known as the Rule oj" Succession. Clearly, as n grows large, the
FINALE SOME GENERAL ISSUES 269
conditional probability that the next observation will reveal a I
given that there have been r in the previous n observations tends
to the observed relative frequency r / 11. Thus, not only should you
adjust your degree of belief in a head falling at the next toss to the
observed relati ve frequency of heads; thi s result tells you exactly
how you should adjust it.
The Rule of Succession was also put to less mundane uses.
Keynes, in a memorable passage, wrote that
no other formula in the alchemy of logic [logic, note] has exerted
more astonishing powers. For it has established the existence of God
from total ignorance, and it has measured with numerical precision
the probability that the sun will rise tomorrow. (Keynes 1921, p. 89)
Indeed, it was Laplace himself who used the Rule to compute the
odds on the sun's rising the next day, given that it had risen
1,826,213 days (by Laplace's calculation) days, to be 1,826,214 to
one. Soon the Rule was seen as a justification for general enumer-
ative induction, adjusting your confidence in any future event
according to the frequency with which you have observed it in the
past. The problem of induction, which Hume had declared with
the most telling arguments (see Howson 2000 for an extended dis-
cussion) to be insoluble, seemed half a century later to be solved.
Hume had argued that all ' probable arguments' from past obser-
vations to future predictions are necessarily circular, presuppos-
ing what they set out to provc. In appearing to show that all that
was required was, on the contrary, a complete lack of commitment
between the possible alternatives, the Principle of Indifference
looked like the answer to Humean sccpticism. Indeed, this is
exactly how Bayes's result was seen by his executor and friend
Richard Price, who argued in his introduction to Bayes's Memoir
that it contained the answer to Hume's sceptical arguments.
Nemesis took some time to arrive, but arrive it eventually did,
in the form of an increasing level of criticism and simple recipes
for generating apparently paradoxical results from the Principle.
Here is a modern example. It is expressed in the context of for-
malised logical language. The purpose of this is to show that the
underlying problem is not, as it is often held to be (this standard
defence was first resorted to by Keynes), the fault of imprecise
definition. Consider two languages L / and L 2 in the usual modern
270 CHAPTER 9
logical symbolism, with the identity symbol = included, as is
usual, as a logical constant in both languages. Suppose L, and L,
possess just one predicate symbol Q, the difference between the
two languages being that L, has two individual names, symbolise
them a and b, while Lo has none. Now it is a remarkable fact that
the propositions
Sf: There is at least one individual having Q
S,: There are exactly two individuals
can be formulated by exactly the same formal sentences in each
language. At their simplest, these are
S',: 3xQ(r)
S'c: 3x3y[('( '" y) & Itz(z = x v z = y)].
Suppose our background information is conveyed by S If we
stipulate that a and b name distinct individuals there are four
L ,-atomic possibilities given S 'c' viz both have Q, a has Q and b
doesn't, b has Q and a doesn't, neither has Q). In L c there are just
three, since with respect to L, the individuals are indistinguish-
able. The Principle of Indifference would seem to demand that,
conditional on S'l' the probability of S', is 3/4 if we are talking in
L" but 2/3 if we are talking in L] (the stipulation that a and b name
distinct individuals is there for simplicity of calculation: the non-
identity of the probabilities persists, though the precise values dif-
fer from this example, if it is dropped).
The standard defence against this sort of example is that since
L, makes more refined discriminations, one should choose that
and not the apparently coarser-grained L l as the basis of any
application of the Principle ofindifference. The defence does not,
however, succeed, for a number of reasons. Firstly, and perhaps
surprisingly, it is simplv not trlle that a language ,vith individual
names makes .finer distinctions than one without.
Modern physics famously supplies striking counterexamples.
Thus paired bosons-particles with spin equal to one in natural
units-are indistinguishable according to quantum mechanics;
this means that the coupled system (tensor product) of two such
FINALE: SOME GENERAL ISSUES 271
particles has only three eigenstates of spin, corresponding to both
particles having spin up (in a given direction), both having spin
down, and one having spin up and the other spin down, giving
exactly the same quantum stati stics as Lc' (Sudbery 1986, pp.
70- 74; the quantum mechanical predictions for bosons and fermi-
ons, which have even more dramatic statistics, are highly con-
firmed, raising the problem of how particles can in principle seem
to be incapable of individuation, while at the same time remain-
ing a plurality. Fortunately, this is a puzzle for the metaphysics of
physics, not for us.) Less exotically, pounds in bank accounts are
also indistinguishable in the same way: it makes no sense to say
that there are four distinct ways in which two pounds can be dis-
tributed between two accounts. In these sorts of cases, therefore,
there simply is no fi ner-grained reality than that described by L l'
Secondly, the objection begs the questi on: why should one
choose the finer partition? We are interested in seeing Bayesian
probability in a logical perspect ive; indeed, we feel that it is only
in such a perspect ive that it makes sense (where, of course, it
makes a lot of sense). But logic does not dictate that one should
choose the finer partition. Logic does not dictate that one ought to
do anything at all, for that matter. But certainly not that.
Thirdly, and devastatingly, the Principle of Indifference gives
different answers for equally fine partitions. It asserts that equal
parts of a possibility-space Q should receive the same a priori
probability in the absence of any discriminating information.
Here there is implicit reference to a metric according to which
parts of Q are judged equal. Where Q is a continuum the metric
is induced by a mapping onto an interval of real numbers, with its
standard Euclidean metric (if Q is itself an interval of real num-
bers the identity map is one such map). Such spaces are of course
typical of those studied in mathematical stati stics, but they have
the property that the choice of metric can be made in di fferent , but
equivalent, ways. In fact, there is an infinity of different mappings
of Q into the real numbers such that the induced distance between
two points in Q depends on which mapping is employed. Thus
equality of distance, which the Principle of Indifference exploits
in continuous spaces, is strongly non-invariant.
Consider, for example, the space [0, I] of possible values of p
that we considered in connection with Bayes's famous result. The
272 CHAPTER 9
mappIng p p l is a continuous di fferentiable bijecti on of
[0,1] onto itself. The Principl e of Indifference applied to the
p-representati on tells us that there is a probability of 112 of p not
exceeding 1/2. But since p is vp' the probability of p not exceed-
ing 112 is, by the probability calculus, the same as the probability
of p ' not exceeding 1/4, and so that must be 1/2. But the Principle
of Indifference applied to p ' tells us that the probability that p'
does not exceed 114 is 1/4. So we have a contradi cti on, and a real
one this time. The obj ection that there is a unique ' natural ' repre-
sentation in the set of real numbers can be dism issed: firstly, it
begs the question, and secondl y, it is not even borne out in the
ordinary practice of science. As Jeffreys points out, it is not
uncommon for different representations to be used in di fferent
contexts; for example, some methods of measuring the charge on
the electron give e, others e
2
(\ 961, p. 120).
As a postscript to this discussi on of the Princi ple of
Indifference we should recall that for large sampl es the form of
the prior di stribution is not of particular importance as long as it
is not too extreme: as we saw earlier, for large random sampl es the
posterior di stribution for a parameter will typi cally converge
within a small interval of its maximum-likelihood estimatc inde-
pendentl y of the exact form of the prior, so long as the latter is not
too extreme (a proof of thi s result is gi ven in Lindley 1965, pp.
128, 129). Thus the posterior distributi on for Bayes's binomial
parameter p will be concentrated within a small interval of the
observed rel ati ve frequency (the maximum-likelihood estimator
of p ) even without assuming the uniform pri or distribution.
Whether the anxiety that Bayes is known to have fe lt about using
the Principl e of Indi fference (though he did not call it that)
mi ght have been miti gated if he had known of thi s result is not
known, but it is doubt ful: we are not all owed very often the lux-
ury of large amounts of data, and for most probl ems the form of
the prior will affect the posterior. Bayes wanted a pri or that
refl ected an even-handed ignorance, whatever the sampl e size,
and that problem, if it is a problem- we shall argue that it is
not- is not solved, though it mi ght on occasion be miti gated, by
asymptotic considerations.
FINALE SOME GENERAL ISSUES 273
9.a.2 Invariance Considerations
The Principle of Indifference is a symmetry principle stating that
logical symmetri es should be reflected, in thc absence of any dis-
criminating information, in uni fo rm a priori probability distribu-
tions. The trouble, as we have seen, is that in continuous spaces
there are too many symmetri es for anyone uniform distribution to
refl ect. Such is logical life. Unfortunately, the Bayesian theory
was historically based on the Principle of Indifference: the princi-
pl e conferred on the theory a supposed status as an objective the-
ory of inductive inference. Without it, as we can see in the case of
Bayes's deri vati on, prior probabilities occur in effect as undeter-
mined parameters in the calcul at ion of the posterior distribution.
Many Bayesians became preoccupied with the questi on of
whcther there is any way of salvaging enough of the substance of
the principle to make the enterprisc of an objecti ve probabili stic
inductive logic worthwhile.
One line of enquiry was conducted by Harold Jeffreys, who
investigated which rules, if any, did not yi eld inconsistencies in
the way the Principle of Indifference did under transformati ons of
a continuum- valued hypothes is space like the transformation
from p to p' above. Jeffreys was a physici st as well as a proba-
bili st, writing in the first half of the twenti eth century, and the
problem of finding a rule whi ch can be invariantly (more accu-
rately, covariantli) expressed in any co-ordinate system was a
familiar one in the phys ics of general relat ivity, recei vi ng its
mathematical solution in the theory of tensors. Jeffreys found
examples which seemed prima facie suitable for prior probabili-
ti es: the best-known is the rul e which can be used wherever there
is a statistical model for the observed vari ate X expresscd in a
density f(x I fJl' . .. ,8,). I f there is just one unknown parameter 8
then this rule assigns as the prior probability p(8) the function
V/(H) , or the square root of the Fisher inj(mnation
2
, which is
I /\n equat ion is covaria nt if it has the same form under all co-ordinate transfor-
mation (the terms COIWT in such a way that an identity between thcm true in one
co-ordi natc system is truc in all: tcnsor cquations famous ly have this property).
2 In thc multiparamcter case the Fisher information is the matrix (A),; = (jJ L I d8,/,
i,j = I ..... k. Jeffreys's rule assigns the square root of its determinant as the prior
probability density.
274 CHAPTER 9
defined as the expected value, with respect to X of - ()l/ () (p,
where L is the logarithm (natural or otherwise; it doesn't matter)
off(y; I e) ; note that as a result of the differentiation, the expected
value may not depend on e. By elementary calculus the densities
prep) and pre), where is some continuous bijection of e, are relat-
ed by the identity
pre) = I dep/ d8 1 prep)
Hence where the rule for assigning a prior density is given by
p (8) = $(8)
for some functional term $ (8), it is covariant just in case it satis-
fies the transformation condition
$ (8) = I dep/ d8 1 $ (ep).
It is not di fficult to show that $(ep) = v l (e) does indeed transform
in just thi s way, and hence is consistent with respect to arbitrary
coordinate transformati ons.
The Jeffreys rule is far from the onl y covariant rul e for gener-
ating prior di stributions, though it has received a good deal of
attention because some of the priors it generates have been sug-
gested independently. There are technical problems with it, how-
ever, principal among whi ch are that (a) the expectation does not
always exist, and (b) among the priors that the rule generates are
so-called improper distributions, that is to say distributi ons which
do not integrate to lover their respective ranges (- ex) to + ex; and
o to (0) as the probability axioms demand they should. Notable
examples of improper di stributions are the prior densities gener-
ated by it for the mean and standard deviation parameters of a nor-
mal model , which are a constant (uniform) and proportional to
(J I (log-uniform) respectively3. A good deal has been written
) Another feature that some Bayes ians take to be probl ematic is that the rule,
depending as it does on the entire outcome space of the experiment, vio lates the
Likelihood Principle (see S.d above).
FINALE: SOME GENERAL ISSUES 275
about the status of improper prior distributions, more than we
could or should go into here, but some brief comments are in
order. Improper distributions might be, and often are, introduced
as no more than convenient approximations to proper ones within
what is thought to be the likely probable error (the prior propor-
tional to CJ 1 is approximately that given by the log Cauchy distri-
bution), so long as due care is taken in computations involving
them to maintain overall consistency (see, for example, Lee 1997,
p.44). That is not the case here, however, where they are deduced
a priori, and where the fact that they conflict with the standard
probability axioms might suggest that the rule is an ad hoc device
meriting a certain dcgree of scepticism.
Perhaps this conclusion is premature, since it has been shown
that there is a plausible way of amending the axioms which con-
sistently accommodates improper distributions, which is to take a
conditional probability as primitive. The mathematician Alfred
Renyi and the philosopher Karl Popper at about the same time
(between 1945 and 1955) independently developed versions of
the probability calculus based on a primitive conditional probabil-
ity function, in which one can in principle have unbounded cumu-
lative distribution functions so long as the conditional
probabilities they define remain finite (that is, they can be nor-
malised) (Renyi 1955, p. 295). This development is of consider-
able interest, without doubt, even though neither Popper's nor
Renyi's axiomatisation has been generally adopted (there is per-
haps more reason to adopt Renyi's system since it is the closest
mathematically to the standard Komogorov axioms, and because
Renyi was able to prove the conditions under which his condition-
al functions arc representable as quotients of finite measures
(1955, p. 292), which is of course how they are introduced in the
standard formalism}.
However, even if there are acceptable ways of taming improp-
er distributions, that would still leave entirely open the question
why the Jeffreys rule, or any other invariant rule, should be adopt-
ed. The choice of any rule would, one would think, need to be jus-
tified, and in terms of the logical interpretation we are advocating
that means that it has to be justified as a general condition on fair
betting odds. The Principle of Indifference looked at first as if it
might have a justification of this type, but as it happened it turned
276 CHAPTER 9
out to be inconsistent. The only justificati on for Jeffreys 's rul e
that he himself gave was-apart from its demonstrable consisten-
cy under transformations of the parameters-its ability to deliver
what he considered to be independentl y desirable priors for cer-
tain parameters, in particular-grant ed the acceptability of
improper di stributi ons along the lines menti oned above-for a
normal mean and standard deviation separately (but not jointly,
since the j oint density is proportional to a 2, which is not the
product, as intuitively it should be, of that of the mean by itself
and that of the standard deviation by itself) and, to a lesser extent,
a binomial probability (JefIreys 1961, p. 182; p. 184). But in
every other respect it seems lacking, a situati on not helped by the
lack of uniqueness in its invariance property.
9.0.3 Informotionlessness
Underl ying the attractiveness of the Principl e of Indifference is
the idea that it all ows the data to .speaklor itself by not in any way
prejudging the parameter or hypothesis that the data carries infor-
mation about. Unfortunately, this vi ew is rather obviously unten-
able. A uniform prior distribution, as we have seen, is uniform
across the elements of a particul ar cl ass if icati on-scheme, for
exampl e as determined by the values of a random variable. It
therefore says that each clement of a basic partiti on determined by
this scheme is equall y probable a priori. But any other di stribution
also gives probabiliti es of these cell s a pri ori . So all prior di stri-
butions 'say' something not implied by the observational data
alone. In other words, there is no a priori di stributi on that lets the
data speak for itself.
Reinforcing thi s conclusion is the fact, noted in Section 9.a. 2,
that in continuous spaces we can redescribe the ' elementary pos-
sibilities' in terms of another, equival ent, classif ication scheme in
such a way that the ori ginal uniform di stributi on transforms into
a highly non-uniform one. Suppose that a quantity x is non-nega-
tive, for exampl e. An attitude of pri or epi stemic neutrality mi ght
seem to demand that no one possibl e value should be regarded as
more likely than any other; or in other words, that a uniform di s-
tribution is appropri ate. But of course there is no proper uniform
density on the non-negative line. Worse, a completely neutral atti-
FINALE: SOME GENERAL ISSUES 277
tude would also seem to demand even-handedness with respect to
thc possible different orders of magnitude of x, those intervals of
x determined by the different powers of ten or some other base
number. In thi s case, by the same reasoning, the log-uniform den-
sity proportional to x I is the appropriate one, for this is exactly
the density that gives intervals of successive powers equal values.
And we have yet another contradiction.
There are no inj(Jrmationless priors, and the quest for them is
the quest for the chimera. On the other hand, it might be argued
that that quest is arguably little more, if indeed any more, than a
quest for in the definition of prior (or indeed any other)
probabilities. Whether objectivity really is the unquestionable
desideratum it is cracked up to be is an issue we shall come back
to later (we shall argue that it is not), but certainly most of the
people who have scientific applications of the Bayesian formal-
ism as their methodological goal think that it is. Foremost among
these so-called Bayesiam' are (or rather were; one died
shortly before the end of the new millennium, the other shortly
after) Jeffreys and Jaynes, who explicitly endorsed Jeffreys's gen-
eral viewpoint and extended some of his methods. Jaynes
demanded that prior distributions be objective in the sense that
they are "independent of thc personality of the user" (1968, p.
117), which he elaborates into the condition that
in t wo prohleJ1/ s where tve have the same prior injcJ/"fI/afion, we
should assign the same probabilities.
But how? Jaynes's answer is to appeal not to the criterion of in for-
mationlessness, which is one which, we have seen, cannot be sat-
isfied, but to one of minimum information: we should choose the
prior containing thc least information beyond our prior data, or
making the leVl 'est assumptions beyond that data (1957, p. 623).
He argues that thi s demand can be satisfied in a uniquely determi -
nate way if the background data are of a suitable form, and even
where there are no background data. The method that achieves
this is, he claims, that of maximum entropy. Jaynes draws here on
thc seminal work of Claude Shannon in information theory.
Shannon showed that if p = (PI' ... ,P
II
) is a probability distribu-
tion taking only finitely many values (i.c. it is the distribution of
278 CHAPTER 9
a random variable X taking n values with those probabilities),
then a very plausible set of conditions on a numerical measure
of the uncertainty H(p) attaching to p determines H(p) to be the
entropy -XpJogp;. unique up to choice of base for the logarithm,
which is maximised by the uniform distribution n-
I
, and min-
imised by the distribution which attaches the value 1 to a single
point. The uncertainty is thus a function of the distribution itself,
satisfying the intuitively satisfactory property that the more
evenly spread out the distribution, the greater the uncertainty
attaching to it.
Given suitable prior information k we should, according to
Jaynes, take the prior distribution over X to be that which max-
imises the entropy subject to the constraints reprcscnted by k.
Maximising H subject to k means of course that k has to express
conditions on p itself. In Jaynes's examples the constraints usual-
ly take the form of expectation values 'derived' from very large
sets of observations (as arise frequently in experimental physics;
other types of constraint might be independence assumptions,
conditions on marginal distributions, etc.), and it is well-known
that for constraints of this form there is always a unique solution
to the maximisation problem (in fact, a unique solution is guaran-
teed for any set of linear constraints, of which expectations are
one type
4
).
It might be objected that outside physics there are few prob-
lems amenable to solution this way- where because of the very
large numbers involved the data can be represented directly as
conditions on the prior p. But recall that Bayes 's virtual invention
of the Bayesian theory as a theory of posterior probabilities was
intended to solve the theoretical problem of determining an exact
epistemic probability distribution given sample data, no matter
how large the sample. Thus the prior use of data to furnish con-
straints on a probability distribution raises the question whether
the maximum-entropy method might be in contlict with condi-
tionalisation (given the latter's own conditions of validity; see
Chapter 3), a question that others have raised, together with exam-
ples where it does seem to be (sec Seidenfeld ) 979, Shimony
4 A unique solution is guaranteed for any closed convex set of constraints.
FINALE SOME GENERAL ISSUES 279
1985), and which seems not to have been satisfactorily answered
by advocates of maximum entropy. To say, as Jaynes does, that
maximum-entropy is a method for selecting prior distributions,
not posterior ones where conditionalisation is the acknowledged
tool , rather sidesteps the problem.
That is one question. Another arises in the context of no non-
trivial prior information, i. e. no constraints at all other than
describing a finit e range of values which the data can in principle
take (so the constraint set is simply of the form IP(r:) = I). the
entropy-maximi si ng distribution is the uniform distribution. This
might seem-disregarding the problem mentioned above-to
amount to a vindication of the Principle of Indifference, since that
was postulated on a state of prior ignorance. But it is easy to see
that equally it can be regarded as bequeathing to the method of
maximum entropy all its most intractable problems, for it implies
that we shall get incompatible uniform distributions as maxi mum-
entropy solutions with respect to different ways of representing
the space of 'elementary' possibilities (as with the different pos-
sibility spaces of L/ and L: above).
Nor can the problem be simply shunted off to one side as a
peripheral problem associated with an extreme and arguably
rather unrepresentative type of background information. It
becomes of central importance when one considers the apparent-
ly purely technical problem of extending the Shannon entropy
measure to continuous distributions p = p(x). The 'obvi ous' way
of doing this, i.e. by taking H to be the integral
- Jp (Y)/ogp (x)dx.
has the serious drawbacks of not being guaranteed to exist and,
where it exists, being description-relative: it is not invariant
under change of variable (because it is the expected value of a
density and hence is dimensional, since it is probability per unit
x). The form chosen by virtually all maximum entropy theorists
(including Jaynes) to extend H to the cont inuum, which is
invariant under coordinate transformations, is the functi onal of
two variables
I(p.q) = Jp(r:)log[p(x) I q(Y)}dx.
280 CHAPTER 9
for some other density q(y) (Jaynes 1988, p. 124). I (p,q) is some-
times called the cross-entropy, sometimes the information in p
given q (Jaynes simply calls it the entropy). Where the range of X
is finite and p(x) and q(x) are discrete distributions with q(j a
uniform distribution I(p,q) reduces to the ordinary entropy5. In
addition, since I(p,q) is never negative, the task is now to min-
imise a functional rather than, in the case of H. to maximise one.
Adopting I rather than H may solve the mathematical prob-
lems, but it leads to a rather obvious interpretative problem: what
is q(y)? The answer is implicit in the fact that where there is no
background information other than that the range of X lies in a
bounded interval, the I-minimising density is p(x) = q('() up to a
constant factor. In other words, as Jaynes himself pointed out, q(x)
virtually has to be interpreted as a density corresponding to
(almost) complete ignorance (1968, p. 125), an answer which, as
he o n e d e ~ appears to lead straight back into the no-go area of
the Principle of Indifference. Notice that since q(x) is now neces-
sarily present in every problem where there is a continuous
hypothesis-space (as for example there is with most parameter-
estimation problems in statistics), the problem of confronting
ignorance distributions is no longer confined to a class of cases
which can be conveniently written off as merely academic.
It is a testament to Jaynes's ingenuity (and mathematical skill)
that he was able to propose a way out of the impasse, and a novel
way of formulating the Principle of Indifference which seems to
offer at least a partial solution to the problem of transformations.
This is his method oj'transj'ormation groups. The idea here is that
a definite solution (in q(x) may be revealed by considering
equivalent representations of the problem, a class which does
not, Jaynes argued, necessarily permit all logico-mathcmatically
equivalent representations of the hypothesis- or parameter-space
to be counted as equivalent. More specifically, investigation of
the problem may well reveal the only degrees of freedom implic-
it in its statement are those generated by the action of some trans-
formation group, or class of transformation-groups. To take an
example that Jaynes himself uses, suppose that the sampling dis-
j There are several deri vations from first principl es of I (p.q) as a measure of rel-
ative information whether p and q are di screte or continuoll s.
FINALE SOME GENERAL ISSUES 281
tribution of an observable variable is known except for a scale
parameter a 6. We are not completely ignorant about since by
assumption we know that it is a scale parameter; but that is all we
know. In that case, according to Jaynes, the problem of determin-
ing the distribution of a remains the same problem under arbi-
trary choices of scale, and these form a group, the group of
transformations cp = aa. a > O. Hence the target distribution q(a)
must be invariant under the action of that group. Moreover, since
all we know is that a is a scale parameter, that group determines
all the degrees of freedom permitted by the problem: it is not per-
missible to demand invariance under taking logarithms, for
example.
The following simple argument now shows that q is uniquely
determined by this condition. Firstly, by the probability calculus
and elementary differential calculus the prior densities q(a) and
h(cp) are related by the equation (dcp / da)h(cp) = q(a), by
ah(aa) = q(a).
The assumption of scale-invariance means that the scale-
shift from to leaves the form of the prior unchanged: hand q are
exactly the same function on their common domain, the set of
positive real numbers. Substituting h(. ) = q(. ) accordingly, we
obtain
aq(aa) = q(a)
Setting (J = I we have aq(a) = q(l) for all a > (), and hence we
infer that q(a) cxa I. In other words, we have, up to proportion-
(; A parameter of a sampling distribution is called a scale parameter if the data
transfclI'Illation taking ,1' to.Y = n' for a constant c has the effect of changing the
likelihood function so that the likelihood of), on ,1' is the same as that of c), on x
(to avoid the problem that the likelihood function is defined only up to a multi-
plicative constant, the function rcferred to here is the so-called standardised, i.e.
normalised, likelihood, which is any version L(), 1.1') divided by L(1c 1.1)d},). ), is
a locatio/1 parameter if the transformation x = l' + (' gives Ic + (' the likelihood
on x corresponding to that of on .1'. Familiar examples of each type of parameter
are the standard deviation of a normal distribution (scale parameter), and the
mean (location parameter),
282 CHAPTER 9
ality, determined the form of the (improper) prior density to be
log-uniform (in fact, this density is invariant under a much larger
class of transformations; namely, all those of the form ax", a > 0,
b", Op
Thi s usc of the statement of the problem to determine the
precise extent to which priors should reflect ignorance suggest-
ed to Jaynes that in other problems where the Principle of
Indifference merely yi elds inconsi stencies his own method
might be successful. The most celebrated of these was
Bertrand's celebrated, or rather notorious, inscribed triangle
problem. This is the problem of determining the probability that
a 'randomly' drawn chord to a circle will have a lengt h less than
the length L of the side of the inscribed equilateral triangle (of
course, there is an uncountable infinity of inscribed triangles,
but they are all congruent). The paradox arises because there are
different , but equivalent, ways of specifying a chord of a given
length. Here are three Bertrand considered (the exposition fol-
lows Gillies 2000, pp. 37- 39): (I) where the midpoint of the
chord occurs on a radius (5); (2) the angle between an end-point
of the chord and the tangent at that point (5]); and (3) the posi-
tion, in pol ar coordinates say, of the midpoint in the circle (5).
5" 5] and 51 determine different possibi lity-spaces: (I) all the
points on the radius; (2) the angles from 0 to Jr, and (3) the set
{(dt): 0 r R, 0 2Jr}, where R is the length of the radius.
Subsets T" T] and Tl of 5,. 5:; and S, respectively correspond to
the event, call it C. that the chord-length is less than L. T, is the
set of point s on a radius between its perpendicular intersection
with a side of the inscribed triangle and its intersection with the
circumference of the circle; T:; is the 60 angle between the tan-
gent and the nearest side of the inscribed triangle whose vertex
is at the tangent: and Ti is the set of points in the interior circle
of radius R12. Both members of the pairs (5, T) have a 'natural'
I I
measure m; induced by their geometrical structure: in the first,
length, in the second, angular measure, and in the third, area
7 Scalc-invari ance has also been used to explain why the leading digit 11 in the
measurement s in signi f'icant figures of many nat urally occurring magnitudes
often has the logar ithmic distribution log(( 1 + /I) / 11) (a regul arity known as
Bell{rml :I' Law; see Lee 1997, pp. 100-02).
FINALE: SOME GENERAL ISSUES 283
measure. The Principl e of Indifference decrees that relative to
each such pair the probability of C is equal to m,(T) I m/Sj
Thus, the probability of C relative to the first pair (S/, T/) is 1/2,
relative to the second is 1/3, and relative to the third is 114. Thus
we have Bertrand s Paradox: relative to three different, appar-
ently equivalent, ways of describing the probl em, we get three
different values.
According to Jaynes, however, Bertrand's probl em does admit
a unique solution, ~ He proposes that we view the problem in the
context of somebody, say Jaynes, throwing longi sh straws ran-
domly so that they intersect the circumference of a circle inscibed
on a plane surface. Suppose f(x,}) is the probability-density that
the midpoint ofthe chord lies at the point (l,y) in or on the circle
(assumed to be centred at the origin). Nothing in the problem
specifies the origin, the orientation of the coordinates, or the size
of the circle, so according to Jaynes this tells us that f should be
translation, rotation and scale-invariant. Rotational invariance
implies that f depends on (x,y) through v'(x:!+y') only.
Transforming to polar coordinates, the joint probability density
p (f; 8) must therefore be rf(rJ, () ::; r::; R, () ::; f} ::; 2n. Now consid-
er a scale transformat ion r' = w; a ::; 0, which we can think of as
blowing up or shrinking the original circle. Suppose it is shrunk,
i.e. a ::; I. The probabi lity densities p(r) and p' (r') are related in the
usual way by
p'(r') dr'ldr = pM (i)
We cannot immediately invoke scale invariance to identify the two
density functions p and p', since they have different domains, [O,aR]
and [O,R] respectively. What we can do, though, is to restrict r to
[O,aR] by conditioningp on the information that r li es in [O,aR], and
then equate the resulting conditional density with p'. This is what
Jaynes does, and differentiates the resulting equati on with respect
to a to obtain a differential equation whose soluti on is
f(r) = qr'l 212rrR "
where () ::; q. Finally, Jaynes shows that translation-invariance
uniquely determines q = I, givingf(rJ = 1 /2nRr (however, as he
284 CHAPTER 9
points out, translation-invariance is by itself so strong a condition
that it already determines the result). Thus the required probabil-
ity density is uniform, and the probability that the chord is less
than the length of the inscribed triangl e is 1- (1/2rrR)r I ~ l r d 8
which is easily seen to be equal to ~ i.e. to the first of the solu-
tions above. Not only, apparently, is the solution uniquely deter-
mined, but it is one which, as Jaynes noted with sati sfaction when
he carried out experiments of actually throwing straws into a cir-
cle, was observed to "an embarrassingly low value of chi-
squared" (1 973, p. 487).
Jaynes conceded that not all the examples where the Principle
of Indifference breaks down yield to this sort of attack: he cites
the well-known wine/ water problem (von Mises 1939, p. 77) as
one which docs not. But even hi s solution of the Bertrand prob-
lem raises serious questions. According to Jaynes, the method of
transformati on groups consists of noting those aspects of the
problem whi ch are left unspecified, and then requiring invariance
of the pri or under the corresponding groups of transformations
(1968, p.1 28, 1973, p.430). But note that in Jaynes 's own state-
ment of Bertrand's chord problem, which actually specifies an
indi vidual , .I, throwing straws into a circle inscribed on a plane
surface, there is no menti on of the relati ve speed of.J and the cir-
cle. Suppose that they are in uniform motion with velocity v, and
that the coordinate frames of J and the circle are coincident when
J notes the midpoint of the chord. It foll ows from the equations of
the Lorentz transformati on that the circle in the frame taken to be
at rest (say, J's) is an ellipse in the moving frame, with eccentric-
ity v (in units of c = I), and minor axis in the directi on of motion.
Thus there is no longer rotational symmetry for J, since ,. is short-
ened in J's frame by the factor [(1 - ,,2)cos
2
(:1 + sin
2
0] 12. The
problem seems no longer so well-posed, at any rate for J.
Jaynes would no doubt reply that it was intended that the cir-
cle be at rest (at any rate up to what is observationall y detectable)
with respect to J. But according to Jaynes 's own criteri on, chang-
ing any characteristic not actually speci f ied in the statement of the
problem should count as giving an equivalent probl em, and the
relati ve speed of J and circle was not mentioned in his reformula-
FINALE: SOME GENERAL ISSUES 285
tion of Bertrand's problem. Even if it were fixed by some addi-
tional stipulation, that would still never succeed in reducing all the
degrees of freedom to a manageable number, since they are in
principle infinite. Why shouldn't arbitrary variations in space-
time curvature be included in the list, or even arbitrary coordinate
transformations, which are not mentioned either? Of course, one
could simply state the 'permitted' transformations at the outset,
but even if they generate a unique solution it will be tantamount
to invoking the sort of personal decision that the idea of objective
priors was intended to replace:
if the methods arc to have any relevance to science, the prior distri-
bution must be completely "objective", in the sense that it is inde-
pendent of the personality of the user ... The measure of success ...
is just the extent to which we are able to eliminate all personalistic
elements and create a completely "impersonalistic" theory. (1968, p.
117)
Jaynes's entropy-based methodology was an attempt to bypass
the problems facing the Principle of Indifference: instead, the
problem of determining the distribution q('t} merely meets them
all again head-on. They arc not to be by-passed so easily. Recall
Jaynes's claim that, because it maximises uncertainty, the max-
imum entropy distribution, where it exists, makes the fewest
commitments or assumptions beyond those in the data. Even
granted the equation of entropy with uncertainty this assertion
rather obviously begs the question, and is certainly incapable of
proof, or even moderately convincing supporting argument ,
because on any reasonable understanding of the terms involved
it is false. A flat distribution maximises uncertainty, but it com-
mits its user to certain views about the probability of every out-
come, or neighbourhood of every outcome if it is continuous.
As far as its probabi listie content is concerned, therefore, it
makes exactly as many commitments as any other distribution.
This conclusion is, of course, just a reprise of our verdict on the
search for informationless priors (there arc none, because every
distribution contains just as much probabilistic information as
any other). Unfortunately, it has yet to be drawn by some
authoritative Bayesians:
286 CHAPTER 9
Even when a scientist holds strong prior beliefs about the value of a
parameter, nevertheless, in reporting his results it would usually be
appropriate and most convincing to his colleagues if he analyzed the
data against a reference prior [a reference prior is a prior distribution
which is slowly varying and dominated by the likelihood function in
the region where the latter takes its largest values]. He could then say
that, irrespective of what he or anyone else believed to begin with,
the prior distribution represented what someone who a priori knew
very little about should believe in the light of the data. (Box and Tiao
1973, p. 22.)
There do admittedly exist alleged proofs that the maximum
entropy method is uniquely determined by plausible assumptions;
indeed, there is "quite an industry", to quote Paris (1994, p. 79).
But closer investigation reveals that there is often (much) more to
these conditions than meets the eye. For example, Paris and
Vencovska's (200 I) apparently innocent 'renaming principle'
implies that, with a constraint set asserting that with probability
one there are n elementary possibilities, each should receive prob-
ability lin (and we are back with the Principle of Indifference in
yet another guise); another of their conditions is that if the con-
straint contains only conditional probabilities of a given c and b
given c, and the unconditional probability of c, then a and b
should be independent given c, a very strong condition reminis-
cent of the default-negation rule in logic programming.
8
There is one other aspect of maximum entropy that we should
mention before leaving the subject, though this concerns exclu-
sively the Minimum Information Principle under the very differ-
ent interpretation as an updating rule, from Jaynes's. On this
interpretation, q(.r:) is the prior probability and p(x) the posterior
updated on the constraints relative to which I is minimised. This
conveniently sidesteps the problems with Jaynes's use of maxi-
mum entropy/minimum I as a determiner of prior distribitions.
With the I-minimising p(x:) treated as a new way of generating a
x A theorem of Williamson, about separating sets of variables in a constraint
graph, implies that the maximum entropy solution for the constraints above auto-
matically renders a and h conditionally indcpcndent given c (2005, p. 86). This
underlines how far the maximum entropy mcthod transforms null information
into positive information.
FINALE: SOME GENERAL ISSUES 287
posterior probability relative to q(x)'s prior, and the minimum-
information principl e thereby transformed into an updating rule,
things look altogether more promising. For a start, the Principle
subsumes both Bayesian and Jeffrey conditionali sation: minimis-
ing l(p.q) subject to a shift from q(e) to pre) = p yields the latter:
p('() = q(x I e)p + q(x I - e)( 1 - p)
(we get the appropriate general form when the shift occurs simul-
taneously on the members of an arbitrary finite partition; for a full
discussion see Will iams 1980). Where the contraint is that q(e) = 1,
we obtain Bayesian conditionalisation: p(x) = q(x I e).
These facts provide another defence of conditionalisation,
Bayesian or Jeffrey, depending on the constraints. Ifwe grant that
l(p.q) can be interpreted as a distance measure in distribution
space between p and q, then minimising l(p.q) for fi xed p means,
on this view, selecting as one's new distribution that which is as
close to p as possibl e subject to the constraints. Thus conditional-
isation becomes justifi ed as the selection of the distribution clos-
est to the prior. There are several problems with this defence,
however. One is that I(p.q) is no ordinary di stance measure since
it is not symmetric. Another is that there are alternative metrics
which do not endorse conditionalisation in thi s way. A deeper
objection is that it simply begs the question why one should
choose the closest measure to p, particularly as the shift on e
might be very considerable. Shore and Johnson ( 1980) prove that
the information-minimi sing measure is the onl y one satisfying a
li st of what they term consistency constraints, but one of these is
the assumption that the task is to extremise a functional (why
should this be a consistency condition?), whil e another turns out
to be a strong independence condition (and rather similar to the
independence condi tion of Paris and Vencovska di scussed in the
preceding paragraph) which is certainly not a mere consistency
constraint and, as Uffink points out with supporting examples,
may in the appropri ate circumstances be very unreasonable (1995,
pp. 245- 247; Uffink 's article al so contains an excellent critical
discussion of Jaynes 's use of Maximum Entropy) . Given the other
problematic features of conditionalisation we pointed to in
Chapter 3, we feel that in linking its fortunes to the principle of
288 CHAPTER 9
minimum information no real advance has been madc in justify-
ing its adoption as an independent Bayesian principl e.
9.0.4 Simplicity
No examination of ways of trying to impose 'obj ective' con-
straints on prior probabilities is complete without a di scussion of
a criterion of ancient pedigree: simplici(v. Relati vised to the
Bayesian way oflooking at things, the idea is that greater relative
simplicity of an expl anatory or predictive theory should be
reflected in its having a hi gher prior probability. But the criterion,
while plausible, has its problems. One is lack ofuni vocality: there
are different ways in which we judge things simple, and these are
not all equi valent, and some are highl y description-relative. For
example, the equation of a circle with radius k centred at the ori-
gin has the equation x
2
+ y 2 = k
2
in Cartesian co-ordinates, but
the apparently much simpler equation r = k in pol ar coordinates.
Tn general there are many different ways of characterising the
main principles of a theory, whose choice may depend on a vari-
ety of factors, and whi ch may seem more or less simple depend-
ing on the application to hand.
But there is a more objective, less language-dependent sense
of simplicity, which al so appears to playa role in at least some
areas of science, and that is simplicity in an Occam's razor-sense
(Occam's famous methodological principle was 'entiti es are not to
be multipli ed without necessity' ), whi ch has a preci se mathemat-
ical expression asje,vness o/"independent adjustable parameters.
This certainl y strikes a respondent chord with scientists: one of
the considerations telling against the current Standard Model of
subatomi c physics is that it contains no fewer than 20 adjusted
parameters. Jeffreys 's modified Galil eo law (4.i above), which no-
onc would accept in preference to Galileo's own, has the form of
Galileo's law with k additional adjustable parameters evaluated at
the data points t
l
, . . . ,!k' A simpler form which fi ts the data as
well is not only more elegant and mathematically tractable but
also, we feel, more I i k e ~ J on that account to he true. In that case,
why not adopt as an independent rul e that hypotheses with fewer
adjustabl e parameters should receive greater prior probability?
This is what Jeffreys himself advocated, calling the rule the
FINALE SOME GENERAL ISSUES 289
Simplicity Postulate (1 961, pp. 46- 50). It may not determine prior
probabilities uniquely, but it docs act as an obj ecti ve constraint on
thcm where it is appli cable.
Rather surpri singly, some have argued that such a principlc is
actuall y inconsi stent. Poppcr was probably the f irst to make this
claim (l959a, pp. 383-84), and Forster and Sober in a series of
papers in effect repeat Popper's argument. Thi s is that since a
polynomial relati on of degree n is also one of every higher degree
111>n, with the coefficients of all terms of degree greater than n set
equal to zero, the lower-degrcc hypothesis cannot have a larger
probability than any of higher degree, sincc the f irst entails the
second and probability must respect entailmcnt (Sober and
Forster 1994). For examplc, a straight line y = I11X + c is also a
parabola with the coefficient ofx2 sct equal to O.
The argument is, however, easily rebuttcd by noting that the
interest is usually in tcsting against each other not compatiblc but
incompatible hypotheses, for cxample whether the data are better
explained by the exi sting hypothesis or by adding a new parameter
in the form of a nonzero coefficient to a hi ghcr-degree term
(Howson 1988a). Thus, to use Forster and Sober's notation, sup-
pose U N is the set of all linear models and QUAD thc set of all
quadratic ones. In testing whether the true model is a J inear one ML
or a quadratic one MQ the tester is not testing UN against QUAD;
since they have common elements it would bc like testing M
j
against Me The test is between UN and QUAD* where QUAD*
contains all the models in QUAD which are not in LIN. While
P(UN) is necessarily no greater than P(QUAD) by the probability
calculus, P(UN) can consistcntly be greater than P(QUAD*).
Jeffreys himsel f regarded discriminating between such disjoint
families as UN and QUAD* in curve-fitting probl ems as a clas-
sic arena for the Simpli city Postulate, pointing out that the penal-
ty of being able to f it the data exactl y by means of a plcntiful
enough supply of free parameters is over(itting:
lfwe admitted the full n [parameters] ... we should change our law
with every observat ion. Thus the principle that laws have some valid-
ity beyond the ori ginal data would be abandoned ( 196 1, p. 245)
Indeed, "the simplest law is chosen because it is the most likely to
give correct predicti ons" (Jeffreys 1961, p.4). Since the promotion
290 CHAPTER 9
of predictive accuracy is Forster and Sober's own declared aim,
their charge that Jeffreys's restriction of the simplicity ordering to
disjoint polynomial families is an "ad hoc maneuver" which
"merely changes the subject" (1994, p. 23; their italics) is simply
incorrect.
Forster and Sober make a further curious claim. The hypothe-
sis that some relationship is a particular degree of polynomial
asserts that it is some (unspecified) member of the corresponding
family of curves, and hence computing its posterior probability
means computing the posterior probability, and hence the likeli-
hood, of that family. On this point Forster and Sober claim that
it remains to be seen how [Bayesians] ... arc able to make sense of
the idea that families of curves (as opposed to single curves) possess
well-defined likelihoods. (1994, p. 23)
This is strange, because the ability to make sense of the idea is
guaranteed in the Bayesian theory, whereas, ironically, Forster
and Sober's charge is rather accurately brought against their own
account: in any theory restricted to considering likelihoods there
is no comparable body of first principles which generates likeli-
hoods offamilies (disjunctions) of hypotheses. Even 'the father of
likelihood', R.A. Fisher, conceded that it makes no sense to talk
of the likelihood of a disjunction (he described it as like talking
about the "stature of Jackson or Johnson" (1930, p. 532). In the
Bayesian theory the likelihood ofa disjoint family {hi} of curves
(determined, say, by a discrete parameter) with respect to data e is
easily obtained via the probability calculus in terms of prior prob-
abilities and the likel ihoods of each member of the family
XP(e I h,JP(h)
L({h)le) 'X P(e lvh) = 2:(h) (where vh
i
is the dis-
junction of the hi' In the continuous case the sum is replaced by
integration where the integrals are defined. Consider, to take a
simpl e example, the hypothesis that the data x are normally dis-
tributed with standard deviation around some unknown value t.
The likelihood is equal to JV(2Jr(J2)- lexp[- (1 /2(J2)(x - t)2]f(t)dt.
wheref(t) is the prior density and the integration is over the range
FINA.LE: SOME GENERAL ISSUES 291
of values of t. The prior may of course be subjective, but that still
does not make the likelihood ill-defined; on the contrary, formal-
ly speaking it is perfectly well defined.
While, pace Forster and Sober, there is no technical problem
with a Simplicity Postulate, we doubt that simplicity, in itself and
divorced from the consideration of how it is related to the scien-
tist's background information, is or should be regarded as a crite-
rion of any great importance in guiding theory-choice. While a
dislike of too many adjustable parameters is often manifested by
scientists, it arguably depends on the merits of a particular case,
and a closer examination usually reveals that it is considerations
of plausibilizv, not simplicity in itself, that ultimately determine
attitudes. Indeed, it is easy to think of circumstances where the
simplest hypothesis consistent with the data would be likely to be
rejected in favour of a more complex one: for example, even
before the data are investigated in some piece of economic fore-
casting, models with very few parameters would be regarded with
great suspicion precisely because there is an equally strong belief
in a (large) multipl icity of independent causes.
This is not to say that nothing definite can be said in favour of
a here-and-now preference for theories with fewer parameters. For
example, it can easily be shown that, other things being equal,
hypotheses with fewer parameters get better confirmed by pre-
dicted observational data: a certain amount of the data is absorbed
merely in evaluating the parameters, leaving the remainder to do
the supporting-or not-of the resulting determinate hypothesis
(Howson 1988b, pp. 388-89). But 'other things being equal' here
means that the two hypotheses have roughly equal prior probabil-
ities. This is an important point in the context of the perennial
' accommodation versus prediction' debatc, since what it points to
are circumstances where the accommodating hypothesis is more
highly confirmed than the independently predicting one, namely
where the prior probability of the latter is sufficiently low com-
pared with that of the former (this is shown in detail in Howson,
op. cit.). Everything, in other words, over and above fit with the
data is a matter of prior judgments of plausibility.
And this seems true even at a very intuitive level. A curve of
high degree in some 'natural' co-ordinate system would be
thought by most people to be more complex than one of low
292 CHAPTER 9
degree. But suppose a hi ghly complex relationship y(x) is thought
to hold between two observable variables x and y. Draw an arbitrary
straight line through the curve intersecting it at k points ('(i'Y). i =
I, ... , k . Now empirically determine the values y (x) for the select-
ed Suppose that, within the bounds of error, Yi = y ('(). Would
you regard the straight line as being confirmed by these data rather
than the favoured hypothesis? No. Ultimately, the criterion count-
ing above all is plausibility in thc li ght of background knowledge;
to the extent that simplicity is a criterion, it is to the extent that it
supervenes on prior plausibility, not the other way round.
There does however remain the legitimate methodological
concern that too plentiful a use of adjustable parameters to fit cur-
rent data ever more exactly invites the risk of future overfitting,
"as a model that fit s the data too closely is likely to be tracking
random errors in the data" (Myrvold and Harper 2002, p. 137).
But then again, a very poor fit to present data with 11 parameters
suggests that it may be necessary to introduce an (n + I )th. It is
all a question of balance. It mi ght be thought that thi s rather mod-
est conclusion sums up all that can usefully be said on the matter,
but Forster and Sober, scourges of Bayesianism, cl aim that a the-
orem of Akaike shows that there is actually a great deal more that
can be said, and of a mathematically precise character. To exam-
ine this claim we need f irst a precise notion of lit to the data of a
hypothesis with 111 free parameters determining (it is the ' di sjunc-
tion' of all of them) a corresponding famil y hll/ of specific
hypotheses, which we shall take to be characteri sed by the density
pry I q) with q an adjustable parameter-vector of dimensionality m.
For any given data x let max (m) be that member of h which has
r 11/
maximum likelihood on x, i.c. the hypothesis whose parameters
are the maximum-like lihood estimates 8('() determined by the
data x. Let us, for the sake of argument, follow Forster and Sober
and regard this as the hypothesis in hll/ which best f its x . Now let
max,(m) be the log-likelihood of xm on x. Note that formall y
max,(m) is a functi on of x. Let p* be the true probability distri-
bution of the possible data generated by the observati ons. Forster
and Sober take the j oint expectation p(yl(}('()} comput-
ed relati ve to p* to measure the predictive accuracy of hll/ ' Again,
for the sake of argument let us agree, though we shall have some-
thing to say shortly about this use of expectations.
FINALE: SOME GENERAL ISSUES 293
The next step is the dramatic one. According to Sober and
Foster, Akaike has shown how to construct a numerically precise
estimate ~ l the degree to which additional parameters 'vvill over-
fil. For Akaike showed that under suitabl e regularity conditions
l(max)m) - /11 for the actual data x is an unbi ased estimate of the
predictive accuracy of hill ' as defined above. Akaike's result, often
known as AIC for 'Akaike Informati on Cri teri on' , thus appears to
tell us that in estimating the predi cti ve accuracy of a hypothesis
with m free parameters from the current data, we must subtract
from its current goodness of fit a penalty equal to In. In other
words, other things being equal (current fit) , we do better from the
point of view of future predictive success to choose the simpler
hypothesis.
What should we make of thi s? One thing we are not question-
ing is the connecti on with information theory. Akaike also showed
that AIC is an estimate of the discrepancy between probability
distributi ons as measured by the Kullback-Leibl er measure of dis-
tance between a model and the 'true' di stribution, a measure
which we have already encountered in the di scussion earlier of the
Principle of Minimum Information (hence 'Akaike Information
Criterion' ). Thi s is eminently discussible, in particular because (i)
the Kullback- Leibler measure is not actuall y a true distance (it is
not symmetri c), and (ii) there are other di screpancy measures in
distributi on space, for example variation di stance, but the Akaike
criteri on fail s to estimate these, that is, it is not robust over di s-
crepancy measures. But that is not our present concern. That con-
cern is that Sober and Forster see ArC as justifying the claim that
on average simpl er models are predicti vely more accurate than
compl ex ones. We shall now say why we think there are serious
grounds for questi oning this claim.
Firstl y, there is the question of how well the 'suitable regular-
it y conditions' arc sati sfied in typical probl ems of scientific theo-
ry choi ce; the conditions are actuall y rather restricti ve (see
Kieseppa 1997 for a full er discussion). Secondl y, this account suf-
fers from an acute version of the reference-class problem.
Suppose r usc as many parameters as is necessary to obtain a per-
fect fit to the data within some polynomi al family. The resulting
hypothesis h is a member of the singl eton family {h}, which is a
family with no adjustable parameters and which has excellent fit
294 CHAPTER 9
to the current data. According to the Akaike criterion no other
family can do better than {h}. But this is absurd, because we know
that h will overfit, and indeed is exactly the type of hypothesis
whose merited killing-off this analysis was developed to justify.
There is an extensive literature on Forster and Sober's advoca-
cy of the Akaike criterion, and they themselves do attempt to
answer this rather devastating objection. We do not think they suc-
ceed, but there is another type of objection to their enterprise
which we believe to be just as undermining. An unbiased esti-
mate, recall, is characterised in terms of its expected value. The
use of estimates based on their expected values is usually justified
in terms of an asymptotic property: the sample average converges
probabilistically to the expectation under suitable conditions. This
is the content of a famous theorem of mathematical probability
known as Chebychev's Inequality, and Bernoulli 's theorem is a
famous special case of it, where the expected value of the sample
average (relative frequency) is the binomial probability. But as we
saw, you cannot straightforwardly infer that probabi lity from an
observed relative frequency; indeed, the only way you infer any-
thing from the relative frequency is via the machinery of Bayesian
inference, and there is none of that here. Moreover, it is easily
shown that there is an uncountable infini(v of unbiased estimates
of any given quantity, all differing from each other on the given
data. How can they all be reliable when they contradict each
other'? And to compound the problem still further is the fact that
the quantity being estimated by the Akaike criterion is yet anoth-
er expectation!
Frequentists usually respond to the question raised by the mul-
tiplicity of unbiased estimators by saying that one should of
course choose that with minimum variance; if the variance is very
small then--so the argument goes-one is more justified in using
that estimator. But even were one to grant that, there is no guar-
antee that the Akaike estimator is minimum vari ance, let alone
small. Even if it were small, that would still leave the question of
how likely it is that this value is the true one, for which, of course,
one needs the Bayesian apparatus of posterior probabi I ities.
Perhaps surprisingly, there is a Bayesian analogue of Akaike's
criterion, due to Schwarz and known as BIC (Bayesian
Information Criterion). BIC replaces the penalty m in Akaike's
FINALE: SOME GENERAL ISSUES 295
estimator by the quantity (1/2)mlog n, where n is the number of
independent observati ons, but is otherwise the same:
l(max,(m)) - (1 /2)mlog n BIC
But the justificati on of BIC, as might be expected, is very differ-
ent from that of AIC: BIC selects the model whi ch maximi ses
posterior probability as the data grows large without bound
(Schwarz 1978, p. 462) . Under fairly general conditions the pos
tcrior probability takes its character from the behaviour of the
likelihood function for a sufficiently large indcpendent sample, a
fact which cxplains the presence of the likelihood term in BIC.
That sounds good, but unfortunately BIC has its own attendant
problems, principal among which is that its justifi cation is asymp-
totic, giving the model with the highest posteri or probability but
only in the limit as the data extends. But we do not live at asymp-
totes. It is a simple matter to choose other sequences depending on
the data which have the same limit properties, but which are all very
different on any finite sample. Again, there is nothing like a guar-
antce from BIC that using it at any given point in the accumulation
of data we are on the ri ght track. From a purely theoretical point of
view, howcver, BIC does, in a way that AIC does not, offer an intel-
ligible and straightforward justification of the intuition that too-pre-
cise fitting to the data means overfitting which means that it is
unlikely that the resul t will survive a long enough run of tests. Thus
Jeffreys 's claim that ' the simplest law .. . is the most likely to gi ve
correct predictions' is clearly underwritten by BIC, at any rate
asymptotically. AIC, by contrast, merely tells us that expected fit is
improved, according to an unbiased estimator, by reducing the num-
ber of free parameters, and as we have seen it in fact tells us noth-
ing about how much more likely a simple theory is to be true.
And now we are back on familiar ground, where the choice
between two accounts of why simple hypothcses are meritorious
is at bottom just the choice between a frequenti st, classical vicw
of statistical inference as against a Bayesian. It has been the bur-
den of thi s book that onl y the Bayesian offers a coherent theory of
valid inductivc inference, and that, despitc its suggesti ve termi-
nology, of unbiased ness, sufficiency, consi stcncy, significance
and the like, the cl assical thcory is in fact shot through with sys-
296
CHAPTER 9
tematic question-begging. The discussion of these two superfi-
cially similar but in reality very different justifications of simplic-
ity underlines that view.
AIC and BIC are actually not the only approaches to trying to
justify simplicity-considerations in terms of some more tangible
methodological goal, though these deal with rather different con-
ceptions of simplicity. One, due to Solomonoff and others, appeals
to Kolmogorov complexity theory. Assume that hypotheses assign
probabiliti es to possible data strings. We can suppose without loss
of generality that both hypotheses and data are coded as finite
strings of Os and I s. The complexity of any such string is defined
to be the length of the shortest computer program which will gen-
erate it. The fact that such program-lengths across different 'uni-
versal' programming languages (like LISP, PROLOG, JAVA etc.)
can be proved to be uniformly bounded by a constant means that
the definition is to that extent relatively language-independent.
Suppose a data sequence of length n is observed, and that P is the
true distribution of the data. Let the error in predicting the next
member of the sequence between P and any other hypothesised
distribution, P " be the square of the difference between the two
probabilities on that member conditional on the data. Solomonov
showed under quite general conditions that if a certain prior distri-
bution A ('the universal enumerable semi-measure') is employed,
which is also a prior distribution weighting compl ex sequences
lower than simpler ones according to the complexity criterion
above, then the expected value of the error convergcs to zero (li
and Vitanyi 1997). But now we have yet another criterion justified
in terms of an expected value, and everything we said earlier about
the Akaike criterion applies here
9.b Summary
Simplicity is a snare, in our opinion, in whatever formal guise.
Ultimately, it is plausihility that is the issue, and this does not
always harmonise with what the a priori theorist takes as his or
" [n fact, model selection criteria of the sort we have mentioned (and others) are
extensively used (see, for exampl e, Burnham and Anderson 2002).
FINALE SOME GENERAL ISSUES 297
her ideal of simplicity. The same general observation also, in our
opinion, undercuts the quest for objective priors in general. Our
view, which we have stated several times already (but see that as
no reason to stop!) and believe the most natural way of interpret-
ing the Bayesian formalism, is that the latter is simply a set of
valid rules for deriving probabilistic consequences from proba-
bilistic premises. If you want accurate conclusions you should
make your assumptions as accurate (in your own eyes) as you
can. But the objectivity of the enterprise consists in the objec-
tive validity with which you draw conclusions from those
assumptions.
In this view the quest by many for 'objective' prior distribu-
tions is not only unnecessary but misconceived, a conclusion is
reinforced by the problems which arise in pursuing that quest, and
which seem to be resolvable only by the sorts of ultimately sub-
jective decision that makes the enterprise self-defeating. People,
even those possessing the same background information, and
even experts, may still have different opinions, pace Jaynes.
Trying to force this, in our view entirely legitimate, diversity of
opinions into a single uniform one is misguided Procrustean ism,
and would have deleterious consequences for the progress of sci-
ence were it to be legislated for.
It is, in addition, certainly not sensible to throwaway relevant
information, yet this is in effect just what is recommended by
those who tell us that we should always use reference priors wher-
ever possible, or give the simplest hypotheses the highest a priori
probability. But none of this means that the Bayesian theory with-
out 'objective' priors effectively imposes no constraints at all (as
has often been charged). On the contrary, the consistency con-
straints represented by the probability axioms are both stringent
and very objective, as stringent and objective as those of deduc-
tive logic. And in a theory of valid inference that is not only as
good as it gets, but quite good enough.
9.c The Old-Evidence Problem
Or is it good enough? There will always be people who object to
any theory, and the Bayesian theory is no exception to this rule.
298 C HAPTER 9
We have tried to deal with the objections which wc feel merit seri-
ous discussion. We shall end, however, with one that doesn't, but
we shall take a look at it nonetheless because it is often seriously
advanced as the most serious objection to using the theory of per-
sonal probability in any methodological role.
It goes as foll ows. The Bayesian theory is supposed to reflect
patterns of accepted reasoning from data in terms of the way the
data change one's probabilities. One type of such reasoning is
assessing the impact of cvidence on a hypothesis of data obtained
before the hypothesis was first proposed. The stock example is the
anomal ous preces sion of Mercury's perihelion, di scovered
halfway through the nineteenth century and widely regarded as
supporting Einstein 's General Theory of Rel ativity (GTR) which
was discovered (by Einstein himself) to predict it in 1915. Indeed,
this prediction arguably did more to establish that theory and di s-
place the classical theory of gravitation than either of its other two
dramatic contemporary predictions, namely the bending of light
close to the sun and the gravitational red-shift. But according to
nearly all commentators, starting with Glymour 1980, this is
something which in principle the Bayesian theory cannot account
for, since e is known then Pre) = I and it is a simpl e inference
from the probability cal culus that P(h I e) = P(h); i.e., such evi-
dence cannot be a ground/hr changing one :s' belielin h.
Despite all thi s, the 'old evidence ' objecti on' is not in fact a
serious problem for the Bayes ian theory; indeed, it is not a prob-
lem at all , certainly in principle. What it reall y demonstrates is a
failure to apply the Bayesi an formul as sensibl y, and to that extent
the 'probl em' is rather analogous to inferring that 3/ 2 = x/x = I
from the fact that 3x = 2x if x = 0. To see cl early why we need
only note an elementary fact about evidence, which is that data
do not constitute evidence for or against a hypothesis in isolation
F om a body of" ambient information. To talk about e being evi-
dence relevant to h obviously requi res a background of fact and
informati on against which e is judged to be evidence. A large
dictionary found in the street is not in itself evidence either for or
against the hypothesis that Smith killed Jones. Relative to the
background information that Jones was killed with a heavy
object, that the dictionary belonged to Smith, and that blood
found on the dicti onary matches Jones's, it is. In other words,
FINALE: SOME GENERAL ISSUES 299
'being evidence for' connotes a three-place relation, between the
data, the hypothesis in question, and a body k of background
information. The evidential weight of e in rel ation to h is
assessed by how much e changes the credibility of h, in a posi-
tive or negative directi on, given k.
Clearly, a condition of applying these obvious criteria is that k
does not contain e. Otherwise, as the old-evidence 'problem'
reminds us, e could not in principle change the credibility of h:
requiring that k not contain e, before judging its evidential import
relative to k is merely like requiring that the car engine is not
already running in any test to see whether a starter motor is work-
ing properly. Granted that, we can see that the old-evidence 'prob-
lem' really is not a problem, merely an implicit reminder that if e
is in k then it should first be deleted, as far as that can be done,
before assessing its evidential weight. I
It is often objected against thi s that there is no uniform
method for deleting an item of information from a database k,
and often it seems that there is no way at all which does not rep-
resent a fairly arbitrary decision. For example, the logical content
of the set {a, b} is identical to that of {a,a --;. b}. where a and b
are contingent propositi ons, but simply subtracting a from each
will leave two different sets of consequences; b will be in the first
and not the second, for example, if the sets are consistent. Much
has been made of thi s problem, and some have been led to
believe that the task is hopeless. Fortunately, this is far from the
truth. Suzuki (2005) has shown that there are consistent proba-
bilistic contraction funct ions which represent the deletion of e
from k relative to plausible boundary conditions on such func-
tions (these conditions are furnished by the well-known AGM
theory of belief-revision;
see Suzuki op. cit. for references). The exhibition of a particlilar
I Given the routine dismissal of the counterfactualmove in the literature, read-
ers may be surprised to learn that it is in fact standard Baycsian procedure. Once
any piece of evi dence is 'learned' it becomes 'old', and according to those who
advance the 'old evidence problem' as an objection to the Bayesian methodolo-
gy, it should no longer confirm any hypothesis ( indeed, conditionalising on e
automatically takes its probability to I). So to regard any evidence, once known,
as confirming one has to go counterfactual.
300 CHAPTER 9
probability function representing the deletion of e will in gener-
al reflect the way the agent her/himself views the problem, and it
is completely in line with the personalistic Bayesian theory
adopted in this book that the request for an objective account of
how this should be done is simply misplaced. Nevertheless, it can
often be expected that the constraints imposed by the background
information will practically determine the result, and this is cer-
tainly true for the example which prompted the discussion, the
observation of the precession of Mercury's perihelion, as we
shall now show.
The discussion follows Howson 2000, p.194. We start, appro-
priately, with Bayes's Theorem, in the form:
P(h I e) =
p + Pre I - p)
p
Pre I h)
where as before h is GTR, e is the observed data on Mercury's
perihelion (including the error bounds), p = P(h), and P is like the
agent's probability function except that it does not 'know' e.
Following category theory, we could call P the 'forgetful functor' ,
meaning in this case that it has 'forgotten' e. We shall now show
that, despite this idea sounding too vague to be useful, or even,
possibly, consistent, the data of" the problem are sufficient to
determine all the terms in the equation above. at any rate to with-
in fairly tight bounds.
Firstly, we have by assumption that h, together with the resid-
ual background information which P is assumed to 'know',
entails e, so pre I h) = I by the probability axioms
of" any particular characteristic of" P. Thus the equation above
becomes
P
P(h I e) = ------
p + pre I - p)
and now we have only p and P(e I to consider. If we were to
expand out pre I we would find that it is a constant less than
1 multiplied by a sum whose terms are products pre I h)P(h),
FINALE: SOME GENERAL ISSUES 301
where hi are alternatives to h. Recall now the assumption, reason-
ably appropriate to the situation in 1915, that the only serious
alternative to GTR was Classical Gravitation Theory (CGT),
meaning that it is the only hi apart from h itself such that P(h) is
not negligible. Now we bring in the additional assumption that P
does not 'know' e. Judged on the residual background informa-
tion alone, the fact that e is anomalous relative to CGT means
therefore that pre I -h) will be vety small, say E.
We are now almost there, with just p itself to evaluate.
Remember that this too is to be evaluated on the residual back-
ground information. Without any of the confirming evidence for
h, including e, this should mean that p, though small by compari-
son with P(CGT), which is correspondingly large (e is now not an
anomaly for CGT, since by assumption e does not exist), is not
negligibl e. It follows that, because of the very large likelihood
ratio in favour of h combined with a non-negligible if small prior
probability, P(h I e) is much larger than p = P(h), and we see that
h is correspondingly highly confirmed bye, even though e is
known. The 'old evidence' problem is solved.
9.d Conclusion
Our view, and we believe the only tenable view, of the Bayesian
theory is of a theory of consistent probabilistic reasoning. Just as
with the theory of deductive consistency, this gives rise automat-
ically to an account of valid probabili stic inference, in which the
truth, rati onality, objectivity, cogency or whatever of the premis-
es, here prior probability assignments, are exogenous considera-
tions, just as they are in deductive logic. Not only are these
features outside the scope of the theory, they are, for the reasons
we have given, incapabl e of being given any coherent or sustain-
able interpretation in any case.
This is not to say that what we have presented herc is the last
word. Modesty alone would preclude this, but it is almost cer-
tainly anyway not true: the model of uncertain reasoning in this
account is a crude and simple one, as crude and simple as the
usual models of deductive inference. But it has also the explana-
tory strengths of these models which, crude as they are, still
302 CHAPTER 9
dominate and mould discussions of deductive reasoning, and will
continue to do so, in one version or another, for the foreseeable
future. Which is saying a great deal. Enough, indeed, to end this
book.
Bibliography
Akaike, H. 1973. Information Theory and an Extension of the Maximum
Likelihood Principle. In Second International Symposium 0/
Inlormation Theory, cds. B.N. Petrov and F Csaki (Budapest:
Akademiai Kiad6), 267-28l.
Anscombe, Fl. 1963. Sequential Medical Trials. Journal ol the
American Statistical Association, Volume 58, 365-383.
Anscombe, Fl., and R.l. Aumann. 1963. A Definition of Subjective
Probability. Annals o/Mathematical Statistics, Volume 34, 199-205.
Armitage, P. 1975. Sequential Medical Trials. Second edition. Oxford:
Blackwell.
Atkinson, A.C. 1985. Plots, Trans/ormations, and Regression. Oxford:
Clarendon.
---. 1986. Comment: Aspects of Diagnostic Regression Analysis.
Statistical Science, Volume I, 397--402.
Babbage, C. 1827. Notice Respecting some Errors Common to many
Tables of Logarithms. Memoirs o(the Astronomical Society, Volume
3,65-67.
Bacon, F 1994 [1620]. Novum Organum. Translated and edited by P.
Urbach and l. Gibson. Chicago: Open Court.
Barnett, V 1973. Comparative Statistical In/erence. New York: Wiley.
Bartha, P. 2004. Countable Additivity and the de Finetti Lottery. British
Journal/hr the Philosophv o/Science, Volume 55, 301-323.
Bayes, T. 1958 [1763]. An Essay towards Solving a Problem in the
Doctrine of Chances. Philosophical Transactions of the Royal
Societv, Volume 53, 370--418. Reprinted with a biographical note by
G.A. Barnard in Biometrika (1958), Volume 45, 293-315.
Belsley, D.A., E. Kuh, and R.E. Welsch. 1980. Regression Diagnostics:
Identifi 'ing In/luential Data and Sources of Collinearitv. New York:
Wiley.
Bernoulli, D. 1738. Specimen theoriae novae de mensura sortis.
Commentarii academiae scientiarum imperialis Petropolitanae,
Volume V, 175-192
Bernoulli, l. 1713. Ars Conjectandi. Basiliae.
304 BIBLIOGRAPHY
Berry, D.A. 1989. Ethics and ECMO. Statistical Science, Volume 4,
306-310.
Blackwell , D. , and L. Dubins. 1962. Merging of Opinions with
Increasing Information. Annals oj'Mathematical Statistics, Volume
33, 882-87.
Bland, M. 1987. An Intmduction to Medical Statistics. Oxford: Oxford
University Press.
Blasco, A. 200 I. The Bayesian Controversy in Animal Breeding. Journal
oIAnimal Science, Volume 79, 2023-046.
Bovens, L. and S. Hartmann. 2003. Bayesian Epistemology. Oxford:
Oxford University Press.
Bourke, G.J., L.E. Daly, and 1. McGilvray. 1985. interpretation and Us'es
oj'Medical Statistics. 3rd edition. St. Louis: Mosby.
Bowden, B,Y 1953. A Brief History of Computation. In Faster than
Thought, edited by B.Y Bowden. London: Pitman.
Bradley, R. 1998. A Representation Theorem for a Decision Theory with
Conditionals. Synthese, Volume 116, 187-229.
Brandt, R. 1986. 'Comment ' on Chatterjee and Hadi (1986). Statistical
Science, Volume 1, 405-07.
Broemeling, L.D. 1985. Bayesian Analysis oj'Linear Models. New York:
Dekker.
Brook, R.J. and G.C Arnold. 1985. Applied Regression Analysis and
Experimental Design. New York: Dekker.
Burnham, K.P. and D.R. Anderson. 2002. Model Selection and
Multimodellnferellce: A Practicallnfimnation- Theoretical Approach.
New York: Springer-Verlag.
Byar, D.P et al. (seven co-authors), 1976. Randomized Clinical Trials.
New England Journal of Medicine, 74-80.
Byar, D.P et al. (22 co-authors). 1990. Design Considerations for AIDS
Trials. New England Journal oj'Medicine, Volume 323, 1343-48.
Carnap, R. 1947. On the Applications of Inductive Logic. Philosophy
and Phenomenological Research, Volume 8, 133 148.
Casscclls w., A. Schoenberger, and T. Grayboys. 1978. Interpretation by
Physicians of Clinical Laboratory Results. New England Journal oj'
Medicine, Volume 299, 999-1000.
Chatterjee, S., and A.S. Hadi. 1986. Influential Observations, High
Leverage Points, and Outliers in Linear Regression. Statistical
Science, Volume I , 379--416.
Chatterjee, S. , and B. Price. 1977. Regression A n azvs is by Example. New
York: Wiley.
Chiang, CL. 2003. Statistical Methods oj' Analysis. World Scientific
Publishing.
BIBLIOGRAPHY 305
Cochran, w.G. 1952. The X
2
Test of Goodness of Fit. Annals of
Mathematical Statistics, Volume 23, 315-345.
---. 1954. Some Methods for Strcngthening thc Common X2 Tests.
Biometrics, Volume 10, 417-451.
Cook, R.D. 1986. Comment on and Hadi 1986. Statistical
Science, Volume 1, 393-97.
Coumot, A.A. 1843. Exposition de la Thcorie des Chances et des
Probabilitcs. Paris.
Cox, D.R. 1968. Notes on Some Aspects of Regression Analysis.
journal of the Royal Statistical Society, Volume 131 A, 265-279.
Cox, R.T. 1961. The Algebra of Probable Inference. Baltimore: The
Johns Hopkins University Press.
Cramer, H. 1946. Mathematical Methods of Statistics. Princeton:
Princeton University Press.
Daniel, c., and FS. Wood. 1980. Fitting Equations to Data. New York:
Wiley.
David, FN. 1962. Games, Gods, and Gambling. London: Griffin.
Dawid, A.P. 1982. The Well-Calibrated Bayesian. Journal of the
American Statistical Association, Volume 77, 605-613.
Diaconis, P. , and S.L. Zabel!. 1982. Updating Subjective Probability.
journal of the American Statistical Association. Volume 77,
822-830.
Dobzhansky, T. 1967. Looking Back at Mendel's Discovery. Science,
Volume 156, 1588-89.
Dorling, J. 1979. Bayesian Personalism, the Methodology of Research
Programmes, and Duhem's Problem. Studies in Historv and
Philosophy ofSciel1ce, Volume 10, 177-187.
---. 1996. Further Illustrations of the Bayesian Solution of Duhem's
Problem.
Downham, .I., cd. 1988. issues in Political Opinion Polling. London: The
Market Research Society. Occasional Papers on Market Research.
Duhem, P. 1905. The Aim and Structure ofPhvsical Theon'. Translated
by P.P. Wiener, 1954. Princeton: Princeton University Press.
Dunn, 1M., and G. Hellman. 1986. Dualling: A Critique of an Argument
of Popper and Miller. British Journal/or the Philosophy of Science.
Volume 37, 220-23.
Earman, J. 1992. Bayes or Bust? A Critical Examination of Bayesian
Confirmation Theory. Cambridge, Massachusetts: MIT Press.
Edwards, A.L. 1984. An Introduction to Lineal' Regression and
Correlation. Second edition. New York: Freeman.
Edwards, A. W.F 1972. Likelihood. Cambridge: Cambridge University
Press.
306 BIBLIOGRAPHY
---. 1986. Are Mendel 's Results Really Too Closc? Biological
Reviews ol the Cambridge Philosophical Society, Volume 61,
295-312.
Edwards, W 1968. Conservatism in Human Information Proccssing. In
Formal Representation ol Human ./udgment. B. Kleinmuntz, cd.,
17-52.
Edwards, W, H. Lindman, and L.J. Savage. 1963. Bayesian Statistical
Inference for Psychological Research. P.I)'ch%gical Review,
Volume 70, 193-242.
Ehrenberg, A.S.C. 1975. Data Reduction: Analvsing and Interpreting
Statistical Data. London: Wiley.
FDA. 1988. Guideline/or the Format and Content ol the Clinical and
Statistical Sections olNelv Drug Applications. Rockville: Center for
Drug Evaluation and Research, Food and Drug Administration.
Fellcr, W 1950. An introduction to Probability TheOlY and its
Applications, Volume I. Third edition. New York: Wiley.
Feyerabend P. 1975. Against Method. London: New Left Books.
Finetti , B. de. 1937. La prevision; ses lois logiques, ses sourccs subjec-
ti ves. Annale.l de I 'lnstitut Henri Poincare, Volumc 7, 168.
Reprinted in 1964 in Engl ish translation as 'Foresight: Its Logical
Laws, its Subjective Sources', in Studies in Subjective Probability,
edited by H.E. Kyburg, Jr. , and H.E. SmokIer (Ncw York: Wiley).
---. 1972. Probability. Induction. and Statistics, New York: Wiley.
---. 1974. Themy olProbability. Volume I. New York: Wiley.
Fisher, R.A. 1922. On the Mathematical Foundations of Theoretical
Statistics. Philosophical Transactions oj' the Royal Society oj'
London. Volume A222, 309-368.
---. 1930. Inverse Probability. Proceedings ol the Camhridge
Philosophical Society, Volume 26, 528 535.
---. 1935. Statistical Tests. Nature. Volume 136,474.
---. 1936. Has Mendel 's Work Been Rediscovered? Annals ol
Science, Volume 1, 115-137.
---. 1947 [1926]. The Design ol E'periments. Fourth edition.
Edinburgh: Oliver and Boyd.
---. 1956. Statistical Methods and Statisticallnlerence. Edinburgh:
Oliver and Boyd.
"--. 1970 [1925]. Statistical Methods for Research Workers.
Fourteenth edition. Edinburgh: Oliver and Boyd.
Freeman, P.R. 1993. The Role of P-valucs in Analysing Trial Results.
Statistics in Medicine, Volumc 12, 1433 459.
Gabbay, D. 1994. What Is a Logical System? What Is a Logical
ed. D. Gabbay, Oxford: Oxford University Prcss, 179-217.
BIBLI OGRAPHY 307
Gaifman, H. 1964. Concerning Measures in First Order Calculi. israel
Journal of Mathematics, Volume 2. 1-18.
- --. 1979. Subjective Probability, Natural Predicates, and Hempel's
Ravens. Erkenntnis, Volume 14, 105-159.
Gaifman, H., and M. Snir. Probabilities over Rich languages. Testing and
Randomness. Journal of Symbolic Logic 47, 495-548.
Giere, R. N. 1984. Understanding Scientific Reasoning. Second edition.
New York: Holt, Rinehart.
Gigerenzer, G. 1991. How to Make Cognitive Illusions Disappear:
Beyond Heuristics and Biases. European Review of Social
Psvchology, Volume 3,83-115.
Gilli es, D.A. 1973. An Objective Theorv of Probabilitv. London:
Methuen.
- '--. 1989. Non-Bayesian Confirmation Theory and the Principle of
Explanatory Surplus. Philosophy o{Science Association 1988, edit-
ed by A. Finc and J. Loplin. Volume 2 (Pittsburgh: Pittsburgh
University Press), 373- 381.
----. 1990. Bayesianism versus Falsificationism. Ratio, Volume 3,
82- 98.
- --. 2000. Philosophical Th eories of Probahility. London:
Routledge.
Girotto, V, and M. Gonzalez. 200 I. Solving Probabilistic and Statistical
Problems: A Matter of Information Structure and Question Form.
Cognition, Volume n, 247-276.
Glymour, e. 1980. Th e(JlY and Evidence. Princeton: Princeton
Uni versity Press.
Good, I.J. 1950. Prohabilitv and the Weighing of Evidence. London:
Griffin.
- --. 1961. The Paradox of Confirmation. British Journal f hr the
Philosophy o(Science, Volume I I. 63-64.
1965. The Estimation oj Probabilities. Cambridge,
Massachusetts: MIT Press.
- '--. 1969. Di scussion of Bruno de Finetti's Paper 'Initial
Probabilities: A Prerequisite for any Valid Induction'. Svnthese,
Volume 20. 17-24.
- --. 1981 . Some Logic and History of Hypothes is Testing. In
Philosophical r(J/lndations o( Economics, edited by .I.e. Pitt
(Dordrecht: Reidel).
- --. 1983. Some Hi story of the Hierarchical Bayes Methodology.
Good Thinking. Minneapoli s: Uni versit y of Minnesota Press,
95- 105.
308 BIBLIOGRAPHY
Goodman, N. 1954. Fact, Fiction, and Forecast. London: Athlone.
Gore, S.M. 1981. Assessing Clinical Trials: Why Randomize'! British
Medical journal, Volume 282, 1958- 960.
Grunbaum, A. 1976. Is the Method of Bold Conj ectures and Attempted
Refutations justifiably the Method of Science? British journal fCJr
the Philosophy (JjScience, Volume 27, 105- 136.
Gumbel , E.J. 1952. On the Reliability of the Classical Chi-Square Test.
Annals of Mathematical Statistics, Volume 23, 253- 263.
Gunst, R.F., and R.e. Mason. 1980. Regression Analvsis and its
Application. New York: Dekker.
Hacking, I. 1965. Logic of Statistical Inlerence. Cambridge: Cambridge
University Press.
---. 1967. Sli ghtl y More Real istic Personal Probability. Philosophv
of 5;cience, Volume 34, 311- 325.
---. 1975. The Emergence of Probability. Cambridge: Cambridge
Uni versity Press.
Halmos, P. 1950. Measure TheOl)'. New York: Van Nostrand.
Halpern, J.y. 1999. Cox's Theorem Revisited. j ournal of Artificial
Intelligence Research, Volume 11,429 435.
Hays, WL. 1969 [1 963]. Statistics. London: Holt, Rinehart and Winston.
Hays, WL., and R.L. Winkler. 1970. Statistics: Probabilitv, inference,
and Decision, Volume I. New York: Holt, Rinehart.
Hellman, G. 1997. Bayes and Beyond. Philosophv of Science, Volume
64.
Hempel , e.G. 1945. Studies in the Logic of Conf irmation. Mind,
Volume 54, 1- 26, 97- 121. Reprinted in Hempel 1965.
---. 1965. A,\jJecls of Scientific Explanation. New York: The Free
Press.
---. 1966. Philosophy of Nalural Science. Englewood Cliffs:
Prentice-Hall.
Hodges, J.L., Jr., and E.L. Lehmann. 1970. Basic Concepts of
Probabilitl' and Statistics. Second edition. San Francisco: Holden-
Day.
Horwich, P. 1982. Prohabilitv and Evidence. Cambri dge: Cambridge
Uni versity Press.
---. 1984. Bayes iani sm and Support by Novel Facts. British journal
for the Philosophy ()fSciel1ce. Volume 35, 245- 251 .
Howson, e. 1973. Must the Logical Probability of Laws be Zero? British
jOl/rnalfiJl' the Philosophy of Science. Volume 24, 153-163.
Howson, e., cd. 1976. Method and Appraisal in the Physical Sciences.
Cambridge: Cambridge University Press.
BIBLIOGRAPHY
309
---. 1987. Popper, Prior Probabilities, and Inductive Inference.
British Journal/hr the Philosophy of Science, Volume 38, 207-224.
--. 1988a. On the Consistency of Jeffreys's Simplicity Postulate,
and its Role in Bayesian Inference. Philosophical Quarterlv, Volume
38,68- 83.
--- . 1988b. Accommodation, Prediction, and Bayesian Confirmation
Theory. PSA 1988. A. Fine and.l. Leplin, eds. , 381 - 392.
---. 1997. Logic With Trees. London: Routledge.
---. 2000. Problem: Induction and the Justification of
Belief Oxford: Clarendon.
---. 2002. Bayesianism in Statistics. Bayes :\. Theorem, ed. R.
Swinburne, The Royal Academy: Oxford University Press, 39-71.
Hume, D. 1739. A Treatise of Human Nature, Books I and 2. London:
Fontana.
---. 1777. A n Concerning Human Understanding. Edited
by L.A. Selby-Bigge. Oxford: Clarendon.
Jaynes, E.T. 1968. Prior Probabilities. institute of Electricaf and
Electronic Engineers Transactions on Systems Science and
(vbernetics, SSC-4, 227-241.
---. 1973. The Well-Posed Problem. Foundations of Physics,
Volume 3, 413- 500.
--. 1983. Papers on Probability, Statistics, and Statistical Physics,
edited by R. Rosenkrantz. Dordrecht: Reidel.
---. 1985. Some Random Observations. Synthese, Volume 63,
115- 138.
---. 2003. Probabilitv Theorv: The Logic of Science. Cambridge:
Cambridge University Press.
Jeffrey, R.C. 1970. 1983. The Logic of Decision. Second edition.
Chicago: University of Chicago Press.
---. 2004. Subjective The Real Thing. Cambridge:
Cambridge University Press.
Jeffreys, H. 1961. Theory of Probability. Third edition. Oxford:
Clarendon.
Jennison, c. , and B. W. Turnbull. 1990. Stati stical Approaches to Interim
Monitoring: A Review and Commentary. Statistical Science, Volume
5,299- 317.
Jevons, W.S. 1874. The Principles of Science. London: Macmillan.
Joyce, J.M. 1998. A Nonpragmatic Vindication of Probabilism.
Philosophy ()fScience, Volume 65, 575- 603.
---. 1999. The Foundations of Causal Decision Theory. Cambridge:
Cambridge University Press.
310 BIBLIOGRAPHY
Kadane, 1., et al. 1980. Interactive Elicitation of Opinion for a Normal
Linear Model. journal of" the American Statistical Association.
Volume 75, 845- 854.
Kadane, .l.B. , M.l. Schervish, and T. Seidenfeld. 1999. Rethinking the
Foundations o(Statistics. Cambridge: Cambridge University Press.
Kadane, lB. and T. Seidenfeld. 1990. Randomi zation in a Bayesian
Perspective. j ournal 0/ Statistical Planning and Inference, Volume
25, 329-345.
Kant, I. 1783. Prolegomena to any FUlllre Metaphysics. Edited by L.w.
Beck, 1950. Indi anapolis: Bobbs-Merrill.
Kempthorne, O. 1966. Some Aspects of Experimental Inference.
journal o/the American Statistical Association. Volume 61, 11 --34.
- ---. 1971. Probability, Statistics, and the Knowledge Business. In
fOllndations of Statistical Inference. edited by VP. Godambe and
D.A. Sprott. Toronto: Holt, Rinehart and Winston of Canada.
-- --. 1979. The Design and Analysis 0/ E'periments. Huntington:
Robert E. Kri eger.
Kendall , M.G., and A. Stuart. 1979. The Advanced Theory o/Statistics.
Volume 2. Fourth edition. London: Griffin.
- - . 1983. The Advanced Theory o/Statistics. Volume 3. Fourth edi-
tion. London: Griffin.
Keynes, l.M. 1921. A Treatise on Probability. London: Macmillan.
Ki eseppa, LA. 1997. Akaike Information Criterion, Curve-fitting, and
the Philosophi cal Problem of Simplicity. British JournalJiJr the
Philosophy o/Science, Volume 48, 21-48.
Kitcher, P. 1985. Vault ing Ambition. Cambridge, Massachusetts: MIT
Press.
Kolmogorov, A.N. [950. rCJUndations of the Theory of" Probability.
Translated from the German of 1933 by N. Morrison. New York:
Chelsea Publi shing. Page references are to the 1950 edition.
Korb, K.B. 1994. Infinitely Many Resolutions of Hempel's Paradox . [n
Theoretical Aspects of" Reasoning abow Kllol1'ledge, 13849, edited
by R. Fagin. Asilomar: Morgan Kaufmann.
Kuhn, T.S. [970 [1 962]. The Structure of"Scielllific Revolutions. Second
edition. Chicago: University of Chicago Press.
Kyburg, H.E. , Jr. , and E. Smoki er, eds. 1980. Studies ill Subjective
Probability. Huntington: Krieger.
Lakatos, L 1963. Proofs and Refutations. Brilish Journal .lor the
Philosophv 0/ Science, Volume 14, 1-25, 120- 139, 221-143, 296,
432.
- --' . 1968. Criticism and the Methodology of Scientific Research
Programmes. Proceedings of" the Aristotelian Societv. Volume 69,
149-186.
BIBLIOGRAPHY 311
---. 1970. falsification and the Methodology of Scientific Research
Programmes. In Criticism and the Growth of Knowledge, edited by
I. Lakatos and A. Musgrave. Cambridge: Cambridge University
Press.
---. 1974. Popper on Demarcation and Induction. In The Philosophy
of Karl Popper, edited by PA. Schilpp. La Salle: Open Court.
---. 1978. Philosophical Papers. Two volumes. Edited by 1. Worrall
and G. Currie. Cambridge: Cambridge University Press.
Laplace, PS. de. 1820. Essai Philosophique sllr les Prohabilites. Page
references arc to Philosophical Essay 011 Probabilities, 1951. New
York: Dover.
Lee, PM. 1997. Bayesian Statistics. Second edition. London: Arnold.
Lewis, D. 1981. A Subjectivist's Guide to Objective Chance. In Studies
in Inductive Logic and Probabili(l', edited by R.c:. Jeffrey, 263-293.
Berkeley: University of California Press.
Lewis-Beck, M.S. 1980. Applied Regression. Beverley Hills: Sage.
Li, M. and PB.M. Vitanyi. 1997. All Introduction to Ko/mogorov
Complexity Theorv and its Applications. Second edition. Berlin:
Springer.
Lindgren, B.W. 1976. Statistical Third edition. New York:
Macmillan.
Lindley, D.V 1957. A Statistical Paradox. Biometrika, Volume 44,
187-192.
---. 1965. Introduction to Probability and Statistics, from a
Bayesian Viewpoint. Two volumes. Cambridge: Cambridge
University Press.
---. 1970. Bayesian Analysis in Regression Problems. In Bayesian
Statistics, edited by D.L. Meyer and R.O. Collier. Itasca: FE.
Peacock.
----. 1971. Bayesian Statistics: A Review. Philadelphia: Society for
Industrial and Applied Mathematics.
---. 1982. The Role of Randomization in Inference. Philosophy of
Science Association, Volume 2, 431-446.
---. 1985. Making Decisions. Second edition. London: Wiley.
Lindley, D.V, and G.M. EI-Sayyad. 1968. The Bayesian Estimation of a
Linear functional Relationship. Journal of' the Roml Statistical
Societv, Volume 30B, 190-202.
Lindley, D.V, and L.D. Phillips. 1976. Inference for a Bernoulli Process
(a Bayesian View). American Statistician, Volume 30,112-19.
Mackie,1.L. 1963. The Paradox of Confirmation. British Journalj(J/' the
Philosophy of Science, Volume 38, 265-277.
312 BIBLIOGRAPHY
Mcintyre, I.M. e. 199 L Tribulations for Clinical Trials. British Medical
journal, Volume 302, I 1100.
Maher, P. 1990. Why Scientists Gather Evidence. British journal/or the
Philosophy olScience, Volume 41 ,
--- 1990. Acceptance Without Belief. PSA 1990, Volume I, eds. A.
Fine, M. Forbes, and L. Wessels,
--- 1997. Depragmatized Dutch Book Arguments. Philosophy ol
Science, Volume 64, 291-305.
Mallet, J W 1880. Revision of the Atomic Weight of Aluminium.
Philosophical Transactions, Volume 17 1, I
--- . 1893. The Stas Memorial Lecture. In Memorial Lectures deliv-
ered hejhre the Chemical Society 1893-1900. Published 190L
London. Gurney and Jackson.
Mann, H.B. , and A. Wald. 1942. On the Choice of the Number of
Intervals in the Application of the Chi-Square Test. Annals ()f
Mathematical Statistics, Volume 13, 306 -317.
Mayo, D.G. 1996. Error and the Grml'th of Experimental Knowledge.
Chicago: University of Chicago Press.
Medawar, P. 1974. More Unequal than Others. New Statesman, Volume

Meier, P. 1975. Statistics and Medical Experimentation. Biometrics,
Volume 31,
Miller, D. 199 L On the Maximization of Expected Futility. PPE
Lectures, Lecture 8. Department of Economics: University of
Vienna.
Miller, R. 1987. Messiah. London: Michael Joseph.
Mises, R. von. 1939 [1928]. Probahility, Statistics. and Truth. First
English edition prepared by H. Geiringer. London: Allen and Unwin.
--- . 1957. Second Engl ish edition, revised, of Probahility, Statistics
and Tmth.
--- . 1964. Mathematical TheolY ol {lnd Statistics. New
York: Academic Press.
Mood, A. M. 1950. Introduction to the Theon; of Statistics. New York:
McGraw-Hill.
MOOlL A.M., and EA. Graybill. 1963. Introduction to the Theorv of
Statistics. New York: McGraw-Hill.
Musgrave. A. 1975. Popper and 'Diminishing Returns from Repeated
Tests' , Australasian journal olPhilosophy, Volume 53,
Myrvold, We. and WL. Harper. 2002. Model Selection and Scientific
Inference. Philosophy ol Science. Volume 69, S 134.
Neyman, J 1935. On the Two Different Aspects of the Representative
Method: the Method of Stratified Sampling and the Method of
Purposive Selection. Reprinted in Neyman 1967, 98-- 14 L
BIBLIOGRAPHY 313
- - - - . 1937. Outline of a Theory of Statistical Estimation Based on the
Classical Theory of Probability. Philosophical Transactions oj the
Royal Societv. Volume 236A, 333-380.
- - - -. 1941. Fiducial Argument and the Theory of Confidence
Intervals. Biometrika, Volume 32, 128--150. Page references are to
the reprint in Neyman 1967.
- - -. 1952. Lectures and Conferences on Mathematical Sialistics and
Probahilitv. Second edition. Washington, D.C.: U.S. Department of
Agriculture.
- - -. 1967. A Selection oj a r ~ v Statistical Papers oj J Neyman.
Cambridge: Cambridge University Press.
Neyman, J. , and E.S. Pearson. 1928. On the Use and the Interpretation
of Certain Test Criteri a for Purposes of Statistical Inference.
Biometrika, Volume 20, 175240 (Part I), 263-294 (Part II).
- - - . 1933. On the Probl em of the Most Efficient Tests of Statistical
Hypotheses. Philosophical Transactions oj the Royal Society,
Volume 231A, 289-337. Page references are to the reprint in
Neyman and Pearson's Joint Statistical Papers (Cambridge:
Cambridge University Press, 1967).
Pais, A. 1982. Subtle is the Lord. Oxford: Clarendon.
Pari s, 1. 1994. The Uncerlain Reasoner:1' Companion. Cambridge:
Cambridge University Press.
Pari s, J. and A. Vencovska. 200 I. Common Sense and Stochastic
Independence. Foundations oj Bayesianisl11, cds. D. Corfield and J.
Williamson. Dordrecht: Kluwer, 203- 241.
Pearson, E.S. 1966. Some Thoughts on Statistical Inference. In The
Selected Papers oj E.5. Pearson, 276-183. Cambridge: Cambridge
University Press.
Pearson, K. 1892. The Grammar oj Science. Page references are to the
edition of 1937 (London: Dent).
Peto, R., et 01. 1988. Randomised Trial of Prophylactic Daily Aspirin in
British Male Doctors. British Medical Journal, Volume 296,
3 13-331.
Phillips, L.D. 1973. Bayesian Sialislics for Social Scielltisls. London:
Nelson.
- ---. 1983. A Theoretical Perspective on Heuri stics and Biases in
Probabilisti c Thinking. In Analysing and Aiding Decision. edited by
Pc. Humphreys, O. Svenson, and A. Van. Amsterdam: North
Holland.
Pitowsky, I. 1994. George Boole's Conditions of Possibl e Experience
and the Quantum Puzzle. Brilish Journal trw Ihe Philosophy oj
Science, Volume 45, 95- 127.
314 BIBLIOGRAPHY
Poincare, H. 1905. Science and Hvpothesis. Page referenccs are to the
edition of 1952 (Ncw York: Dover).
Polanyi, M. 1962. Personal Knowledge. Second edition. London:
Routledge.
Pollard, W 1985. Bayesian Statistics jhr Evaluation Research: An
Introduction. Bevcrly Hills: Sage.
Pol ya, G. 1954. Mathematics and Plausihle Reasoning. Volumes I and
2. Princeton: Princeton University Press.
Popper, K.R. 1959. The Propensity Interpretation of Probability. British
Journal/hI' the Philosophy of Science. Volume 10, 25--42.
---. 1959a. The Logic ofScienti/lc DiscoverT. London: Hutchinson.
---. 1960. The Poverty of Historicism. London: Routledge.
---. 1963. Conjectures and Re/illations. London: Routledge.
~ . . 1972. Objective KmHv/edge. Oxford: Oxford University Press.
---. 1983. A Proof of the Impossibility of Inductivc Probability.
Nature, Volume 302, 687-88.
Pratt, J. W 1962. On the Foundations of Statistical Inference. Journal of
the American Statistical Associatioll. Volume 57, 269-326.
1965. Bayesian Interpretation of Standard Infercncc
Statcments. Journal of the Royal Statistical Socielv. 278, 169-203.
Pratt, J.W, H. Raiffa, and R. Schlaifer. 1965. Introduction to Statistical
Decision Theon'.
Prout, W 1815. On thc Rclation Bctwccn thc Spccific Gravities of
Bodies in Their Gascous Statc and thc Wcights of Their Atoms.
Annals of Philosophy, Volumc 6, 321-330. Rcprinted in Alembic
Club Reprints, No. 20, 1932,25-37 (Edinburgh: Olivcr and Boyd).
Prout, W. 1816. Correction of a Mistake in the Essay on thc Relations
Between the Specific Gravities of Bodies in Their Gaseous State and
the Weights of their Atoms. Annals 0/ Philosophy, Volumc 7,
111-13.
Putnam, H. 1975 . Collected Papers, Volumc 2. Cambridge: Cambridgc
Univcrsity Press.
Ramsey, FP 1931. Truth and Probability. In Ramsey. The Foundations oj'
Mathematics and Other Logical Essa\'s (London: Routledge).
Rao, e.D. 1965. Linear Statistical Inference and its Applications. New
York: Wiley.
Renyi, A. 1955. On a New Axiomatic Theory of Probability. Acta
A1athematica Academiae Scientiarul11 Hungaricae, Volumc VI,
285-335.
Rosenkrantz, R.D. 1977. Ill/erence. Method. and Decision: Towards a
Bayesian Philosophy a/Science. Dordrecht: Reidel.
Salmon, We. 1981. Rational Prediction. British Journal jc)r the
Philosophv of Science. Volume 32, 115-125.
BIBLIOGRA PHY 315
Savage, L.1. 1954. The Foundations of Statistics. New York: Wiley.
---. 1962. Subjective Probability and Statistical Practice. In The
Foundations of Statistical In/Crence, edited by G.A. Barnard and
D.R. Cox (New York: Wiley). 9-35.
---. 1962a. A Prepared Contribution to the Discussion of Savage
1962, 88- 89, in the same volume.
Schervish, M., T. Seidenfeld and 1.B. Kadane. 1990. State-Dependent
Utiliti es. Journal of the American Statistical Association, Volume 85 ,
840-847.
Schroeder, L.D. , D.L. Sjoquist, and P.E. Stephan. 1986. Understanding
Regression Analysis. Beverly Hills: Sage.
Schwarz, G. 1978. Estimati ng the Dimension of a Model. Annals of
Statistics, Volume 6, 46 1--464.
Schwartz, D. , R. Flamant and 1. Lcllouch. 1980. Clinical Trials [L'essay
therapeutique che:: I 'hommej. New York: Academic Press.
Transl ated by M.J.R. Healy.
Scott, D. and P. Krauss. 1966. Assigning Probabilities to Logical
Formul as. Aspects of/nductive Logic, cds. J. Hintikka and P. Suppes.
Amsterdam: North Holl and, 219-264.
Seal , H.L. 1967. Thc Hi storical Devel opment of the Gauss Linear
Model. Biometrika, Volume 57, 1-24.
Seber, G.A.F. 1977. Linear Regression Ana"vsis. New York: Wiley.
Seidenfeld, T. 1979. Philosophical Problems of Statistical Inference.
Dordrecht: Reidel.
---. 1979. Why I Am Not an Objecti ve Bayesian: Some Reflections
Prompted by Rosenkrantz. and Decision, Volume 11 ,
413-440.
Shimony, A. 1970. Scientific Inference. In Pittsburgh Studies in the
Philosophy of Science. Volume 4, edited by R.G. Colodny.
Pittsburgh: Pittsburgh University Press.
---. 1985. The Status of the Principle of Maximum Entropy.
Volume 68, 35- 53.
--. 1993 [1988]. An Adamite Derivat ion of the Principles of the
Calculus of Probability. In Shimony, The Search fhl' a Naturalistic
Vie.I', Volume I (Cambridge: Cambridge Uni versity Press),
151-1 62.
Shore, J.E. and R.W. Johnson. 1980. Axiomatic Derivation of the
Principle of Maximum Entropy and the Principle of Minimum
Cross- Entropy. IEEE Transactions on Infimnatiof} Them:1' 26: I,
26-37.
Skynns, B. 1977. Choice and Chance. Belmont : Wadsworth.
Smart, W.M. 1947. John Couch Adams and the Discovery of Neptune.
Occasional Notes of'the Royal Astronomical Societv, No. 11.
316 BIBLIOGRAPHY
Smith, T.M. 1983. On the Validity of Inferences from Non-random
Samples. Journal of the Royal Statistical Society. Volume 146A,
394-403.
Smullyan, R. 1968. First Order Logic. Berlin: Springer.
Sober, E. and M, Forster. 1994. How to Tell When Simpler, More
Unified, Or Less Ad Hoc Theories Will Provide More Accurate
Predictions. British Journal/hr the Philosophy ~ f Science, Volume
45, 1- 37.
Spielman, S. 1976. Exchangeability and the Certai nty of Objective
Randomness. Journal of Philosophical Logic, Volume 5, 399--406.
Sprent, P1969. Models in Regression. London: Methuen.
Sprott, W..l.H. 1936. Review of K. Lewin 's A Dynamical Theory of
Personality. Mind, Volume 45, 246- 251.
Stachel, .I. 1998. Einstein:1' Miraculous Year: Five Papers Ihat Changed
the fc[(;e of Physics. Princeton: Princeton University Press.
Stas, .l.S. 1860. Researches on the Mutual Relations of Atomic Weights.
Bulletin de I Acadcmie Royale de Belgique, 208- 336. Reprinted in
part in Alembic Cluh Reprints, No. 20, 1932 (Edinburgh: Oliver and
Boyd),41--47.
Stuart, A 1954. Too Good to Be True. Applied Statistics. Volume 3,
29-32.
"---. 1962. Basic Ideas ofScienfijic Sampling. London: Griffin.
Sudbery, A. 1986. Quantum Mechanics and the Particles oj' Nature.
Cambridge: Cambridge University Press.
Suzuki, S. 2005. The Old Evidence Problem and AGM Theory. Allnals
of the Japan Association for Philosophy oj'Science, 120.
Swinburne, R.G. 1971. The Paradoxes of Confirmation: A Survey.
American PhilosophicaL Quarterlv, Volume 8, 318 329.
Tanur, 1M .. et al. 1989. Statistics: A Gliide to the Unknown. Third
Edition. Duxbury Press.
Teller, P 1973. Conditionalisation and Observation. Srllthese. Volume
26,218- 258.
Thomson, T. 1818. Some Additional Observations on the Weights of the
Atoms of Chemical Bodies. Annals of Philosoph.\'. Volume 12,
338- 350.
Uffink, J. 1995. Can the Maximum Entropy Method be Explained as a
Consistency Requirement? Studies ill the His/on-and Philosophy of
Modern Pln'sics, Volume 268, 223- 261 .
Urbach, P. 1981. On the Uti lity of Repeating the 'Same Experiment'.
Australasian./o!/J'I1al oj'Philosopliy, Volume 59, 151- 162.
---, 1985 . Randomization and the Design of Experiments.
Philosophy of Science, Volume 52, 256 273 .
BIBLIOGRAPHY 317
--- -. 1987. Francis Philosophy oj Science. La Salle: Open
Court.
- - - . 1987a. Clinical Trial and Random Error. New Scientist, Volume
116, 52- 55.
--- . 1987b. The Scientific Standing of Evolutionary Theories of
Society. The LSE Quarter(v, Volume I, 23- 42.
- - - . 1989. Random Sampling and the Principles of Estimation.
Proceedings olthe Aristotelian Societv, Volume 89, 143-164.
- - - . 1991. Bayesian Methodology: Some Criticisms Answered.
Ratio (New Volume 4, 170- 184.
---- . 1992, Regression Analysis: Classical and Bayesian. British
lournal for the Philosophy oj Science, Volume 43, 311-342.
- - - . 1993. The Value of Randomi zation and Control in Clinical
Trials. Statistics in Medicine, Volume 12, 1421- 431.
Van Fraassen, B.C. 1980. The Scientific Image, Oxford: Clarendon.
- - - . 1983. Ca libration: A Frequency Justification for Personal
Probability, In R.S. Cohen and L. Laudan, eds ., Physics, Philosophy,
and (Dordrecht: Reidel) , 295- 321.
- - - , 1984. Belief and the Will. Journal of Philosophy, Volume
LXXXI, 235- 256.
1989. Laws and Svmmelry. Oxford: Clarendon.
Velikovsky, I. 1950. Worlds in Collision. London: Gollancz. Page refer-
ences are to the 1972 edition, published by Sphere.
Velleman, PF. 1986. Comment on Chatterjee, S., and Hadi , A.S. 1986.
Statistical Science, Volume 1, 412- 15.
Velleman, P.F. , and R.E. Welsch. 1981. Effici ent Computing of
Regression Diagnostics. American Statislician, Volume 35,
234- 242.
Venn, J. 1866. The Logic ojChance. London: Macmillan.
Vranas, PB.M. 2004. Hempel's Raven Paradox: A Lacuna in the
Standard Bayesian Solution. British Journal Jhr the Philosophyof'
Science, Volume 55, 545-560.
Wall , P. 1999. Pain: The Science oj Suffering. London: Weidenfeld and
Nicolson.
Watkins, J. WN. 1985. Science and Scepticism. London: Hutchinson and
Princeton: Princeton University Press.
--- . 1987. A New View of Scientific Rationality. In Rational
Change in Science, edited by J. Pitt and M. Pera. Dordrecht: Reidel.
Weinberg, S. and K. Goldberg. 1990. Statistics Jhr the Behavioral
Sciences. Cambridge: Cambridge University Press.
Weisberg, S. 1980. Applied Linear Regression. New York: Wiley.
318 BIBLIOGRAPHY
Welsch, R.E. 1986. Comment on Chatterjee, S., and Hadi, A.S. 1986.
Statistical Science. Volume 1,403-05.
Whitehead, 1. 1993. The Case for Frequentism in Clinical Trials.
Statistics in Medicine, Volume 12, 1405-413.
Williams, PM. 1980. Bayesian Conditionalisation and the Principle of
Minimum Information. British Journal fc)r the Philosophy 0/
Science. Volume 31 , 131-144.
Williamson, 1. 1999. Countable Additivity and Subjective Probability.
British Journal fhr the Philosophy 0/ Science, Volume 50, 401-416.
Williamson, J. 2005. Bayesian Nets and Causality: Philosophical and
Computational Foundations. Oxford: Oxford University Press.
Williamson, J. and D. COffield. 2001. Introduction: Bayesianism into the
Twenty-First Century. In Corfield, D. and Williamson, 1., cds.,
Foundations of Bayesian ism (Dordrecht: Kluwer).
Wonnacott, T.H. , and R.J. Wonnacott. 1980. Regression: A Second
Course in Statistics. New York: Wiley.
Wood, M. 2003. Making Sense 0/ Statistics. New York: Palgrave
Macmillan.
Yates, F. 1981. Sampling Methodsfor Censuses and Surveys. Fourth edi-
tion. London: Gliffin.
Index
Abraham, Max, 7
Adams, John Couch, 121, 124
Adams Principle, 59
additivity condition, 18
ad hoc, 122
adhocness criteria, as unsound,
123-25
ad hoc theory, 11 9, 120, 121, 122,
125
agricultural field trials, 183-84
Airy, Sir George, 124
Akaike, H., 292 -94
Akaikc Informati on Criterion
(AIC), 293, 295
Armitage, P., 198, 200-0 I
Arnold, G .c., 210, 211, 222
Atkinson, A.C., 233
atomic weight measurements,
108-110, 11 2- 13
axiom of continuity, 27
axiom of countable additivity, 27
critique t ~ 27- 29
Babbage, Charles, 98
Bacon, Francis, 121, 122, 126
Barnett, V, 166
Bartha, P., 29
base- rate fallacy, 24
Bayes, Thomas, 20, 76, 266-68,
272, 278
Memoir, xi, 269
Bayes factor, 97
Bayesian conditionalisation,
80-82, 85
Bayesian convergence-of-opinion
theori es, 28
Bayesian induction, 237
limit theorems of, 238
and posterior probability, 238
Bayesian Information Criteria
(BIC),294-95
BayesianismlBayesian theory, 30 I
on clinical trials, 255- 260
confirmation in, 97, 99
credible intervals in, 244
and deductive inference, 79-80
as epistemi c, 50
estimating binomial proportions
in, 242-43
evidential relevance in,
247- 25 1
and family of curves, 290
and frequentism, 263-64
and influence points, 235
and inductive inference, 79
and least squares method, 217
and obj ectivity, 237-38, 273
old-evidence objection, 298--99
and posterior probabilities, 54,
241 ,278
and Princi ple of Indifference,
273
prior di stri bution in, 246-47
and pri or probabilities, 129- 130
and regression analysis, 23 1-32
on relevant information, 251
revival of, 8
and sampling method, 253- 54
on scientific investigation, 127
320
and stopping rules, 160- 61 ,
250-51
subjectivity in, 237, 241 , 262,
265
sufficiency in, 164
and testing causal hypotheses,
254-55
and updating rules, 80- 81
versus classical approach
to inductive reasoning, xi-xii
to stati stical inference, 295
Bayes ian probability, 45
and hypotheses, 75- 76
and problem of logical
omniscience, 75
Bayes's theorem, 8, 99, 114, 236,
237,262,265,267,299-300
and Bernoulli parameters, 243
on confirmation of theory, by
consequences, 93- 94
for densiti es, 38
first form, 20-21
on posterior and prior
probabiliti es, 92, 11 3, 108-09
and randomization, 202
second form, 21
third form, 21
Bels ley, D.A. , 233
Bernoulli, James, 8, 40, 42, 266,
268
Ars Conjecrundi, 39
Bernoulli parameters, 242, 243
Bernoulli process, 242, 268
Bernoulli sequences, 42
Bernoulli 's Theorem, 266- 67, 294
inversion of, 266 -67
Bernoulli trial s, 4748
Berry, D.A., 26 1
Bertrand's Paradox, 283
beta distributi ons, 242
bett ing quoti ents, 53
binomial distribution, 39-40
bi valent statistical tests, 6
bivariate normal distribution, 38
Bl ackwell , D. , 245
Bland, M. , 150
Blasco, A., 263- 64
Boolean algebra, 14
INDEX
and propositional language, 15
Bourke, G.J., 190, 191
Bradley, F.H , 59
Bovens, L., II I
Brandt, R., 236
Brook, RJ., 210, 211, 222
Bucherer, Alfred, 7
Byar, D.P., 188, 195,260- 61
Caratheodory Extension Theorem,
74
Carnap, Rudolf, 8, 74, 164
categorical-assertion
interpretati on, 171
chance-based odds, 53-54
Chatterjee, S., 227-28
Chebychev's Inequality, 294
chi-square statistic, 137
chi-square test, 137140
problem o ~ 139-140
Church, Alonzo, 50, 62
classical estimates, objecti ons to,
182
Classical Statistical Inference, 131
Classical Theory of Probability, 35
classic law of large numbers, 56
clini cal trials
Bayesian analysis, 255- 59
Bayes ian versus classical
prescriptions, 259-260, 262
central probl em, 183-85
control in, 185- 86
hi storically controlled, 260-61
randomization in, 186-87
sequential, 198- 20 I
without randomi zation. 260
Cochran,W.G .. 140
Cohen, A.M .. 62
composite hypotheses, testing,
161 62
Comrie, L.J., 98
condi tional distributions, 37- 38
conditionalisation, 84-85, 287
INDEX
conditional probabi I ities, 16
conditional probability axiom, 37
Condorcet, Marqui s dc, 55
confidence coeffici ent, 170, 173
confidence interval s, 169- 171,
218-19,244- 45
competing, 171 -72
and stopping rul e, 176
subj ective--confidence
interpretation, 173- 75
consequences, as confirmi ng
theory, 9396
consistency
deductive, 63-66, 73- 74
mathematical concept o ~ 63
of probability axioms, 63
consistent estimators, 166- 68
continuous distributi on, 3 1
Cook, R.D., 221 , 233
Cook's Di stance, 234
Coumot, A.A., 49, 132
Cournot's Pri nciple, 49
covariant rule
for generating pri or
distributions, 273 74
problems with, 274
Cox, R.T., 76, 85- 87, 98, 222,
223,225
Cox-Good argument, 85
CramEr, H., 153
credibl e intervals, 244
and confidence intervals,
comparing, 244-45
critical region, choice of: 14849
Daly, L.E. , 190
Daniel, c., 214
data analysis, 225
and scatter plots, 225--26
data patterns, influent ia l poi nts in,
232
data too good to be true, I 16- 18
Dawid, A.P. , 66
deductive consi stency, 63- 66, 73
local character of, 73-74
deductive inference, 79
analogy to probabilistic
inference, 79- 80
32 1
deductive logic, constraints in, 66
de Finetti, Bruno, 8,28-29, 52, 62,
63,67, 71 , 73, 74,265
on exchangeability, 88-89
de Moivre, Abraham, 40, 41
Di anetics, 120
di stribution fu nct ions, 30 3 I , 34
and density functions, 32
distributions
binomial, 39- 40
bivariate normal, 38
conditional, 37- 38
continuous, 3 I
normal , 32, 33
uniform, 3 1
Dobzhansky, T., 11 8
Dorling, Jon, 8, 107, 110, 114, 11 7
doubl e-blind trials, 255
Downham, V, 18 1
Dubins, L. , 245
Duhem, P., 105
Duhem probl em, 103, 107,119
Bayesian resolution of, 110, 114
Dutch Book Argument, 52, 83
Dutch Book Theorem, 62, 71
dynamic modus ponens, 83, 84
Earman, .I. , 57
Edwards, w., 245
efficient estimators, 168-69
Ehrenberg, A.S.C., 211
Einstein, Albert , 7 8, 103, 262,
298
probabili sm of, 7- 8
eliminative inducti on, 184
epistemic probability, 25, 51 , 6 L
88
formalism oC as model, 61
utility-bascd account , 57
crit ique of, 57- 58
and valuing of consequences,
57-59
322
Equi valcncc Condition, 100, 102
estimatcs
classical, objection to, 182
and prior knowledgc, 17677
estimating binomial proportion,
242
estimating mean, of a normal pop-
ulation, 239- 241
Estimation Thcory, 163
estimators
consistent, 166-68
efficient, 168 -69
sufficient, 164
unbiased, 165-66
exchageability, 90
exchangcable random quantities,
88
expected values, 32-- 33
experiments, and repeatability, 160
fair betting quotients, 67- 68, 73
and probability axioms, 62
fair odds, 54
fal sifiability, 103
problems for, 104 -05
Fal sificationism, 2
Feyerabcnd, Paul, 2
Fisher, R.A., xi,S 6,9,49, 118,
133, 140, 148, 290
on clinical trials, 185- 86
on estimators, 166-67
on randomi zation, 18688,
191 -- 93, 202
on refutati on of statistical
theories, 150
on significance tests, 188
on suffici ent statistics, 142
Fi sherian significance tcsts, 133,
141,143
Fi sher information, 273
formal languages, and
mathematical structurcs, 75
Forster, M., 289-94
Frecman, P.R., 15556
Frequcntism, 131
Freud, Sigmund, 103
Fundamental Lemma, 148
Gaitinan, H. , 74
INDEX
Galileo, 55, 128, 129, 288
Gauss, le.E, 213
Gauss- Markov theorem, 209,
213-15
generalization, from experiments,
96
General Thcory of Relativity
(GTR),298
Gierc, R.N., 195
Gigercnzer, G., 23
Gillies, D.A. , 160
Glymour, e., 298
G6del, Kurt, 62, 73
Goldberg, K., 153
Good, 1.1., 85, 140
goodness-of-fit test, 137
Gore, S.M., 195
Gossett, W.S., 133
Graybill , EA. , 212, 214, 215,
220
Hacking, L 61
Hadi , A.S., 227
Hartmann, S., III
Harvard Mcdical School Test, 22
lessons of, 25
Hays, w.L., 139, 172, 215, 222
Hempel, e.G., 100, 126
Hcrschel, Sir John, 124
HcrseheL William, 121
homascedasticity, 206
Horwich, P, 103
Howson, e. , 128
Hubbard L. Ron, 120, 122
Humc, David, 1- 2,79,80,269
hypotheses
auxiliary, 113,1 16, 119
composite, testing of, 161 - 62
observation in confirmation o ~
91 - 92
INDEX
posteri or probability of, 92 93,
97
vari ed evidence for, 126
improper distributions, 274- 75
independent evidcnce, 125
indicator functi on, 67
induction, problem of, 1-2, 269
inducti ve inference, thcory t ~ 265
inductive probability
objccti vist interpretation, 8
subjccti vist interpretation, 8
influencc funct ions, 234
influcnce measuring, 234-35
influcnce mcthodology, 234
influence/ influential points
and Bayesiani sm, 235
in data patterns, 232
and insecure conclusions, 233
informati onless priors,
impossibility of, 276-77
inscribed triangle probl cm, 282
intcrval estimati on, 169
Jaynes, E.T. , 8, 76, 277- 286, 297
Jeffrey, R.C. , 85
decision theory, 59
Jeffrcy conditi onali sati on, 82, 85
Jeffrey 's rule, 83, 85, 274- 76
Jcffreys, Harold, 8, 76, 128, 272,
273, 277, 288, 289-290, 295
Jenni son, c., 198
Jevons, WS. , 7, 98
Johnson, R. W, 287
judgment sampl ing, 178
advantagcs of, 180 8 J
objections to, 178- 79
Kadanc, 1. , 251
Kant, Immanuel , I
Kaufmann, Walter, 7
Kcll y, 1. , 29
Kcndall, M.G., 139, 166, 168--69,
187, 193-94, 211 , 225
Kcynes, J.M., 266, 269
Kinetic Thcory, 132
Kollektiv, 50, 77, 90
and behavior of limits, 50
Kolmogorov, A. N., 27, 49, 296
Korb, K.B. , 103
Krauss, P., 74
Kuhn, T.S. , 105, 106, 107
Kullback- Lei blcr measure of
discrepancy, 293
Kyburg, H.E. , 144, 147
323
Lakatos, I., 9, 62, 105-08, 114-1 5,
11 9, 122
Laplace, P.S. de, 20, 54, 76, 268,
269
least squares, in regression, 209
least squares line, 208
least squares method, 207-208
and Bayesianism, 217
Gauss- Markov justification,
213- 16
intuitive arguments for, 209-21 2
maximum likclihood
justification, 215- 17
and vertical dcviati ons, 211-12
Icast squares principl e, 236
Lc ibniz, G.W, 115, 262
NOl/veal/x E.I'sais, 5 I
Le Vcrri er, Urbain, 12 I
Lewis, David, 59, 83
The Pri ncipa l Princi ple, 77
Lewis-Beck, M.S., 222, 225
Likelihood Principle, 78, 156
li ke lihood rati o, 155
limit, in probability, 46 47
Lindenbaum algebra, 15
Lindgren, B.W, 151
Lindlcy, D. V., 76, 128, 154, 196,
239
Lindley 's Paradox, 154-56
Lindman, H., 245-46
linear regrcssi on, and stati st ics,
221
logical falsehood, 13
logical truth, 13
324
logic of uncertain inference, 51
L6wenheim, Leopold, 62
MAD method of estimation. 210
Maher. P. , 57, 127
Mallet,lW. , 109, 112, 114
Markov, Andrey, 213
mathematical statistics, 30, 32
maximum- entropy method,
278- 79,285- 86
Maximum Likelihood Principle,
209
Mayo, D.G., 25 1
McGil vray, J. , 190
mean value, 33
measure- theoretic framework, of
probability, 27
Medawar, Peter, 120, 122
Meier, P. . 20 I
Mendel,G.J., 1, 5, 117, 123. 140,
153
method of least squares, 207-01;
method of transformation groups,
280. 284
Miller, R., 127
minimal- sufficient stati sti c, 142
Minimum Information Principle.
286
Mood, A.M, 2 12, 2 14, 215, 220
Musgrave, A .. 96
Neptune. di scovery o ~ 124
Newton, Sir Isaac, 262
Principia, 268
Newton 's laws, 4, 6, 121
Neyman, Jerzy. 6, 9, 49, 141 , 143,
152,155, 16 1, 167, 169,2 13
categorical assertion
interpretation, 171
on confidence intervals, 172-73
Neyman- Pearson significance
tests, 26, 143--48, 156
and deci sions to act, 151-52
null hypotheses in, 144
INDEX
and sufficient statistics, 149
Nieod, Jean, 100
Nicod's Condition, 100
fai lure of, 102
No Miracles Argument, 26
normal di stributions, 32. 33
normal di stribution function, 34
null hypothesis, 5- 6, 133, 143
choice of, 156
grounds for rejecti ng, 140,
149- 150, 155
and li kel ihood ratio, 155
testing of, 133- 35,137-38,141,
148
Object ive Bayesians, 277
objective priors, 296-97
objective probability, 25, 45, 88
and frequency, 45--46. 77
objectivist ideal, of theory, 9
critique o ~ 9
observati ons, as fallible, 104 05
Occam's razor. 288
odds
chance based, 53-54
fair, 53- 54
probability based, 53
old-evidence problem, 298-99
outcome space, 133
out li ers, 226 28
and significance tests, 228- 23 1
P(a), meaning of, 15
Paradox of Confirmation. 99
Pari s, J., 64, 65, 70-7 1,286, 287
Pearson, Egon, 6, 9, 49, 141 , 155,
16 1
Pearson, KarL 133. 138
Peto. R., 18 1
phil osophy of science, 91
placebo effect, 185
Planck, Max, 7
Pitowsky, 1. , 45
PoincarE, Henri , 6- 7, 76
INDEX
point estimation, 163
Pol anyi, Michael, 105
Popper, Karl R., xi, 9, 46, 54. 96,
106, 119, 122, 127, 129, 132,
275,289
on confirmation, 99
on fal sification/falsifiability,
2-3, 5,103- 04.105, 132
on problem of induction, 2-3
on scientific method, 3--4
posterior probabilities, 242
in Bayesianism, 54, 241
precision of a di stribution, 240
prediction
by confidence intervals, 2 18- 19
and prior knowledge, 222-25
and regression, 217 18, 220
predicti on interval, 217
Price, Ri chard, 269
Price. Thomas. xi , 227
The Principal Principle, 77
Principle of Direct Probability, 77,
174, 175
Principle of Indifference, 266-69,
273, 275 76, 279- 2S0. 282,
284 86
paradoxical results from,
269- 272
Principle of Insuffic ient Reason,
26S
Principl e of Random Sampling,
178
Principle of Stable Estimation,
245- 26
Principle of the Uniformity of
Nature, 2
prior probabilities, 129- 130
probabili stic independence, 35- 36
probabilistic induction, 6
probability
and additivity principle, 69
and quotient definitions,
connections between, 68
Bayesian. 45
classical definition, 35
conditional , 46
325
and domains of propositions, 15
epistemi c, 25, 51,61 , 88
limiting relative frequency in.
46- 47
logi cal interpretation of, 74-75
and measure- theoretic
framework, 27
objective, 25,45, 88
and propositional logic, 69
soundness considerations, 27
probability axioms, 16, 45
and coherence, 72
as consistency constrai nts, 63
and deductive consistency, 63
epistemic interpretation, 45
and personal probability, 76- 77
probability calculus, 13, 15, 70
and algebra of propositions, 75
and consistency constraints, 7 1
domain of discourse, 13
fundamental postulates, 16
interpretations of, 88
theorems of, 16- 2 2
probabilit y densiti es, 31 - 32
probability function, 70
probability logic
and coll ective conditions, 66
constraints in, 66
and deductive logic, 72
and fair bets, 66
and sanctions for viol ations of
rules, 72- 73
probability- system, 14
Problem of Induction, 1- 2
prognosti c factors, 184
programmes
degenerating, 107
progressive, 106
Prout, William, 108- II S
quota sampling, 178, lSI
Ramsey, Frank, 8, 51, 63, 80, 82
'Truth and Probability', 57
326
randomization, 190, 193- 94
arguments for, 187- 88
and Bayes's theorem, 202
critique of, 187
eliminative-induction defense
of, 194-95
ethical concerns, 203
practical concerns, 202
and significance tests, 188- 190
unknown factors in, 195- 96
randomized tests, 147 48
randomized trial , 186
controlled, concerns about,
197-98
random quantity, ill betting, 67
random sampling, 177--- 79
random sequence, 50
random vairables, 29- 30, 33
Rao, CD., 166
Ravens Paradox, 99
reference population, 190
regression, 205
examining form of, 220--21
least squares in, 209
linear, 206-297, 221
and prediction, 220
simple linear, 206- 07
regression analysis, 39
and Bayesianism, 231 - 32
regression curve, and prediction,
21718
regression models
and prior knowledge, 222- 25
and scatter plots, 225- 26
regression to the mean, 39
relevant information
in Bayesianism, 251
in classical statistics, 251
Renyj, Alfred, 275
repeatability, in experiments, 160
representative sample, 178
Rosenkrantz, R.D., 102
Rule of Succession, 268- 69
sampling
judgment, 178, 180- 8 1
quota, 178, 181
random, 177--78
representative, 178
stratified random, 180
sampling method, and
Bayesiani sm, 153- 54
Savage, L.J., 55, 57 -59, 76,
245-46,251
critique of, 60
scatter plots, 225- 26
Schervish, M_, 58- 59
Schwarz, G., 294
INDEX
scientific evidence, describing,
247-251
scientific inference, patterns of,
93-94
scientific method, 3 -4
scientific realism, 26
scientific reasoning, 4
scientific theory
and empirical evidence, gap
between, 1,4 5
probabilistic, 5
Scott, D., 74
Seber, G.A.F, 222, 223
Second Incompleteness Theorem,
73
Shakespeare, William
Macheth ,98
Shannon, Claude, 128, 277, 279
sharp probability va lues, 62
Shimony, Abner, 76
Shore, J.E. , 287
significance levels, and inductive
support, 153 54
significance tests, 6, 25-26
and decisions to act, 151--52
Fisherian, 133, 141. 143
int1uenees on, 156
Neyman- Pearson, 26, 143-49,
151-52, 156
and outliers, 228- 23 1
and randomi zation, 198-99
size of, 147
and stopping rule, J 57
simplicity, of predictive theory,
288,291 , 296
INDEX
Simplicity Postulate, 289, 291
Skolem, M.B .. 62
Skyrms. B. . 66
small-world versus grand world
acts, 58
Snir, M., 74
Sober, E., 289- 294
o l o m o n o t l ~ 296
Sorites, 62
Spielman, S .. 90
Sprcnt. P., 222
standard deviation, 33
Stas,J.S. , 109, 114
statistical hypothcses, and
fa lsifi abili ty, 132
statisti cal inference, 9 1
Stone's Theorem. 14
stopping rule, 157, 160
and Bayesianism, 250
irrelevance of. 250
and subj ective intenti on. 159
St. Petersburg problem, 54-55
strat ified random sampling, 180
Strong Law of Large Numbers, 43.
47
Stuart , A. , 135, 14 1, 166, 168- 69,
187. 193, 21 I, 225, 253
sutTicient est imators, 164
suffici ent stati stics, 141-42,
25 1-52
Suzuki , 299
Tanur. J.M .. 195
Tarski, Alfred. 62
Teller, Paul , 83
test-statistic, 133
choosi ng, 136
theorem of total probability. 18
theories
auxi Iiary. 105- 06, I 13
obj ecti vist ideal, 9
probability of, effect of
observation on, 114
under-determination, 128
Thompson, Thomas, 108
Thomson, J.J. , 109
327
Total Ev idence Requirement, 164
Turnbull, B.W., 198
Utfink, Jos, 287
unbiased estimators, 165- 66
uniform di stribution, 31
Uniformly Most Powerful (UMP)
tests, 16 1-62
Uniformly Most Powerful and
Unbiased (UMPU) tests, 161 - 62
updating rul es, 80-8 1, 83
utility-revolution, 56, 57
vari able quantiti es, relati onships
between. 205-06
Velikovsky, I. , 116, 122
Worlds in Collision. 11 9
Velleman, P.E, 234- 35
Vencovsk, A. , 286, 287
von M ises, Richard, 46, 50, 78,
90
Watkins, J.W. N., 104
Weak Law of Large Numbers, 39,
41-43
Weinberg, S. , 153
Weisberg, S., 212, 214, 224, 225.
226, 229-231, 233
Welsch, R. E. , 234 35
Whitehead .I., 159
Winkler, R.L., 139
Wonnacott, R . ./., 209. 210. 222,
236
Wonnacott, T. H., 209. 210, 222,
236
Wood. M. , 2 14
Yat es, F.. 179
zero- expectation bets, 56
zero- expected-gain condition,
54- 55