Foundations of statistics
The Foundations of statistics concerns the
epistemological debate in statistics over how one
should conduct inductive inference from data. Among
the issues considered in statistical inference are the ques-
tion of Bayesian inference versus frequentist inference,
the distinction between Fisher's signicance testing
and NeymanPearson hypothesis testing, and whether
the likelihood principle should be followed. Some of
these issues have been debated for up to 200 years
without resolution.[1]
Bandyopadhyay & Forster[2] describe four statistical
paradigms: "(1) classical statistics or error statistics, (ii)
Bayesian statistics, (iii) likelihood-based statistics, and
(iv) the Akaikean-Information Criterion-based statistics.
Savages text Foundations of Statistics has been cited over
12000 times on Google Scholar.[3] It tells the following.
It is unanimously agreed that statistics de-
pends somehow on probability. But, as to what
probability is and how it is connected with
statistics, there has seldom been such com-
plete disagreement and breakdown of commu-
nication since the Tower of Babel. Doubtless,
much of the disagreement is merely termino-
logical and would disappear under suciently
sharp analysis.
1 Fishers signicance testing
vs NeymanPearson hypothesis
testing
In the development of classical statistics in the second
quarter of the 20th century two competing models of in-
ductive statistical testing were developed.[4][5] Their rel-
ative merits were hotly debated[6] (for over 25 years) un-
til Fishers death. While a hybrid of the two methods is
widely taught and used, the philosophical questions raised
in the debate have not been resolved.
1.1 Signicance testing
Fisher popularized signicance testing, primarily in two
popular and highly inuential books.[7][8] Fishers writing
style in these books was strong on examples and relatively
weak on explanations. The books lacked proofs or deriva-
tions of signicance test statistics (which placed statis- Hypothesis testing requires multiple hypotheses. A hy-
tical practice in advance of statistical theory). Fishers
more explanatory and philosophical writing was written
much later.[9] There appear to be some dierences be-
tween his earlier practices and his later opinions.
Fisher was motivated to obtain scientic experimental re-
sults without the explicit inuence of prior opinion. The
signicance test is a probabilistic version of Modus tol-
lens, a classic form of deductive inference. The signi-
cance test might be simplistically stated, If the evidence
is suciently discordant with the hypothesis, reject the
hypothesis. In application, a statistic is calculated from
the experimental data, a probability of exceeding that
statistic is determined and the probability is compared to
a threshold. The threshold (the numeric version of su-
ciently discordant) is arbitrary (usually decided by con-
vention). A common application of the method is de-
ciding whether a treatment has a reportable eect based
on a comparative experiment. Statistical signicance is a
measure of probability not practical importance. It can
be regarded as a requirement placed on statistical sig-
nal/noise. The method is based on the assumed existence
of an imaginary innite population corresponding to the
null hypothesis.
The signicance test requires only one hypothesis. The
result of the test is to reject the hypothesis (or not), a sim-
ple dichotomy. The test distinguish between truth of the
hypothesis and insuciency of evidence to disprove the
hypothesis; so it is like a criminal trial in which the de-
fendants guilt is assessed in (so it is like a criminal trial
in which the defendant is assumed innocent until proven
guilty).
1.2 Hypothesis testing
Neyman & Pearson collaborated on a dierent, but re-
lated, problem selecting among competing hypotheses
based on the experimental evidence alone. Of their joint
papers the most cited was from 1933.[10] The famous re-
sult of that paper is the NeymanPearson lemma. The
lemma says that a ratio of probabilities is an excellent
criterion for selecting a hypothesis (with the threshold for
comparison being arbitrary). The paper proved an opti-
mality of Students t-test (one of the signicance tests).
Neyman expressed the opinion that hypothesis testing
was a generalization of and an improvement on signi-
cance testing. The rationale for their methods is found in
their joint papers.[11]
1
2 1 FISHERS SIGNIFICANCE TESTING VS NEYMANPEARSON HYPOTHESIS TESTING

pothesis is always selected, a multiple choice. A lack of Fisher and Neyman were separated by attitudes and per-
evidence is not an immediate consideration. The method haps language. Fisher was a scientist and an intuitive
is based on the assumption of a repeated sampling of the mathematician. Inductive reasoning was natural. Ney-
same population (the classical frequentist assumption). man was a rigorous mathematician. He was convinced
by deductive reasoning rather by a probability calculation
based on an experiment.[4] Thus there was an underlying
1.3 Grounds of disagreement clash between applied and theoretical, between science
and mathematics.
The length of the dispute allowed the debate of a wide
range of issues regarded as foundational to statistics.
In this exchange Fisher also discussed the requirements 1.4 Related history
for inductive inference, with specic criticism of cost
functions penalizing faulty judgments. Neyman coun- Neyman, who had occupied the same building in England
tered that Gauss and Laplace used them. This exchange as Fisher, accepted a position on the west coast of the
of arguments occurred 15 years after textbooks began United States of America in 1938. His move eectively
teaching a hybrid theory of statistical testing. ended his collaboration with Pearson and their develop-
ment of hypothesis testing.[4] Further development was
Fisher and Neyman were in disagreement about the foun- continued by others.
dations of statistics (although united in opposition to the
Bayesian view): Textbooks provided a hybrid version of signicance and
hypothesis testing by 1940.[16] None of the principals
had any known personal involvement in the further de-
The interpretation of probability
velopment of the hybrid taught in introductory statistics
[5]
The disagreement over Fishers inductive rea- today.
soning vs Neymans inductive behavior con- Statistics later developed in dierent directions includ-
tained elements of the Bayesian/Frequentist ing decision theory (and possibly game theory), Bayesian
divide. Fisher was willing to alter his opinion statistics, exploratory data analysis, robust statistics and
(reaching a provisional conclusion) on the ba- nonparametric statistics. NeymanPearson hypothesis
sis of a calculated probability while Neyman testing contributed strongly to decision theory which is
was more willing to change his observable be- very heavily used (in statistical quality control for ex-
havior (making a decision) on the basis of a ample). Hypothesis testing readily generalized to ac-
computed cost. cept prior probabilities which gave it a Bayesian a-
vor. NeymanPearson hypothesis testing has become
The proper formulation of scientic questions with
[6][15] an abstract mathematical subject taught in post-graduate
special concern for modeling
statistics,[17] while most of what is taught to under-
Whether it is reasonable to reject a hypothesis based graduates and used under the banner of hypothesis testing
on a low probability without knowing the probability is from Fisher.
of an alternative

Whether a hypothesis could ever be accepted on the 1.5 Contemporary opinion

basis of data
No major battles between the two classical schools of
In mathematics, deduction proves, counter-
testing have erupted for decades, but sniping continues
examples disprove
(perhaps encouraged by partisans of other controversies).
In the Popperian philosophy of science, ad- After generations of dispute, there is virtually no chance
vancements are made when theories are dis- that either statistical testing theory will replace the other
proven in the foreseeable future.
Subjectivity: While Fisher and Neyman struggled to The hybrid of the two competing schools of testing can be
minimize subjectivity, both acknowledged the im- viewed very dierently as the imperfect union of two
portance of good judgment. Each accused the mathematically complementary ideas [14] or as the fun-
other of subjectivity. damentally awed union of philosophically incompatible
ideas.[18] Fisher enjoyed some philosophical advantage,
Fisher subjectively chose the null hypothesis. while Neyman & Pearson employed the more rigorous
NeymanPearson subjectively chose the crite- mathematics. Hypothesis testing is controversial among
rion for selection (which was not limited to a some users, but the most popular alternative (condence
probability). intervals) is based on the same mathematics.
Both subjectively determined numeric thresh- The history of the development left testing without a sin-
olds. gle citable authoritative source for the hybrid theory that
2.1 Major contributors 3

reects common statistical practice. The merged termi- 2.1 Major contributors
nology is also somewhat inconsistent. There is strong em-
pirical evidence that the graduates (and instructors) of an Main article: History of statistics
introductory statistics class have a weak understanding of
the meaning of hypothesis testing.[19] Two major contributors to frequentist (classical) meth-
ods were Fisher and Neyman.[4] Fishers interpreta-
tion of probability was idiosyncratic (but strongly non-
1.6 Summary
Bayesian). Neymans views were rigorously frequentist.
The interpretation of probability has not been re- Three major contributors to 20th century Bayesian sta-
solved (but ducial probability is an orphan). tistical philosophy, mathematics and methods were de
Finetti,[21] Jereys[22] and Savage.[23] Savage popular-
Neither test method has been rejected. Both are ized de Finettis ideas in the English-speaking world and
heavily used for dierent purposes. made Bayesian mathematics rigorous. In 1965, Dennis
Lindleys 2-volume work Introduction to Probability and
Texts have merged the two test methods under the Statistics from a Bayesian Viewpoint brought Bayesian
term hypothesis testing. methods to a wide audience. Statistics has advanced over
the past three generations; The authoritative views of
Mathematicians claim (with some exceptions)
the early contributors are not all current.
that signicance tests are a special case of hy-
pothesis tests.
Others treat the problems and methods as dis- 2.2 Contrasting approaches
tinct (or incompatible).
2.2.1 Frequentist inference
The dispute has adversely aected statistical educa-
tion. Main article: Frequentist inference

Frequentist inference is partially and tersely described

2 Bayesian inference versus fre- above in (Fishers signicance testing vs Neyman
quentist inference Pearson hypothesis testing). Frequentist inference
combines several dierent views. The result is capa-
ble of supporting scientic conclusions, making opera-
Two dierent interpretations of probability (based on ob-
tional decisions and estimating parameters with or with-
jective evidence and subjective degrees of belief) have
out condence intervals. Frequentist inference is based
long existed. Gauss and Laplace could have debated alter-
solely on the (one set of) evidence.
natives more than 200 years ago. Two competing schools
of statistics have developed as a consequence. Classical
inferential statistics was largely developed in the second 2.2.2 Bayesian inference
quarter of the 20th Century,[5] much of it in reaction to
the (Bayesian) probability of the time which utilized the Main article: Bayesian inference
controversial principle of indierence to establish prior
probabilities. The rehabilitation of Bayesian inference
was a reaction to the limitations of frequentist probabil- A classical frequency distribution describes the probabil-
ity. More reactions followed. While the philosophical ity of the data. The use of Bayes theorem allows a more
intepretations are old, the statistical terminology is not. abstract concept the probability of a hypothesis (cor-
The current statistical terms Bayesian and frequentist responding to a theory) given the data. The concept was
stabilized in the second half of the 20th century.[20] The once known as inverse probability. Bayesian inference
(philosophical, mathematical, scientic, statistical) ter- updates the probability estimate for a hypothesis as addi-
minology is confusing: the classical interpretation of tional evidence is acquired. Bayesian inference is explic-
probability is Bayesian while classical statistics is fre- itly based on the evidence and prior opinion, which allows
quentist. Frequentist also has varying interpretations it to be based on multiple sets of evidence.
dierent in philosophy than in physics.
The nuances of philosophical probability interpretations 2.2.3 Comparisons of characteristics
are discussed elsewhere. In statistics the alternative in-
terpretations enable the analysis of dierent data using Frequentists and Bayesians use dierent models of prob-
dierent methods based on dierent models to achieve ability. Frequentists often consider parameters to be xed
slightly dierent goals. Any statistical comparison of the but unknown while Bayesians assign probability distri-
competing schools considers pragmatic criteria beyond butions to similar parameters. Consequently, Bayesians
the philosophical. speak of probabilities that don't exist for frequentists; A
4 2 BAYESIAN INFERENCE VERSUS FREQUENTIST INFERENCE
Bayesian speaks of the probability of a theory while a true
frequentist can speak only of the consistency of the evi-
dence with the theory. Example: A frequentist does not
say that there is a 95% probability that the true value of a
parameter lies within a condence interval, saying instead
that 95% of condence intervals contain the true value.
2.3 Mathematical results
Neither school is immune from mathematical criticism 2.6 Illustrative quotations
and neither accepts it without a struggle. Steins paradox
(for example) illustrated that nding a at or uninfor- carefully used, the frequentist approach yields
mative prior probability distribution in high dimensions
[1]
is subtle. Bayesians regard that as peripheral to the core
of their philosophy while nding frequentism to be rid-
dled with inconsistencies, paradoxes and bad mathemati-
cal behavior. Frequentists can explain most. Some of the
bad examples are extreme situations - such as estimat-
ing the weight of a herd of elephants from measuring the
weight of one (Basus elephants), which allows no statis- real line. No wonder that mathematicians nd it of-
tical estimate of the variability of weights. The likelihood
principle has been a battleground.
2.4 Statistical results
Both schools have achieved impressive results in solving
real-world problems. Classical statistics eectively has
the longer record because numerous results were obtained
with mechanical calculators and printed tables of special
statistical functions. Bayesian methods have been highly
successful in the analysis of information that is naturally
sequentially sampled (radar and sonar). Many Bayesian
methods and some recent frequentist methods (such as
the bootstrap) require the computational power widely
available only in the last several decades.
There is hint that Bayesian philosophy is book smart
compared to Frequentist street smarts. Bayesian phi-
losophy has sometimes been silent on shuing the cards.
The design of experiments teaches the importance of
the source of statistical data. Fisher was a major contrib-
utor to the theory.
There is active discussion about combining Bayesian and
frequentist methods,[27][25] but reservations are expressed
about the meaning of the results and reducing the diver-
sity of approaches.
2.5 Philosophical results
Bayesians are united in opposition to the limitations of
frequentism, but are philosophically divided into numer-
ous camps (empirical, hierarchical, objective, personal,
subjective), each with a dierent emphasis. One (fre-
quentist) philosopher of statistics has noted a retreat from
the statistical eld to philosophical probability interpreta-
tions over the last two generations.[28] There is a percep-
tion that successes in Bayesian applications do not jus-
tify the supporting philosophy.[29] Bayesian methods of-
ten create useful models that are not used for traditional
inference and which owe little to philosophy.[30] None of
the philosophical interpretations of probability (frequen-
tist or Bayesian) appears robust. The frequentist view is
too rigid and limiting while the Bayesian view can be si-
multaneously objective and subjective, etc.
broadly applicable if sometimes clumsy answers[31]
To insist on unbiased [frequentist] techniques may
lead to negative (but unbiased) estimates of a vari-
ance; the use of p-values in multiple tests may
lead to blatant contradictions; conventional 0.95-
condence regions may actually consist of the whole
ten dicult to believe that conventional statistical
methods are a branch of mathematics.[32]
Bayesianism is a neat and fully principled philoso-
phy, while frequentism is a grab-bag of opportunis-
tic, individually optimal, methods.[24]
in multiparameter problems at priors can yield
very bad answers[31]
"[Bayes rule] says there is a simple, elegant way to
combine current information with prior experience
in order to state how much is known. It implies that
suciently good data will bring previously disparate
observers to agreement. It makes full use of avail-
able information, and it produces decisions having
the least possible error rate.[33]
Bayesian statistics is about making probability
statements, frequentist statistics is about evaluating
probability statements.[34]
"[S]tatisticians are often put in a setting reminiscent
of Arrows paradox, where we are asked to provide
estimates that are informative and unbiased and con-
dence statements that are correct conditional on the
data and also on the underlying true parameter.[34]
(These are conicting requirements.)
formal inferential aspects are often a relatively
small part of statistical analysis[31]
The two philosophies, Bayesian and frequentist, are
more orthogonal than antithetical.[24]
An hypothesis that may be true is rejected be-
cause it has failed to predict observable results
that have not occurred. This seems a remarkable
procedure.[22]
 5

2.7 Summary Some (frequentist) signicance tests are not consistent

with the likelihood principle. Bayesians accept the prin-
Bayesian theory has a mathematical advantage ciple which is consistent with their philosophy (perhaps
encouraged by the discomture of frequentists). "[T]he
Frequentist probability has existence and con- likelihood approach is compatible with Bayesian sta-
sistency problems tistical inference in the sense that the posterior Bayes
But, nding good priors to apply Bayesian the- distribution for a parameter is, by Bayess Theorem,
ory remains (very?) dicult found by multiplying the prior distribution by the like-
lihood function.[35] Frequentists interpret the principle
Both theories have impressive records of successful adversely to Bayesians as implying no concern about
application the reliability of evidence. The likelihood principle of
Bayesian statistics implies that information about the ex-
Neither supporting philosophical interpretation of perimental design from which evidence is collected does
probability is robust not enter into the statistical analysis of the data.[38] Many
There is increasing skepticism of the connection be- Bayesians (Savage for example)[39] recognize that impli-
tween application and philosophy cation as a vulnerability.
The likelihood principle has become an embarrass-
Some statisticians are recommending active collab- ment to both major philosophical schools of statis-
oration (beyond a cease re) tics; It has weakened both rather than favoring ei-
ther. Its strongest supporters claim that it oers a
better foundation for statistics than either of the two
3 The likelihood principle schools. "[L]ikelihood looks very good indeed when
it is compared with these [Bayesian and frequentist]
Main article: Likelihood principle alternatives.[40] These supporters include statisticians
and philosophers of science.[41] The concept needs fur-
ther development before it can be regarded as a serious
Likelihood is a synonym for probability in common us- challenge to either existing school, but it seems to oer
age. In statistics it is reserved for probabilities that fail a promising compromise position. While Bayesians ac-
to meet the frequentist denition. A probability refers knowledge the importance of likelihood for calculation,
to variable data for a xed hypothesis while a likelihood they believe that the posterior probability distribution is
refers to variable hypotheses for a xed set of data. Re- the proper basis for inference.[42]
peated measurements of a xed length with a ruler gen-
erate a set of observations. Each xed set of observa-
tional conditions is associated with a probability distri-
bution and each set of observations can be interpreted as 4 Modeling
a sample from that distribution the frequentist view of
probability. Alternatively a set of observations may result
Main articles: Statistical model and Structural equation
from sampling any of a number of distributions (each re-
sulting from a set of observational conditions). The prob- modeling
abilistic relationship between a xed sample and a vari-
able distribution (resulting from a variable hypothesis) is Inferential statistics is based on models. Much of clas-
termed likelihood a Bayesian view of probability. A sical hypothesis testing, for example, was based on the
set of length measurements may imply readings taken by assumed normality of the data. Robust and nonparamet-
careful, sober, rested, motivated observers in good light- ric statistics were developed to reduce the dependence on
ing. that assumption. Bayesian statistics interprets new ob-
A likelihood is a probability (or not) by another name servations from the perspective of prior knowledge as-
which exists because of the limited frequentist denition suming a modeled continuity between past and present.
of probability. Likelihood is a concept introduced and The design of experiments assumes some knowledge of
advanced by Fisher for more than 40 years (although prior those factors to be controlled, varied, randomized and
references to the concept exist and Fishers support was observed. Statisticians are well aware of the diculties
half-hearted).[35] The concept was accepted and substan- in proving causation (more of a modeling limitation than
tially changed by Jereys.[36] In 1962 Birnbaum proved a mathematical one), saying "correlation does not imply
the likelihood principle from premises acceptable to most causation".
statisticians.[37] The proof has been disputed by statis- More complex statistics utilizes more complex models,
ticians and philosophers. The principle says that all of often with the intent of nding a latent structure under-
the information in a sample is contained in the likelihood lying a set of variables. As models and data sets have
function, which is accepted as a valid probability distri- grown in complexity,[43][44] foundational questions have
bution by Bayesians (but not by frequentists). been raised about the justication of the models and the
6 7 NOTES

validity of inferences drawn from them. The range of History of statistics

conicting opinion expressed about modeling is large.
Philosophy of probability
Models can be based on scientic theory or on ad- Philosophy of mathematics
hoc data analysis. The approaches use dierent
methods. There are advocates of each.[45] Philosophy of science

Model complexity is a compromise. The Akaikean Evidence

information criterion and Bayesian information cri- Probability interpretations
terion are two less subjective approaches to achiev-
ing that compromise.[46] Founders of statistics

Fundamental reservations have been expressed

about even simple regression models used in the so-
cial sciences. A long list of assumptions inherent
7 Notes
to the validity of a model is typically neither men-
[1] Efron 1978.
tioned nor checked. A favorable comparison be-
tween observations and model is often considered [2] Bandyopadhyay & Forster 2011.
sucient.[47]
[3] Citations of Savage (1972)
Bayesian statistics focuses so tightly on the posterior
[4] Lehmann 2011.
probability that it ignores the fundamental compar-
ison of observations and model.[30] [5] Gigerenzer et al. 1989.

Traditional observation-based models are inade- [6] Lou 1993.

quate to solve many important problems. A much
[7] Fisher 1925.
wider range of models, including algorithmic mod-
els, must be utilized. If the model is a poor emula- [8] Fisher 1935.
tion of nature, the conclusions may be wrong.[48]
[9] Fisher 1956.
Modeling is often poorly done (the wrong methods
[10] Neyman & Pearson 1933.
are used) and poorly reported.[49]
[11] Neyman & Pearson 1967.
In the absence of a strong philosophical consensus review [12] Fisher 1955.
of statistical modeling, many statisticians accept the cau-
tionary words of statistician George Box, "All models are [13] Neyman 1956.
wrong, but some are useful.
[14] Lehmann 1993.

[15] Lenhard 2006.

5 Other reading [16] Halpin & Stam 2006.

For a short introduction to the foundations of statis- [17] Lehmann & Romano 2005.
tics, see ch. 8 (Probability and statistical inference) [18] Hubbard & Bayarri c. 2003.
of Kendalls Advanced Theory of Statistics (6th edition,
1994). [19] Sotos et al. 2007.

In his book Statistics As Principled Argument, Robert P. [20] Fienberg 2006.

Abelson articulates the position that statistics serves as
[21] de Finetti 1964.
a standardized means of settling disputes between scien-
tists who could otherwise each argue the merits of their [22] Jereys 1939.
own positions ad innitum. From this point of view,
statistics is a form of rhetoric; as with any means of set- [23] Savage 1954.
tling disputes, statistical methods can succeed only as [24] Efron 2013.
long as all parties agree on the approach used.
[25] Little 2005.

[26] Yu 2009.
6 See also
[27] Berger 2003.

Philosophy of statistics [28] Mayo 2013.

 7

[29] Senn 2011. Bandyopadhyay, Prasanta; Forster, Malcolm, eds.

(2011). Philosophy of statistics. Handbook of the
[30] Gelman & Shalizi 2012. Philosophy of Science. 7. Oxford: North-Holland.
[31] Cox 2005. ISBN 978-0444518620. The text is a collection of
essays.
[32] Bernardo 2008.
Berger, James O. (2003). Could Fisher, Jereys
[33] Kass c. 2012. and Neyman Have Agreed on Testing?". Statistical
Science. 18 (1): 132. doi:10.1214/ss/1056397485.
[34] Gelman 2008.

[35] Edwards 1999. Bernardo, Jose M. (2008). Comment on Arti-

cle by Gelman. Bayesian Analysis. 3 (3): 453.
[36] Aldrich 2002. doi:10.1214/08-BA318REJ.
[37] Birnbaum 1962. Birnbaum, A. (1962). On the foundations of statis-
tical inference. J. Amer. Statist. Ass. 57: 269326.
[38] Backe 1999.

[39] Savage 1960, p. 585.
[40] Forster & Sober 2001.
[41] Royall 1997.
[42] Lindley 2000.
[43] Some large models attempt to predict the behavior of
voters in the United States of America. The popula-
tion is around 300 million. Each voter may be inu-
enced by many factors. For some of the complica-
tions of voter behavior (most easily understood by the
natives) see: http://www.stat.columbia.edu/~{}gelman/
presentations/redbluetalkubc.pdf
[44] Efron mentions millions of data points and thousands of
parameters from scientic studies.
[45] Tabachnick & Fidell 1996.
[46] Forster & Sober 1994.
[47] Freedman 1995.
[48] Breiman 2001.
Breiman, Leo (2001). Statistical Modeling: The
Two Cultures. Statistical Science. 16 (3): 199231.
doi:10.1214/ss/1009213726.
Chin, Wynne W. (n.d.). Structural Equation Mod-
eling in IS Research - Understanding the LISREL
and PLS perspective. University of Houston lec-
ture notes?
Cox, D. R. (2005). Frequentist and Bayesian
Statistics: a Critique. Statistical Problems in Par-
ticle Physics, Astrophysics and Cosmology. PHYS-
TAT05.
de Finetti, Bruno (1964). Foresight: its Logical
laws, its Subjective Sources. In Kyburg, H. E. Stud-
ies in Subjective Probability. H. E. Smokler. New
York: Wiley. pp. 93158. Translation of the 1937
French original with later notes added.
Edwards, A.W.F. (1999). Likelihood. Prelimi-
nary version of an article for the International Ency-
clopedia of the Social and Behavioral Sciences.

[49] Chin n.d. Efron, Bradley (2013). A 250-Year Argument:

Belief, Behavior, and the Bootstrap. Bulletin
(new series) of the American Mathematical Society.
50 (1): 129146. doi:10.1090/s0273-0979-2012-
8 References 01374-5.

Abelson, Robert P. (1995). Statistics as Principled
Argument. Lawrence Erlbaum Associates. ISBN 0-
8058-0528-1. ... the purpose of statistics is to orga-
nize a useful argument from quantitative evidence,
using a form of principled rhetoric.
Aldrich, John (2002). How likelihood and iden-
tication went Bayesian. International Statisti-
cal Review. 70 (1): 7998. doi:10.1111/j.1751-
5823.2002.tb00350.x.
Backe, Andrew (1999). The likelihood principle
and the reliability of experiments. Philosophy of
Science. 66: S354S361. doi:10.1086/392737.
Efron, Bradley (1978). Controversies in the
foundations of statistics (PDF). The Ameri-
can Mathematical Monthly. 85 (4): 231246.
doi:10.2307/2321163.
Fienberg, Stephen E. (2006). When did Bayesian
inference become Bayesian"?". Bayesian Analysis.
1 (1): 140. doi:10.1214/06-ba101.
Fisher, R. A. (1925). Statistical Methods for Re-
search Workers. Edinburgh: Oliver and Boyd.
Fisher, Sir Ronald A. (1935). Design of Experi-
ments. Edinburgh: Oliver and Boyd.
8 8
Fisher, R (1955). Statistical Methods and Scien-
tic Induction (PDF). Journal of the Royal Statisti-
cal Society, Series B. 17 (1): 6978.
Fisher, Sir Ronald A. (1956). The logic of scientic
inference. Edinburgh: Oliver and Boyd.
Forster, Malcolm; Sober, Elliott (1994). How to
Tell when Simpler, More Unied, or Less Ad Hoc
Theories will Provide More Accurate Predictions.
British Journal for the Philosophy of Science (45):
136.
Forster, Malcolm; Sober, Elliott (2001). Why like-
lihood. Likelihood and evidence: 8999.
Freedman, David (March 1995). Some issues in
the foundation of statistics. Foundations of Science.
1 (1): 1939.
Gelman, Andrew (2008). Rejoinder. Bayesian
Analysis. 3 (3): 467478. doi:10.1214/08-
BA318REJ. A joke escalated into a serious discus-
sion of Bayesian problems by 5 authors (Gelman,
Bernardo, Kadane, Senn, Wasserman) on pages
445-478.
Gelman, Andrew; Shalizi, Cosma Rohilla (2012).
Philosophy and the practice of Bayesian statis-
tics. British Journal of Mathematical and Statis-
tical Psychology. 66: 838. doi:10.1111/j.2044-
8317.2011.02037.x.
Gigerenzer, Gerd; Swijtink, Zeno; Porter,
Theodore; Daston, Lorraine; Beatty, John; Kruger,
Lorenz (1989). Part 3: The Inference Experts.
The Empire of Chance: How Probability Changed
Science and Everyday Life. Cambridge University
Press. pp. 70122. ISBN 978-0-521-39838-1.
Halpin, P F; Stam, HJ (Winter 2006). Inductive
Inference or Inductive Behavior: Fisher and Ney-
man: Pearson Approaches to Statistical Testing in
Psychological Research (19401960)". The Amer-
ican Journal of Psychology. 119 (4): 625653.
doi:10.2307/20445367. JSTOR 20445367. PMID
17286092.
Hubbard, Raymond; Bayarri, M. J. (c. 2003).
P Values are not Error Probabilities (PDF). A
working paper that explains the dierence between
Fishers evidential p-value and the NeymanPearson
Type I error rate .
Jereys, H. (1939). The theory of probability. Ox-
ford University Press.
Kass (c. 2012). Why is it that Bayes rule has not
only captured the attention of so many people but in-
spired a religious devotion and contentiousness, re-
peatedly across many years?" (PDF).
REFERENCES
Lehmann, E. L. (December 1993). The Fisher,
NeymanPearson Theories of Testing Hypotheses:
One Theory or Two?". Journal of the Ameri-
can Statistical Association. 88 (424): 12421249.
doi:10.1080/01621459.1993.10476404.
Lehmann, E. L. (2011). Fisher, Neyman, and the
creation of classical statistics. New York: Springer.
ISBN 978-1441994998.
Lehmann, E.L.; Romano, Joseph P. (2005). Testing
Statistical Hypotheses (3E ed.). New York: Springer.
ISBN 0-387-98864-5.
Lenhard, Johannes (2006). Models and Statisti-
cal Inference: The Controversy between Fisher and
NeymanPearson. Brit. J. Phil. Sci. 57: 6991.
doi:10.1093/bjps/axi152.
Lindley, D.V. (2000). The philosophy of statistics.
Journal of the Royal Statistical Society, Series D. 49.
pp. 293337. doi:10.1111/1467-9884.00238.
Little, Roderick J. (2006). Calibrated Bayes: A
Bayes/Frequentist Roadmap. 60 (3).
Lou, Francisco (2008). Should The Widest Cleft
in Statistics-How and Why Fisher opposed Neyman
and Pearson (PDF). Working paper contains nu-
merous quotations from the original sources of the
dispute.
Mayo, Deborah G. (February 2013). Discussion:
Bayesian Methods: Applied? Yes. Philosophical
Defense? In Flux. The American Statistician. 67
(1): 1115. doi:10.1080/00031305.2012.752410.
Neyman, J; Pearson, E. S. (January 1, 1933). On
the Problem of the most Ecient Tests of Statistical
Hypotheses. Phil. Trans. R. Soc. Lond. A. 231
(694706): 289337. doi:10.1098/rsta.1933.0009.
Neyman, J (1967). Joint statistical papers of
J.Neyman and E.S.Pearson. Cambridge University
Press.
Neyman, Jerzy (1956). Note on an Article by Sir
Ronald Fisher. Journal of the Royal Statistical So-
ciety, Series B. 18 (2): 288294.
Royall, Richard (1997). Statistical evidence : a like-
lihood paradigm. London New York: Chapman &
Hall. ISBN 978-0412044113.
Savage, L.J. (1972). Foundations of Statistics (sec-
ond ed.).
Senn, Stephen (2011). You May Believe You Are
a Bayesian But You Are Probably Wrong. RMM.
2: 4866.
 9

Sotos, Ana Elisa Castro; Vanhoof, Stijn; Noortgate,

Wim Van den; Onghena, Patrick (2007). Students
Misconceptions of Statistical Inference: A Review
of the Empirical Evidence from Research on Statis-
tics Education. Educational Research Review. 2:
98113. doi:10.1016/j.edurev.2007.04.001.
Stuart A., Ord J.K. (1994). Kendalls Advanced
Theory of Statistics, volume I: Distribution Theory
(Edward Arnold).

Tabachnick, Barbara G.; Fidell, Linda S. (1996).

Using Multivariate Statistics (3rd ed.). ISBN 0-673-
99414-7. Principal components is an empirical ap-
proach while factor analysis and structural equation
modeling tend to be theoretical approaches. p 27

Yu, Yue (2009). Bayesian vs. Frequentist (pdf).

Lecture notes? University of Illinois at Chicago

9 Further reading
Barnett, Vic (1999). Comparative Statistical Infer-
ence (3rd ed.). Wiley. ISBN 978-0-471-97643-1.

Cox, David R. (2006). Principles of Statistical In-

ference. Cambridge University Press. ISBN 978-0-
521-68567-2.
Efron, Bradley (1986). Why Isn't Everyone a
Bayesian? (with discussion). The American Statisti-
cian. 40. pp. 111. doi:10.2307/2683105. JSTOR
2683105.
Good, I. J. (1988). The Interface Between Statis-
tics and Philosophy of Science. Statistical Science. 3.
pp. 386397. doi:10.1214/ss/1177012754. JSTOR
2245388.
Kadane J.B., Schervish M.J., Seidenfeld T. (1999),
Rethinking the Foundations of Statistics (Cambridge
University Press). [Bayesian.]
Mayo, Deborah G. (1992). Did Pearson reject the
NeymanPearson philosophy of statistics?. Synthese.
90. pp. 233262. doi:10.1007/BF00485352.

10 External links
Citations of Savage (1972) at Google Scholar. [Over
10000 citations.]

Stanford Encyclopedia of Philosophy entry on prob-

ability interpretations.

Stanford Encyclopedia of Philosophy entry on phi-

losophy of statistics.
10 11 TEXT AND IMAGE SOURCES, CONTRIBUTORS, AND LICENSES

11 Text and image sources, contributors, and licenses

11.1 Text
Foundations of statistics Source: https://en.wikipedia.org/wiki/Foundations_of_statistics?oldid=776786769 Contributors: Michael
Hardy, Dcljr, Jeodesic, Avenue, Rjwilmsi, Gwernol, Grafen, Crasshopper, SmackBot, Chris the speller, Nbarth, Colonies Chris, Sct72,
Shalom Yechiel, Lambiam, Myasuda, PamD, TheSeven, The Tetrast, Geometry guy, Melcombe, Tautologist, TerryM--re, Qwfp, Dthom-
sen8, Yobot, AnomieBOT, Apollo1758, Citation bot, James.h.saunders, Quebec99, Citation bot 1, I dream of horses, Kiefer.Wolfowitz,
Trappist the monk, SporkBot, Danim, Helpful Pixie Bot, Marcocapelle, BattyBot, Illia Connell, Anrnusna, Loraof, Isambard Kingdom and
Anonymous: 13

11.2 Images
File:Edit-clear.svg Source: https://upload.wikimedia.org/wikipedia/en/f/f2/Edit-clear.svg License: Public domain Contributors: The
Tango! Desktop Project. Original artist:
The people from the Tango! project. And according to the meta-data in the le, specically: Andreas Nilsson, and Jakub Steiner (although
minimally).
File:Fisher_iris_versicolor_sepalwidth.svg Source: https://upload.wikimedia.org/wikipedia/commons/4/40/Fisher_iris_versicolor_
sepalwidth.svg License: CC BY-SA 3.0 Contributors: en:Image:Fisher iris versicolor sepalwidth.png Original artist: en:User:Qwfp (origi-
nal); Pbroks13 (talk) (redraw)

11.3 Content license

Creative Commons Attribution-Share Alike 3.0