Bayesian Statistics

Training Objectives

• To learn about pdf, Cdf & R distributions

• To learn the definition of conditional probability
• To learn about prior, posterior & likelihood
• To know the difference between Classical & Bayesian statistics
• To learn how Bayesian Statistics are used in RE tests

February 4-6, 2003 Bayesian Statistics Page 2 of 20

Outline

• pdf, Cdf, R
• Conditional Probability
• Prior, Posterior, Likelihood
• Classical vs. Bayesian Statistics
• Bayesian Testing
• Bayesian Parameter Estimation
• Bayesian Modeling

February 4-6, 2003 Bayesian Statistics Page 3 of 20

 pdf, Cdf, R
• Probability Density Function (pdf or f)
f(x)dx = P(x≤true_x<x+dx)
probability the true value of x is between x and x+dx
Integrate over all x equals 1
• Cumulative Distribution Function (Cdf or F)
F(x) = P(true_x<x)
probability the true value of x is less than x
F(x) = area under f(x), F(x) goes from 0 to 1
• Survival or Reliability Function (R)
R(x) = P(x≤true_x) = 1-F(x), goes from 1 to 0
• For any x
time, defect probability, R(t), quality variable, MTBF (or 1/MTBF), MTTR, etc.

February 4-6, 2003 Bayesian Statistics Page 4 of 20

 f(x): mean=0, std=1

Example 0.45
0.4
0.35
0.3
0.25

f(x)
Probability Density Function 0.2
0.15

f(x) 0.1
0.05

integrate over all x = 1 0

-4 -3 -2 -1 0 1 2 3 4
x

F(x): mean=0, std=1

1
0.9
Cumulative Distribution Function 0.8
0.7
F(x) = area under f(x) 0.6
F(x)

0.5

F(x) goes from 0 to 1 0.4

0.3
0.2
0.1
0
-4 -3 -2 -1 0 1 2 3 4
x

February 4-6, 2003 Bayesian Statistics Page 5 of 20

Cdf(x)

If we know the Cdf(x) for any x, then we have the

information we need to make confidence statements

• The confidence that true_x<x is Cdf(x)

• The confidence that true_x>x is 1-Cdf(x)
• The confidence that x1<true_x<x2 is Cdf(x2)-Cdf(x1)
• We can also solve for x that gives Cdf(x)=value:
e.g. the x that gives Cdf(x)=0.9
e.g. the x that gives Cdf(x)=0.1

February 4-6, 2003 Bayesian Statistics Page 6 of 20

Math to Create a Survival Plot from Raw Data
Step #1: List (e.g.) Uptimes in minutes 1, 8, 2, 16, 1, 7, 11, 3, 6, 13

Step #2: Rank Order 1, 1, 2, 3, 6, 7, 8, 11, 13, 16

Step #3: Plot # of Occurrences per Time Interval

4
probability density
# of 3
function (pdf)
Occur. 2
1

0 5 10 15 20

February 4-6, 2003 Bayesian Statistics Page 7 of 20

Math to Create a Survival Plot from Raw Data
Step #4: Create Table of Interval Occurrence Probabilities
and Cumulative Probabilities

Time # of Interval Total # of Cumulative

Interval Occurrences Probability Time Occurrences Probability
0-5 4 0.4 0-5 4 0.4
5-10 3 0.3 0-10 7 0.7
10-15 2 0.2 0-15 9 0.9
15-20 1 0.1 0-20 10 1.0

Step #5: Create Cumulative Distribution Plot (Cdf) and Survival Plot

Cdf(t) or F(t)

Survival=R(t)=S(t)=1-Cdf(t)=1-F(t)

February 4-6, 2003 Bayesian Statistics Page 8 of 20

Thomas Bayes
• 1702-1761
• Non-conformist Presbyterian
Minister
• Probably tutored by de Moivre
• “Essay towards solving a
problem in the doctrine of
chances” 1764

February 4-6, 2003 Bayesian Statistics Page 9 of 20

 Conditional Probability
0.1 = (1/3) * 0.3 = (1/3) * 0.3
P(A&B) = P(A|B)*P(B) = P(B|A)*P(A)
Probability of A given B

A A&B B
0.3 0.1 0.3

February 4-6, 2003 Bayesian Statistics Page 10 of 20

 Bayes Theorem
P(A&B) = P(A|B)*P(B) = P(B|A)*P(A) Definition

P(A|B) = P(B|A) * P(A) / P(B) Divide by P(B)

or
P(x|data) = P(data|x) * P(x) / P(data) Substitution
or
P(x|data) = Constant * P(data|x) * P(x) Substitution

February 4-6, 2003 Bayesian Statistics Page 11 of 20

 Definition of Terms

P(x|data) = Constant * P(data|x) * P(x)

P(x|data): Posterior distribution of x (pdf of x)

Constant: Normalization Factor
P(data|x): Likelihood function
P(x): Prior distribution of x (pdf of x)

February 4-6, 2003 Bayesian Statistics Page 12 of 20

 Bayes vs. Classical Approach

Bayesian Statistics:
P(x|data) = Constant * P(data|x) * P(x)

Classical Statistics:
P(x|data) = Constant * P(data|x)
where, P(x) = a constant (e.g. flat prior)

February 4-6, 2003 Bayesian Statistics Page 13 of 20

Questions?

• How do I determine the Likelihood function: P(data|x)?

• we can derive it for each model with parameter(s) x
• How do I figure out what the normalization constant is?
• the constant that makes the posterior P(x|data) integrate to 1 (to make it a
pdf)
• How do I know what the posterior P(x|data) is?
• it will be constant*P(data|x) for classical statistics
• It will be constant*P(data|x)*P(x) for Bayesian statistics

February 4-6, 2003 Bayesian Statistics Page 14 of 20

Clean Slate vs. Updating?

• Clean Slate:
• set P(x) (prior) back to e.g. flat prior & retest.
• Bayesian Updating:
• set P(x) to previous posterior P(x|previous data) from before & retest.
• Project to decide whether to wipe the slate clean or include
previous data.

February 4-6, 2003 Bayesian Statistics Page 15 of 20

Summary: Bayesian Testing

• Avoids Coin Flipping by including prior knowledge

• General formula which we will now apply to different models &
parameters for testing
• Classical Statistics is just a subset of Bayesian statistics

February 4-6, 2003 Bayesian Statistics Page 16 of 20

Bayesian Parameter Estimation

P({x}|data) = Constant * P(data|{x}) * P({x})

{x}: one or more parameters (e.g. a & b)

P({x}|data): Posterior distribution of {x}
Constant: Normalization Factor
P(data|{x}): Likelihood function
P({x}): Prior distribution of {x}

February 4-6, 2003 Bayesian Statistics Page 17 of 20

Bayesian Modeling

P(model|data) = Constant * P(data|model) * P(model)

model: one of a set of models (e.g. {Weibull, Normal, etc.})

P(model|data): Posterior probability of model given data
Constant: Normalization Factor
P(data|model): P(data) when model is chosen (evidence)
P(data)=1/constant from parameter estimation with model
P(model): Prior probability of model

February 4-6, 2003 Bayesian Statistics Page 18 of 20

Bayesian Example

• A prize is behind one of three doors.

• After you initially choose one of the three, the host shows you one of
the two remaining where the prize is not located.
• What is the probability that the prize is behind the door you chose?
• What is the probability that the prize is behind the other remaining
door?

February 4-6, 2003 Bayesian Statistics Page 19 of 20

Training Summary

• Learned about pdf, Cdf & R distributions

• Learned the definition of conditional probability
• Learned about prior, posterior & likelihood
• Know the difference between Classical & Bayesian statistics
• Learned how Bayesian Statistics are used in RE tests

February 4-6, 2003 Bayesian Statistics Page 20 of 20