PROBABILITY THEORY

INTRODUCTION
Probability Theory: Deals with statistical characterization of random signals, which are viewed as second pillar of communication theory. [Refer --Haykin] In every practical communication system, random signals are encountered, e.g. A voice signal may comprise of randomly spaced bursts of energy of random duration.

UNCERTAINTY
What is Uncertainty ? The quality or state of being not clearly/completely known. Causes Ignorance, complexity, randomness, vagueness. Our knowledge about uncertainty Past happenings of similar events, analysis.

DECISION MAKING UNDER UNCERTAINITY

Collect Data Model the Phenomenon Extrapolate Make Decisions

Probability : Approaches
1. Based on intuition : He will probably come today. 2. Classical : Ratio of favorable to total outcome 3. Empirical : Probability as a measure of frequency of occurrences.

4. Probability based on Axiomatic Theory

Definition of Probability Classical Definition : Mathematical or a priori definition of probability : If there are n exhaustive, mutually exclusive and equally likely cases and m are favourable to an event A , the probability of happening of A is defined as m/n. Fails if cases are not equally likely, and if total number of cases are infinite.

Definition of Probability
Relative Frequency Definition: Statistical or Empirical Definition of Probability

: If trials are repeated great number of times under essentially the same conditions, then the limit of ratio of the number of times that an event happens to the total number of trials as the number of trials increases indefinitely is called the probability of the event.

Relative Frequency Definition:

It is assumed that the ratio approaches a finite and unique limit. It is the limit of the relative frequency of the happening of the event.

Terminology(1)
Experiment, trial, outcome, event, cases Repeated trial, Frequency, Equally likely, mutually exclusive, exhaustive, equiprobable Set Theory ; Element, set, subset, universal set, union, intersection, complement, difference, Cartesian product, Venn diagram, tree diagram A priori, a posteriori Probability Space

Terminology
Experiment : Action we do, intend to do or conceive Trial : Performing the experiment once. Outcome : Result of a trial. Case : Outcome Event : An event comprises of one or more outcomes. Null event comprises of no outcome. An event occurs if any of the comprising outcome results. Experiment are of two types : Deterministic and Nondeterministic/ random Scientific experiments are deterministic The subject of probability is concerned with non-deterministic experiments.

Terminology(2)
Equally likely/Equiprobable : Two events are equally likely or equiprobable if there is no reason to expect any one of the events in preference over the other event. Mutually Exclusive : Two events are mutually exclusive if they cannot occur at the same time. They do not have common outcome. Occurrence of one ensures non occurrence of the other. Exhaustive : A set of events is exhaustive if they encompass all possible outcomes and one must occur in a trial. There is no outcome of a trial that is not included in the set of exhaustive events.

Terminology
A priori : From cause to effect, from generalization to particular instances, based on theory, logic instead of on experience. Derived by logic without observed fact. A posteriori : From effect to causes, from particular instances to generalizations, based on observed facts and experience. Derived from observable fact. Field : Mathematical entity, algebraic structure, A set with two operations that satisfies certain conditions.

Definition of Probability
Non deterministic experiment, A trial is single performance of the experiment, Outcome is the result of performing the experiment. A trial gives an outcome, Sample Space : Set of all possible outcomes of an experiment. An outcome is an element of sample space Sample space may be finite or infinite, discrete or continuous Countable or un-countable. Event : An event is a subset of Sample space. Null set is considered an event which never happens Sample space is also an event which certainly occur. Probability is a function of event satisfying certain axioms. The Domain is the set of events of a Sample space. Co-domain or Range is the set of real numbers between 0 and 1. If A is an event of Sample Space S, then P(A) denotes the probability of A

Axiomatic Definition of Probability

The probability P(A) of an event A of a Sample Space S is a function satisfying the following Axioms Axiom 1 : P(A) 0 Axiom 2 : P(S) = 1 Axiom 3 : P(AUB) = P(A) + P(B) if A and B are mutually exclusive events. If A1, A2,. An are mutually exclusive events then P(A1+A2+.+An) = P(A1) + P(A2) + + P(An) If not mutually exclusive then P(A1+A2+.+An) P(A1) + P(A2) + + P(An) These axioms do not assign probabilities to events. They control and restrict assignment of probability to events.

Assignment of Probability :
Must satisfy the axioms. Probabilities are assigned on the basis of

1. 2. 3. 4.

Estimate obtained from past experience Analysis of conditions underlying the experiment Reasonable assumptions ( e.g. all outcomes are equally likely ) Evaluations based on reasons and judgments ( e.g. based on feeling, not based on mathematical deductions )

P(A)=1 Event A will certainly happen P(A)=0 Event A will certainly not happen P(A)= 0.6 : Nothing can be said about A in the next trial If the experiment is repeated large number of times say 1000 Then A will happen approximately 600 times Say it is certain that Number of times A will happen is between 590 to 610

Assignment of Probability
If the Sample Space is Infinite, continuous, uncountable , say the set of real numbers, we look for a function f(x) such that

1. 2. 3.

f ( x)dx 1
b

P ( x b)

f ( x)dx
b a

P (a x b) f ( x)dx

Equality of Events
Let A, B be any two events The two events A,B are called equal if they consists of the same elements. They are called equal with probability 1 , iff P(A)=P(B)=P(AB). If P(A)=P(B) then the events are called equal in probability. No conclusion can be drawn about P(AB). The events may be mutually exclusive.

Conditional Probability :
If A and B are any two events in Sample space S and P(B) is non zero, the conditional probability of A relative to B denoted by P( A|B ) is defined as

P( A B) P( A | B) P( B)

Conditional Probability is a defined quantity and cannot be proven P(A|S)=P(A) P(S|B)=1 If B is a sub set of A then P(A|B)=1 If A is a subset of B then P(A|B) => P(A) P((A1+A2)|B)=P(A1|B) + P(A2|B)

Conditional Probability :
P(A) is defined assuming Sample Space S. P(A) is the conditional probability of A relative to the Sample Space S P(A|B) is defined assuming B i.e. assuming that B has already happened. P(A|B) can be thought as the probability of A relative to the reduced Sample Space B. Every event A and its complement A partitions the Sample space into two mutually exclusive (disjoint) and exhaustive sets. Unconditional or Total Probability P(A) can be computed in terms of Conditional Probabilities P(A|Bi)

Bayes Theorem
Let A1, A2, A3,........, An be n mutually exclusive, exhaustive events on Sample Space S. Let B be any event in S

i 1

Ai A1 A2 A3 .......... An S

P ( Ai B) P( B | Ai ) P( Ai ) P( Ai | B ) n P( B) P( B | Ai ) P( Ai )
P(A) is priori probability, P(A|B) is a posteriori probability. P(A) is not based on actual happening, P(A|B) is based on actual happening B
i 1

When to Apply Bayes' Theorem

The sample space is partitioned into a set of mutually exclusive events { A1, A2, . . . , An }. Within the sample space, there exists an event B, for which P(B) > 0. The analytical goal is to compute a conditional probability of the form: P( Ak | B ). We know at least one of the two sets of probabilities described below. P( Ak B ) for each Ak P( Ak ) and P( B | Ak ) for each Ak

Important Example
BINARY SYMMETRIC CHANNEL Refer to Haykin

Independent Event :
Two events A, B with non zero probabilities are called independent events if P(A|B)=P(A). For two independent events A, B P(B|A)=P(B) P(AB)=P(A)P(B) P(AB)=P(A)P(B) is a necessary and sufficient condition for the events A,B to be independent. Events cannot be both mutually exclusive and independent. P(AB) must not be zero. There will be a non zero intersection of A and B. More than two events are independent if they are independent pair wise. If A, B are two independent events, occurrence of one is not affected by the occurrence of the other.

Joint Event :
A joint event comprises of two or more events happening together. AB, ABC are joint events. P(AB)=P(A|B)P(B)=P(B|A)P(A) P(ABC) = P(A) P(B|A) P(C|AB)