Inequalities and Limit Theorems

Subject: Probability and Statististics

Semester-VI
Lesson: Inequalities and Limit Theorems
Lesson Developer: Dr. Lakshmisree
Bandopadhyaya
College/Department: Deshbandhu College,
Retd. Associate Professor
Department of Mathematics,
University of Delhi

Institute of Lifelong Learning, University of Delhi 1

 Inequalities and Limit Theorems

Index

Markov’s Inequality 3

Chebyshev’s Inequality 3

Exercises 8

Central Limit Theorem 8

Weak Law of Large Numbers 13

Interpretation of Weak Law of Large Numbers 14

Strong Law of Large Numbers 14

Interpretation (Strong Law of Large Numbers) 15

Exercises 17

Institute of Lifelong Learning, University of Delhi 2

 Inequalities and Limit Theorems

MARKOV'S INEQUALITY
We start this chapter by proving a result known as Markov's Inequality
Theorem 1.1 (Markov's Inequality)
Let X be a random variable, which takes only non-negative values. Then
E( X )
P[ X  a]   a  R
a
Proof:
Let X be a continuous random variable with probability density f ( x), x  0


E( X ) = 0 x. f ( x)dx
a 
= 0 xf ( x)dx  
a
xf ( x)dx
a 
 0 xf ( x)dx  
a
af ( x)dx

 a a f ( x)dx ( x  a  xf ( x)  af ( x)
 
  xf ( x)  a  f ( x)dx ,
a a
x
also xf ( x)  0  0 xf ( x) dx  0 )
= aP( X  a)
 E ( X )  aP( X  a)
1
 E ( X )  P( X  a ) (a > 0)
a
Note: If X is discrete random variable, same proof works, replacing integral by
summation.

CHEBYSHEV'S INEQUALITY
Theorem 1.2 (Chebyshev's inequality):
If X is a random variable with mean µ and variance  2 , then
2
P(| X   |  k )  ,  kR
k2
Proof:
P(| X   |  k ) = P(| X   |  k )
2 2

= P(Y  k ), where Y  | X   |
2 2

So Y takes only non-negative values, using Markov's Inequality

Institute of Lifelong Learning, University of Delhi 3

 Inequalities and Limit Theorems

E (Y )
P(Y  k 2 ) 
k2
E (| X   |2 )
 P(| X   |2  k 2 ) 
k2

Var X  2
 P(| X   |  k )   2
k2 k
Note: In Chebyshev's inequality, replacing k by K , we get P(| X   |  K ) 
Var X  2
1
 
K 2 2 K 2 2 K 2
1
So, a restatement of Chebyshev's inequality becomes P(| X   |  K )  . . .
K2
(1)
Again P(| X   |  K ) = 1  P(| X   |  K )
. Replacing this value in (1), we get another form of the same inequality as
1
P(| X   |  K )  1 
K2
Example 1.1
Use Chebyshev's Theorem to find the least value of k, such that the probability that
X lies between   k and   k is atleast 0.95.
Solution:
By Chebyshev's inequality
1
P(| X   |  k )  1
k2
1
 P(  k  X    k )  1
k2
We have to choose k, such that
1
1  0.95
k2
1
0.05 
k2
1 100
k2    20
0.05 5
k  20
So the least value of k is 20 .

Institute of Lifelong Learning, University of Delhi 4

 Inequalities and Limit Theorems

Example 1.2
If E ( X )  3 and E ( X )  13 , find a lower bound for P(2  X  8)
2

Solution:
  E( X )  3
 2 = E( X 2 )  ( E( X ))2  13  9  4
  = 2
P(2  X  8) = P(2  3  X  3  8  3) (   3)
= P(5  X  3  5)
= P(| X  3|  5)
By Chebyshev's Inequality
2
P(| X   |  k )  1 
k2
4 21
 P(| X  3|  5)  1  
52 25
21
Hence the lower bound for P(2  X  8) is
25
Example 1.3
A random variable X takes the values –1, 1, 3, 5 with associated probabilities
1 1 1 1
, , and . Find by direct computation P(| X  3|  1) . Find an upper bound to this
6 6 6 2
probability by applying Chebyshev's Inequality.
Solution :
The distribution for the random variable X is as follows:
X f ( x)
1
–1
6
1
1
6
1
3
6
1
5
2
1
P(| X  3|  1) = 1  P(| X  3|  1)
= 1  P(1  X  3  1)

Institute of Lifelong Learning, University of Delhi 5

 Inequalities and Limit Theorems

= 1  P(2  X  4)

= 1  P( X  3) (only value X takes between 2 and 4 is 3)

1 5
= 1 
6 6
2
Now by Chebyshev's inequality P(| X   |  k )  , where   E( X ) and
k2
 2  Var X
1 1 1 1 1 5
 2  Var X = 1   1   3   5     3
6 6 6 2 2 2
1 1 1 1 11 25 86 43
E ( X 2 ) = (1)2   (1)2   (3)2   (5)2     
6 6 6 2 6 2 6 3
43 16
Var X   2 = E ( X 2 )  ( E ( X ))2 = 9 
3 3
Putting k = 1 in the above form of Chebyshev's Inequalty (1)
16
16
 P(| X  3|  1) < 32 
1 3
16
So the upper bound for the required probability is .
3
Example 1.4
An unbiased coin is tossed 100 times. Show that the probability that the number of
heads will be between 30 and 70 is greater than 0.93.
Solution :
Coming of head be taken as a success when an unbiased coin is tossed.
1
Let p = Probability of success =
2
Number of tosses = n = 100
1
  Mean = np  100   50
2
1 1
 2  Variance = npq  100    25
2 2
P(30  X  70) = P(30  50  X  50  70  50)  P(20  X  50  20)
= P(| X  50 |  20)
By Chebyshev's inequality

Institute of Lifelong Learning, University of Delhi 6

 Inequalities and Limit Theorems

Var X
P(| X  50 |)  k )  1
k2
Putting k = 20, we get
25 25 15
P(| X  50 |  20)  1  2
1   0.937  0.93
(20) 400 16
Example 1.5
For the Geometric Distribution f X ( x)  2 x , x  1,2,3,....., prove that Chebyshev's
1 15
inequality gives P(| X  2 |  2 |)  , while the actual probability is .
2 16
Solution:
Put X = Y + 1
If X varies as 1, 2, 3, .... then Y will vary as 0, 1, 2, 3 ....
fY ( y)dy = f X ( x)dx
y
dx 1 1
 fY ( y ) = f X ( x)  2 x 1  2 y 1     , y = 0, 1, 2, ...
dy 2 2
1
 Y is G(p), where p =
2
1
q 2
Mean = E (Y )    1
p 1
2
1
q
Var Y =  2 2
p2 1
4
P(| X  2 |  2) = P(| Y  1  2 |  2)
= P(| Y  1|  2)  P(| Y  E(Y ) |  2)

By Chebyshev's Inequality
VarY 2 1 1
P(| Y  1|  2)  1  2
1 2 1 
2 2 2 2
Actual probability of | X  2 |  2 is P(0  X  4)

= P(0)  P(1)  P(2)  P(3)  P(4)

1 1 1 1
= 0    (X does not take the value '0')
2 22 23 24

Institute of Lifelong Learning, University of Delhi 7

 Inequalities and Limit Theorems

1 1 
1  4 
=
2  2   1  1  15
1 16 16
1
2

Exercises
1. Let X be the score of a die. Show that Chebyshev's inequality gives
P(| X   |  2.5)  0.47 , while the actual probabilty is zero.
2. Two dice are tossed. Let X be the sum of the scores. Show that Chebyshev's
35 1
inequality gives P(| X   |  3)  , while the actual probability is .
54 3
3. A symmetric die is thrown 720 times. Use Chebyshev's inequality to find the
lower bound for the probability of getting 100 to 140 sixes.
Ans. : 0.75
4. Use Chebyshev's inequality to determine how many times a fair coin must be
tossed in order that the probability will be atleast 0.95 that the ratio of the
number of heads to the number of tosses will be between 0.45 and 0.55.
Ans. : atleast 2000 times.
5. If the number of items produced is a random variable with mean 500 and
variance 100, what can be said about the probability that the production is (a)
atleast 1000 (b) between 400 to 600
1
Ans. : (a) atmost (b) at least 0.99
2

CENTRAL LIMIT THEOREM

Probability Theory's number one result is this Central Limit Theorem. Besides its
theoretical interest and importance, this theorem provides a simple method for
computing approximate probabilities for sums of independent random variables.

Theorem 1.3 (Central Limit Theorem)

Let X1 , X 2 ,...., X n be independent and identically distributed random variables, each
X1  X 2  ....  X n
with mean µ and variance  2 . Let X n  be their mean variable, then
n
Xn  
the distribution of Z  tends to standard normal distribution, as n   .

n

Institute of Lifelong Learning, University of Delhi 8

 Inequalities and Limit Theorems

Proof: To show that Z tends N(0, 1) as n   , we prove that M.G.F. of Z is same as

t2
the M.G.F. of N(0, 1) i.e. equal to e 2 as n   .
M Z (t ) = E (e ) tz

 t  X n   
  

= Ee  n  

= e

t n
  
E e
t n

Xn

= e

t n

E e
t n  X1  X 2 .... X n 
 

n

 
= e

t n
 E e
t
 n
( X1  X 2 .... X n )

t n

= e  E  et ( X  X .... X ) 
1 2 n
t
 t
 n
t n

= e  E  et( X )  E  et( X ) ......E  et( X ) 
1 2 n

( X1 , X 2 ,.... X n are independent)

t n

= e 
M X1 (t ) M X 2 (t )....M X n (t )
t n

= e 
( M (t ))n
As X1 , X 2 ,...., X n are identically distributed, therefore they have a common M.G.F.
M (t ) .
Now taking logarithm of both sides
t n
log M z (t ) =   n log M (t )

t n
=   nK (t )

( K (t ) is the cummulant generating function of any of the identically distributed
variables X1 , X 2 ,..., X n )

t n  t 2 t 3 
=   n  k1t   k2  k3  .... 
  2! 3! 
where k1   , k2  2   2 k3  3 ,....
t
Replacing t  by , we get
 n

Institute of Lifelong Learning, University of Delhi 9

 Inequalities and Limit Theorems

t n tn n 2t 2 n3t 3
=      .....
  n 2 2 n  3n n
t n t n t 2  t3  1 
=     33  involving terms of O  
  2  n  n
Let n  
t2
log M Z (t ) 
2
t2
i.e. M z (t )  e 2 , which is the M.G.F. of N(0, 1). Hence Z has the distribution of
N(0, 1) when n  
 X1 , X 2 ,...., X n  1 1
Note: E ( X n ) = E    ( E ( X1 )  E ( X 2 )....E ( X n )) = n  .
 n  n n

 X  X 2 .... X n  1 1 2
Var X n = Var  1   2 (Var X 1  Var X 2 ........  Var X n ) = (n 2 ) 
 n  n n 2
n
( X1, X 2 ,...., X n are independent).

Thus central limit theorem (CLT) shows that X n is normally distributed with mean
2
µ and variance , where n is very large and X1 , X 2 ,...., X n are independent and
n
identically distributed random variables.
n

 X i  n
Also a restatement of C.L.T. can be Z  i 1
is a standard normal variate
 n
when n   , where X1 , X 2 , ...., X n are independent and identically distributed
random variables with mean µ and variance  2 .
Example 1.6
Let X1 , X 2 ,.... X10 be ten independent and identically distributed random variables
 10 
each following Unif (0, 1) . Find P   X i  7
 i 1 
Solution:
Xi  (   ,   0,   1
  0 1 1
So,   E( X i )   
 2 2
(     1
Variance ( X i ) =  2 =  ,  i  1, 2,...,10
12 12

Institute of Lifelong Learning, University of Delhi 10

 Inequalities and Limit Theorems

1
Using Central Limit Theorem, X i is a normal variate with mean n  10  5
2
10 5
and variance  n   , when n is large.
2

12 6
So estimation for P(X i  7) can be found using standard normal distribution
table.

 X i  n 7  5 
P(X i  7) = P   
  n 5 

 6 

 2 6
= P Z  
 5 
= P(Z  2.2)

= 1   (2.2) (Table II of Normal distribution)

= 1  0.9861
= 0.0139
Example 1.7
A research worker wishes to estimate the population mean using a sample large
enough that the probability will be 0.95 that the sample mean will not differ from the
population mean by more than 25% of the population S.D. Use C.L.T. to find how
large a sample should be taken.
Solution :
Let the sample be X1 , X 2 ,..... X n taken from the population with mean µ and
variance  2 . So each X i have the same mean and variance.

X1  X 2  ....  X n
Sample Mean =  Xn
n
Xn  
By C.L.T., Z   N (0, 1) when n  

n
We have to find n, such that
P(| X n   |  25% ) = 0.95
| X   | 1 n
P n  .  = 0.95
  4  

 n 

Institute of Lifelong Learning, University of Delhi 11

 Inequalities and Limit Theorems

 n
P | Z |    0.95
 4 
From the standard Normal table P(| Z |  1.96)  0.95

n
  1.96  n  1.96  4  n  61 Approx.
4
Example 1.8
X  np
Use C.L.T. to show if X  B(n, p), then  N (0, 1) as n  
npq
Proof:
Consider a sequence of n Bernoulli trials with constant probability 'p' of success at
each trial.
For each i = 1, 2, ...., n, define a random variable Xi as follows
Xi = 1, if ith trial is a success
= 0, if ith trial is a failure
Then clearly the Xi's are independent and each having E ( X i )  1  p  0  q  p

 2  Var X i = E( X i2 )  ( E( X i ))2  12  p  02  p  p 2
= p  p  p(1  p)  pq
2

X  B(n, p) given
X = No. of successes in n trials
By definition of Xi's, X1  X 2  ....  X n = No. of successes in n trials = X
X 1  X 2  ....  X n
p
By C.L.T. n  N (0, 1) when n  
pq
n
X  np
 Z  N (0, 1) when n  
npq
Example 1.9
The life of a battery has mean 40 and S.D. 20. Find the probability that a stock of 25
batteries will together last atleast 1100 hours.
Solution:
Let Xi denote the life of the ith battery, i = 1, 2, ..., 25.
  40,   20

Institute of Lifelong Learning, University of Delhi 12

 Inequalities and Limit Theorems

25
Total life of 25 batteries = 
i 1
Xi

By re-statement of CLT
X i  n
Z  N (0, 1) when n is large.
 n
 P(X i  1100)
 X  n 1100  25  40 
= P i  
  n 20 25 
 100 
= PZ  
 100 
= P( Z  1)  1  P( Z  1)
= 1 – 0.8413 (Using Table II of Normal Distribution)
= 0.1587

WEAK LAW OF LARGE NUMBERS

Theorem 1.4 (Weak Law of Large Numbers)
Let be a sequence of random variables, such that E ( X n )  n , Var ( X n )   n . For
2
Xn
Bn
each n, let Bn  Var ( X1  X 2  ....  X n ) and suppose that lim 0 , then given any
n2
n 

  0, and   0 , there exist n0  N , such that P(| X n  E( X n ) |  )  1    n  n0

X1  X 2  ....  X n
Proof: X n 
n
1 B
Var X n  2
Var ( X1  X 2  ....  X n )  2n
n n
Using Chebyshev's inequality to the variable X n

Var X n
P(| X n  E ( X n ) |  ) > 1
2
Bn
= 1 ... (1)
n 2
2

Bn B
Given lim  0  lim 2 n 2  0
n  n 2 n  n 
 corresponding to   0,  n0  N , such that

Institute of Lifelong Learning, University of Delhi 13

 Inequalities and Limit Theorems

Bn
0 <   n  n0
 n2
2

Bn
  <   n  n0
 n2
2

 Bn
> 
2 n 2
Bn
 1 > 1 
 n2
2

From (1), we get

Bn
P(| X n  E ( X n ) |)  1   1   ,  n  n0
 n2
2

Interpretation of Weak Law of Large Numbers

P(| X n  E( X n ) |  )  1    n  n0 for all  howsoever small (a)

Also P(| X n  E ( X n ) | )  1 (b)

Combining (a) and (b)

P(| X n  E ( X n ) |  )  1 as n  

that is | X n  E ( X n ) |   becomes a sure event when n is large.

STRONG LAW OF LARGE NUMBERS

Theorem 1.5 (Strong Law of Large Number)
Let X1 , X 2 ,...., X n be independent and identically distributed with mean µ and same
variance  2 .
X1  X 2 .....  X n
Let X n  be their mean variables, then for each  0 ,
n
lim P(| X n   | ) = 1
n 

Proof:
1 n
E( X n ) = ( E ( X1 )  E ( X 2 )....E ( X n ))  
n n
1
Var X n = 2 (Var X 1  Var X 2 .....Var X n ) ( X1 , X 2 ,.... X n are independent)
n
n 2  2
= 
n2 n

Institute of Lifelong Learning, University of Delhi 14

 Inequalities and Limit Theorems

2
Bn 2
we have lim 2
= lim n2  lim 3  0
n  n n  n n  n

So, all the conditions of Weak Law of Large numbers hold

It follows P(| X n  E ( X n ) |)  1     ,howsoever small  n  n0 ... (1)
Also P(| X n  E ( X n ) |)  1 ... (2)
 P(| X n   | )  1 as n   (from (1) and (2))

Interpretation (Strong Law of Large Numbers)

X1  X 2 ..... X n
With probability 1,  , as n  
n
i.e. it becomes almost certain that a sample mean tends to population mean,
when large number of samples are taken from a population.
Example 1.10
Let there be a sequence of independent random variables defined as

P( X k   2 k )  2 k 1 and P( X k  1)  1  2 k  . Show that Weak Law of Large

1
2
Numbers hold.
Solution:
Probability distribution of Xk is as follows:

Xk P( X k )

2 k 2 k 1
2 k 2 k 1
1
1 (1  2 k )
2
1
–1 (1  2 k )
2
1 1
E ( X k ) = (2 k )(2 k 1 )  2 k  2 k 1  1  (1  2 k )  1  (1  2 k )  0
2 2
Var ( X k ) = E ( X k2 )  ( E ( X k ))2

2 k 1 1
= 2  2 k 1  22 k  2 k 1  1  (1  2 k )  1  (1  2 k )  02
2 2
3 k 1 1
= 22  2  (1  2 k )
2

Institute of Lifelong Learning, University of Delhi 15

 Inequalities and Limit Theorems

3k 1 1
= 2  1  2 k  k
1 k
8 2
Bn = Var ( X1  X 2  ....  X n )

= Var X1  Var X 2  .....  Var X n ( X1 , X 2 ,... X n being independent)

n n
1 1 
= 
k 1
Var X k    k  1  k 
k 1  8 2 
1 1  1 1 
1  n  1  n 
=
8 8   n 2 2 
1 1
1 1
8 2
1 1 1
= 1  n   n  1  n .
7 8  2

Now,
1 1 1
1  n   n  1  n
Bn
lim 2 = lim 7  8  2 0
n  n n  n2
Hence Weak Law of Large Numbers hold.

Example 1.11
Let Xi assume value i and  i with equal probabilities. Prove that the sequence
Xn of independent random variables satisfies Weak Law of Large Numbers if
1

2
Solution:
X i has the following distribution
Xi P( X i )
1
i
2
1
i
2
 E( X i ) = 0
Var ( X i ) = E ( X i2 )  ( E ( X i ))2

Institute of Lifelong Learning, University of Delhi 16

 Inequalities and Limit Theorems

1 2 1 2
2
= i . i . i
2 2
Bn = Var ( X1  X 2 ....  X n )
= Var X1  Var X 2  ...  Var X n ( X1 , X 2 ,..., X n are independent )
n n
=  Var X )   i   1   2   ......  n 
i 1
i
i 1
2 2 2 2

2 1
n
 (By Euler Mclaurin Approximation)
2  1
Bn n2 1 n2 1
=   0 as n  
n2 (2  1)n2 2  1
1
 Weak Law of Large Numbers hold only when 2  1  0    .
2
Exercises
1. Let X n be a sequence of random variables, such that
(a) E ( X i )  E ( X j ),  i, j

(b) is Var X n bounded above.

(c) Cov ( X i , X j )  0,  i  j

Prove that weak law of large number holds for Xn .

Hint: When X1 , X 2 , ...., X n are not independent
n n
Var ( X1  X 2  .......  X n ) = i 1
Var X i  2  Cov( X i , X j )
i , j 1
i j

2. For the sequence Xn of random variables, show that Weak Law of Large
1 2
Numbers hold, where P( X k  1)  and P( X k  0)  1  , k  1, 2, ...., n
k k
1
3. For the sequence Xn of random variables, where P( X k  )  pk and
k
 1 
P Xk 1   1  pk , k  1,2,3......, show that Weak Law of Large Numbers
 k
hold.
Hint: Assume lim pn  l (finite) and then use Cauchy's first theorem on limits.
n 

Suggested Books

Institute of Lifelong Learning, University of Delhi 17

 Inequalities and Limit Theorems

1. "Probability and Queueing Theory" by Dr. R. Paugalarasu Tata Magraw Hill

Education Private Limited.
2. Robert V. Hogg, Joseph W. McKean, Allan T. Craig, "Introduction to
Mathematical Statistics", Pearson Education, Asia 2007.
3. Sheldon Ross, "Introduction to Probability Models" (9th Edition) Academic
Press, Indian Reprint 2007.
4. "Introduction to the theory of statistics" by Mood and Graybill, Tata McGraw
Hill, Reprint, 2007.
5. "Probability and Random Process" by R. Pugalarasu, Tata-McGraw Hill
Education Private Limited.

Institute of Lifelong Learning, University of Delhi 18