CME308: Assignment 1 Solutions April 13, 2016

Problem 1: On the convergence of sequences of random variables.

This problem covers some useful convergence results. In particular implications among various types of
convergence. It is more detailed and contains more than what was asked. Only items 6 and 7
are in problem 1.
If {Xn }
i=1 are a sequence of random variables. We have the following definitions:
P

We say Xn converges to X in probability, denoted as Xn X , if for any  > 0,

lim P(|Xn X | > ) = 0.

a.s.

We say Xn converges to X in probability 1 (almost surely), denoted as Xn X , if

P({ : lim Xn () = X ()}) = 1.
n

Here is the underlying probability space on which all the random variables are defined. That is,
there is a set , a probability space, and the random variables are real valued functions defined on
this set, Xn (). There is also a family F of subsets of that is closed under complements, countable
unions and intersections. In addition, the set of all such that Xn () x belongs to F for
all real x (which says Xn is measurable). There is also a probability measure P on (, F) such that
FXn (x) = P { : Xn () x}.
Lp

We say Xn converges to X in the mean of order p, Lp , p 1, denoted as Xn X , if

lim E|Xn X |p = 0.

We say Xn converges to X in distribution (or weakly), denoted as Xn X , if

lim Fn (x) = F (x)

for any continuous points x of F . Here Fn (), 1 n is the cdf of Xn . In this case, we also say
w
Fn () converges to F () weakly, denoted as Fn F .
Prove the following statements.
Lp

1. Show that for p 1, Xn X implies Xn X . And give an example to show the converse
statement doesnt hold.
a.s.

2. Show that Xn X implies Xn X . Use the bounded convergence theorem which says that is
a.s.
if |g(x)| is bounded uniformly for all real x and Xn X then Eg(Xn ) Eg(X ).
L1

3. Show that Xn X implies EXn EX .

D

4. Show that if Xn X then Xn X . Conversely, if Xn X and X is a non-random

P
constant, then Xn X .
P

5. Show that if Xn X and f () is a continuous function, then f (Xn ) f (X ).

6. Show that Xn converges to X in distribution if and only if E{g(Xn )} converges to E{g(X )} for
every bounded and continuous function g.
D

7. Show that if Xn X and Y c where c is a constant, then (Xn , Yn ) (X , c). More

specifically, prove that for
lim P(Xn x, Yn y) = P(X x)1(c y)

for any continuous points of F (x, y) := P(X x)1(c y).

1

CME308: Assignment 1 Solutions April 13, 2016

Solution:

1. Chebyshev inequality:
P(|Xn X | > ) = P(|Xn X |p > p )

E|Xn X |p
0 (n )
p

2. Almost sure convergence can be expressed in terms of the set: { : limn Xn () = X ()} =
T
S
T
a.s.
>0 N 1 nN { : |Xn () X ()| < }. Therefore if Xn X , then  > 0,
P(

[ \

{ : |Xn () X ()| < }) = 1

N 1 nN
a.s.

Set Yn = Xn X . Then Yn 0 by assumption. Define g(x) = I|x|> which is uniformly bounded

for all x. By the bounded convergence theorem,
Eg(Yn ) Eg(0) = 0, that is, limn P({ : |Xn () X ()| > }) = 0
3. |EXn EX | E|Xn X | 0

(n )

4. Denote Fn as the cdf for Xn , 1 n . Then for any continuity point x of F (x), we have
Fn (x) = P(Xn x) P(X x + ) + P(|Xn X | ).
Let n and then let  0, we have lim supn Fn (x) F (x). Similarly,
Fn (x) = P(Xn x) P(X x ) P(|Xn X | ).
Let n and then let  0 we have lim inf n Fn (x) F (x). Hence limn exists and equals
to F (x).
D

On the other hand, if Xn X c,

P(|Xn c| > ) = P(Xn > c+)+P(Xn < c) = 1Fn (c+)+Fn ((c)) 1Fn (c+)+Fn (c).
Here Fn ((c )) means lim0+ Fn (c ). Taking c + , c as two continuity points of F (x)
with F (c ) = 0, F (c + ) = 1, the limit yields
lim sup P(|Xn c| > ) 1 1 + 0 = 0.
n

Hence Xn X in probability, where X c.

5. For each  > 0, define
A = {x : y, s.t. |x y| < , |f (x) f (y)| > }.
Since f () is continuous everywhere, thus A as 0.
Now |f (Xn ) f (X )| >  implies that |Xn X | > for some , or X A . Thus
P(|f (Xn ) f (X )| > ) P(|Xn X | > ) + P(X A ).
First take n we have
lim sup P(|f (Xn ) f (X )| > ) P(X A ).
n

Take 0 proves the problem.

CME308: Assignment 1 Solutions April 13, 2016

6. We show that if Xn converges to X in distribution then E{g(Xn )} converges to E{g(X )} for every
bounded and continuous function g. The converse is shown in the notes.
Given  > 0, choose points x0 < x1 < < xN such that (a) they are continuity points of F , (b)
F (x0 ) < , (1 F (xN )) < , and (c) the given continuous function g(x) can be approximated by a
step function gN (x) in the interval x0 x xN such that maxx0 xxN |g(x) gN (x)| < . Now we
have that
Z
Z
|
g(x)dFn (x)
g(x)dF (x)|
(1)

Z x0
Z x0
|
g(x)dFn (x)
g(x)dF (x)|
(2)

Z xN
Z xN
+|
g(x)dFn (x)
g(x)dF (x)|
(3)
x0
x0
Z
Z
+|
g(x)dFn (x)
g(x)dF (x)|
(4)
xN

xN

The first and third terms on the right are less than a constant times  as n because of (a),(b)
and the fact that g is bounded. In the middle term, we add and subtract gN to get
Z xN
Z xN
|
g(x)dFn (x)
g(x)dF (x)|
(5)
x0
x0
Z xN
Z xN
|
g(x)dFn (x)
gN (x)dFn (x)|
(6)
x0
x0
Z xN
Z xN
+|
gN (x)dFn (x)
gN (x)dF (x)|
(7)
x0
x0
Z xN
Z xN
+|
gN (x)dFn (x)
g(x)dF (x)|
(8)
x0

x0

The first and third terms on the right side are less than a constant times  because of (c). The middle
term is now a finite sum of differences Fn F at continuity points of F , so it goes to zero as n
by the hypothesis.
7. For any continuity point x of FX (x) and y < c, we have
P(Xn x, Yn y) P(Yn y) 0 = P(X x, Y y).
For any continuity point x of FX (x) and y > c, we have
P(Xn x, Yn y) = P(Xn x) P(Xn x, Yn > y).
As P(Xn x, Yn > y) P(Yn > y) 0, we obtain that P(Xn x, Yn y) FX (x) = P(X
x, Y y). This completes the proof.

CME308: Assignment 1 Solutions April 13, 2016

Problem 2: On the use of basic conditional expectation
Keep tossing a coin repeatedly and denote the outcome of a head as H and of a tail as T, with P(H) =
P(T ) = 1/2. An n pattern is a sequence of n tosses with a fixed outcome, e.g. (HHTH) is 4 pattern,
(TT) is a 2 pattern. Denote by N the time that the 3 pattern (HTH) first appears ( if it doesnt appear,
N = ). Compute the expectation of N .
Solution:
The lwa of total probability states that:
EX =

E(X|n )P (n )

where {n : n 1} is a partition of the probability space , that is, it is the union of disjoint sets
{n : n 1}.
Condition on the first outcome:
EN =

1
1
(E(N |H) + E(N |T )) = (E(N |H) + 1 + EN )
2
2

Condition on the second outcome:

E(N |H) =

1
1
(E(N |HH) + E(N |HT )) = (E(N |H) + 1 + E(N |HT ))
2
2

Condition on the third outcome:

E(N |HT ) =

1
1
(E(N |HT H) + E(N |HT T )) = (3 + 3 + EN )
2
2

EN = 10.

CME308: Assignment 1 Solutions April 13, 2016

Problem 3: Expected value nonnegative r.v.
Let X be a nonnegative r.v., i.e. X 0 w.p. 1. Show that,
Z
E[X] =
P(X > t) dt
0

Now assume E[|X|] < . Show that,

P(X > t) dt

E[X] =

P(X < t) dt

Solution:
We have
Z
EX

xdF (x)
Z Z x
=
dydF (x)
Z0 Z0
=
dF (x)dy
0

P(X > y)dy.

0

For general X, let X = X + X where X + is the positive part of X and X is the negative part of X, i.e.,
X = max(X, 0).

X + = max(X, 0),
Then
EX

=
=

EX + EX
Z
Z
P(X + > t)dt
0

P(X > t)dt

P(X < t)dt.

P(X > t)dt

0
0

CME308: Assignment 1 Solutions April 13, 2016

Problem 4: Independence.
Let (X1 , X2 ) be multivariate normal random variables with marginal distribution X1 N (1 , 12 ), X2
N (2 , 22 ). Also assume that X1 and X2 are uncorrelated, i.e.,
E[(X1 1 )(X2 2 )] = 0
Show that X1 and X2 are independent, i.e.,
P(X1 x1 , X2 x2 ) = P(X1 x1 ) P(X2 x2 )
Now let Y U ([0, 1] be uniformly distributed in the interval [0, 1]. Consider the random variables,
Y1 = sin(2Y )
Y2 = cos(2Y )
Show that Y1 and Y2 are uncorrelated. Are they independent?
Solution:
The density for (X1 , X2 ) is
f (x1 , x2 ) =



1
(x1 1 )2
(x2 2 )2
exp

.
21 2
21
22

It can be written as a function of x1 times a function of x2 . Hence X1 and X2 are independent.

For the second part, we have
Z

EY1 =

sin(2y)dy = 0,

EY2 =

cos(2y)dy = 0

Moreover
Z
EY1 Y2 =

Z
sin(2y) cos(2y)dy =

Hence Y1 and Y2 are uncorrelated.

However since Y12 + Y22 = 1, they cannot be independent.

1
sin(4y)dy = 0.
2

CME308: Assignment 1 Solutions April 13, 2016

Problem 5: Box-Muller transform
Let U1 , U2 be two independent random variables uniformly distributed in [0, 1]. Define
p
p
X1 = cos(2U1 ) 2 log U2 ,
X2 = sin(2U1 ) 2 log U2 .
Show that X1 , X2 are two independent random variables with Gaussian distribution of mean zero and
variance one.
Solution:
Let g1 (u1 , u2 ), g2 (u1 , u2 ) be
p
g1 (u1 , u2 ) = cos(2u1 ) 2 log u2 ,

p
g2 (u1 , u2 ) = sin(2u1 ) 2 log u2 .

Then
fX1 ,X2 (x1 , x2 ) = fU1 ,U2 (u1 , u2 )| det J|
where
J=

g11
x1
g21
x1

and (u1 , u2 ) = g 1 (x1 , x2 ). It is easy to check that



1 2 1 2
1
g1 (x1 , x2 ) = exp x1 x2 ,
2
2

g11
x2
g21
x2

!
.

g21 (x1 , x2 )

 
1
x2
=
arctan
.
2
x1

Hence u1 , u2 [0, 1]. This gives fU1 ,U2 (u1 , u2 ) = 1. On the other hand, it is easy to check that


1
1 2 1 2
exp x1 x2 .
| det J| =
2
2
2
Hence



1 2 1 2
1
fX1 ,X2 (x1 , x2 ) =
exp x1 x2 .
2
2
2

This is exactly the density of a standard normal distribution in R2 .

CME308: Assignment 1 Solutions April 13, 2016

Problem 6: Coupon collection problem.
Consider i.i.d. random variables U1 , U2 , . . ., each distributed uniformly on {1, . . . , n}. Let |{U1 , . . . , U` }| be
the number of distinct elements among the first ` elements. Define
Tn = inf{l : |{U1 , . . . , Ul }| = n}
be the first time to collect the whole set.
1. Calculate ETn .
2. Prove that Var(Tn ) < Cn2 for some constant C > 0.
3. Prove that

Tn
1
n log n

in L2 .
Solution:

1. Define 1 , . . . , n such that i is the number

of additional drawings to get the i-th new coupon. Then
Pn
1 , . . . , n are independent and Tn = i=1 i .
Each i is a geometrical distributed random variable with success rate (n + 1 i)/n. Hence
ETn =

n
X

Ei =

i=1

n
X
i=1

X1
n
=n
.
n+1i
k
k=1

2. Since all the i s are independent, we have

Var(Tn ) =

n
X
k=1

Var(k )

n
X
k=1

X 1
1
2
=
n
Cn2
(n + 1 k)2
k2
k=1

for some constant C.

3. We have ETn = n log n + n where n is bounded for all n. Hence


(Tn n log n)2
E(Tn ETn + n )2
E
=
2
n2 log n
n2 log2 n
Var(Tn ) + n2
=
n2 log2 n
C

0.
log2 n

CME308: Assignment 1 Solutions April 13, 2016

Problem 7: Occupancy problem.
Suppose we distribute at random r distinct balls among n distinct boxes, where each of the possible nr
assignments are equally likely. Denote Nn to be the number of empty boxes.
1. Calculate ENn .
2. Calculate Var(Nn ).
3. Prove that, as n, r but r/n (0, ), then Nn /n e in L2 .
Solution:

1. Define Xi to be the indicator function of whether the i-th box is empty. That is,

1 if the i-th box is empty
Xi =
0 otherwise
Then Nn =

Pn

i=1

Xi . Moreover EXi = (1 1/n)r . Hence

r

1
.
ENn = n 1
n

2. We have
Var(Nn )

= ENn2 (ENn )2
X
2
n
n
X
X
EXi
=
EXi +
EXi Xj
i=1

1
= n 1
n

i=1

i6=j

r

2
+n(n 1) 1
n

r
n

1
1
n

2r

3. Now

2
Nn
E
e
n

=
=

1
2
ENn2 e ENn + e2
2
n
n

r

r

r
1
1
n1
2
1
1
+
1
2e 1
+e2
n
n
n
n
n

e2 2e2 + e2

0.

CME308: Assignment 1 Solutions April 13, 2016

Problem 8: Convergence of densities
Suppose that fn (x) and g(x) are probability density functions such that for all x real, fn (x) g(x) as
n . Show that
Z
|fn (x) g(x)|dx 0 , as n
Solution:
We write
|fn g| = (fn g) + 2(g fn )I{g>fn }
Taking expectations
Z
Z
Z
Z
|fn g|dx = (fn g)dx + 2(g fn )I{g>fn } dx = 2(g fn )I{g>fn } dx
since fn and g are densities and integrate to one. We now note that 2(g fn )I{g>fn } 2g and that g is
integrable. Since 2(g fn )I{g>fn } 0 as n poitwise, the dominated convergence theorem gives the
result
Z
lim
|fn g|dx = 0
n

10

CME308: Assignment 1 Solutions April 13, 2016

Problem 9: Cauchy distribution
A cauchy distribution has a probability distribution function:
(x) =

1
f or x R
(1 + x2 )

1. What is a moment generating function of a Cauchy random variable X?

Solution:
It does not exist.
2. What is a characteristic function of a Cauchy random variable X?
Solution:
X (t)

=
=

E(eitX )
Z
(x)eitx dx

eitx
dx
(1 + x2 )

This integration can be calculated using complex integral techniques. Let us assume t > 0 without
1
loss of generality. The function 1+z
2 has singularities z = i and z = i. When R > 1, the singular
point z = i lies in the interior of the semicircular region bounded by the segment z = x(R x R)
of the real axis and the upper half CR of the circle |z| = R from z = R to z = R. Integrating
counter-clockwize around the boundary of this semicircular region C, we see that
Z R
Z
Z
eitz
eitz
eitx
dx
+
dz
=
dz
2
2
2
CR (1 + z )
C (1 + z )
R (1 + x )
=
=

eitz
z=i (z + i)(z i)

eitz 
2i

(z + i) 

2iRes

z=i

et

R
eitz
Also, it can be shown that the integral CR (1+z
2 ) dz tends to 0 as R . To do this, we observe that
when |z| = R, |z 2 + 1| R2 1, and |eitz | 1. So if z is any point on CR ,


 eitz 
1



 MR , where MR = 2
 (1 + z 2 ) 
R 1
and this means that
Z



eitz


dz

 MR R 0 as R
 CR (1 + z 2 ) 
where R is the length of the semi-circle CR . Thus, it now follows that
Z R
eitx
lim
dx = et
R R (1 + x2 )
Similarly, we can do with t < 0, and it is easy to check that the characteristic function is
Z
eitx
1
X (t) =
dx
(1 + x2 )
=

e|t| .
11

CME308: Assignment 1 Solutions April 13, 2016

3. Show that if X1 , , Xn are independent Cauchy random variables, then
S = (X1 + + Xn )/n
also has a Cauchy distribution.
Solution:
From the previous derivation, it is easy to check that the characteristic function of X/n is e|t|/n .
X1 (t)/nXn (t)/n = e|t|/n e|t|/n = e|t|
which is a characteristic function of a Cauchy random variable.
(**Another method) Consider a sum of Cauchy random variable. Noting
X1 (t)Xn (t) = en|t| ,
The probability density of X1 + + Xn can be found the inverse transform:
Z
1
X1 ++Xn (x) =
eitx en|t| dt
2
0
 #
"


1
1
1
(nix)t 
(nix)t 

e
+
e
=




2
n ix
n ix

0
!
1
1
1
=

2
n ix n ix
n
=
.
(n2 + x2 )
The probability density of linear transformed variable will give the same result. (Y (y) =
where y = ax)

y
1
a X ( a ),

4. Suppose Xn is a sequence of independent Cauchy random variables. Show that there is no number
such that we have convergence in probability
Sn = (X1 + + Xn )/n
Solution:
The density function of Sn does not change at all, as n increases.
5. Why is this not a contradiction to weak law of large numbers?
Solution:
E(|X1 |) does not exist.

12

CME308: Assignment 1 Solutions April 13, 2016

Problem 10: Ideal gas in a box
Suppose there are 2N non-interacting particles in a box (cube). (i) What is the probability Pm that N + m
of them will be in the right half? (ii) Use Stirlings formula
 n n
 n n
2n
n!
,
e
e
to approximate Pm when N is large
Solution:
There are 22N equally likely ways the distinct particles could sit in the two sides of the box. Of these,


2N
(2N )!
=
N +m
(N + m)!(N m)!
have m extra particles in the right half. Thus,


2N
(2N )!
2N
Pm = 2
= 22N
(N + m)!(N m)!
N +m
Using the Stirlings formula
n!

 n n
e

2n

 n n
e

We can show that

Pm

 2N 2N  N + m N  N m N m
/
22N
e
e
e

2 N 
m
m m 
m m
=
1 2
1+
1
N
N
N
2
2
(em /N )N (em/N )m (em/N )m
2

em
Using the fact the

/N

(m  N )

Pm = 1, we find
r
Pm

 m2 
1
exp
N
N

p
It says that the number fluctuations are distributed in a Gaussian distribution with = N/2.
Note that

if we have Avogadros number of particles N 1024 , then the fractional fluctuations N

= 1/ 2N 1012 .
So almost exactly half of the particles are on the right half of the box.

13

CME308: Assignment 1 Solutions April 13, 2016

Problem 11: 3-D Isotropic scattering
A gas molecule moves an equal distance l between collisions which occur with equal probability in any direction (isotropically). After a total N displacements (collisions), what is the mean square displacement R2
of the molecule from its starting point?

Solution:

2
2
The total distance vector R =
r1 + +
r
N . Noting that R = R R , and rm rn = l cos mn = 0, it is
2
2
easy to see that R = N l .

14

CME308: Assignment 1 Solutions April 13, 2016

Problem 12: Correlation coefficient
Let the random variables X and Y have the joint probability density function
f (x, y)

= x + y, 0 < x < 1, 0 < y < 1

=

0, elsewhere

Find the correlation coefficient of X and Y . Interpret the result graphically.

Solution:
Z
E(X)

=
1

E(XY )
Cov(X, Y )

y 2 (x + y)dxdy =

5
12

y 2 (x + y)dxdy =

1
3

0
1

=
0

5
12

=
Z

x2 (x + y)dxdy =

E(Y 2 )

7
12

=
Z

y(x + y)dxdy =
0

E(X 2 )

7
12

=
Z

x(x + y)dxdy =
0

Z
E(Y )

= E(XY ) E(X)E(Y ) =

11
144
11
V ar(Y ) = E(Y 2 ) E(Y )2 =
144

V ar(X)

= E(X 2 ) E(X)2 =

Thus,

Cov(X, Y )
p
V ar(X)V ar(Y )
1
=
11
=

15

1
144

CME308: Assignment 1 Solutions April 13, 2016

Problem 13: Empirical variance. 298/308 Let X1 , X2 , . . . , Xn be independent identically distributed
n =
random variables with mean and variance 2 . Assume they also have finite fourth moment. Let X
(X1 + X2 + + Xn )/n be the empirical mean and
n

n2 =

1 X
n )2
(Xj X
n 1 j=1

be the empirical variance. Show (a) that E(n2 ) = 2 , and (b) n2 2 in probability as n .
Solution:
We may assume = 0. We then see that
n

n2 =

1 X 2
n
n 2
(X 2 ) +
2
X
n 1 j=1 j
n1
n1 n

The WLLN applies to the sum on the right and it goes to zero in probability as n . The forth moment
n 0
condition is used here (but could be weakened). The last term also goes to zero in probability since X
in probability. This gives the result.

16

CME308: Assignment 1 Solutions April 13, 2016

(n)

(n)

(n)

Problem 14: Poisson approximation. 298/308 Let X1 , X2 , . . . , Xn be independent Bernoulli ran(n)

(n)
dom variables with P (Xj = 1) = pn and P (Xj = 0) = 1pn . That is, for each n they are independent and
(n)

(n)

(n)

identically distributed but the probability of success pn depends on n. Let Zn = X1 + X2 + + Xn .

Suppose that pn = /n with a positive number. Show that Zn converges in distribution to a Poisson
random variable Z, where
k
P (Z = k) = e , k = 0, 1, 2, . . .
k!
Solution:
We use characteristic functions
E{e

(n)

i(X1

(n)

+X2

(n)
++Xn
)

}=

n
Y

(n)

E{e

i(Xj

j=1


n
i
(ei 1)
}= 1+
e(e 1)
n

We now note that the limit on right is the characteristic function a Poisson random variable Z with parameter
. Since is any real number this implies convergence in distribution of Zn to Z.

17