ISYE 3232C Spring

Fall
Fall 2015 Quiz 75 - Solutions
1. (8) You attended this quiz.
2 A system can be in state 0,1 or 2. Let Xn be the state of the system at
2.
1 (6)
time n. We know P(Xn+1 = 1|Xn = 0, Xn1 = 2) = 0.4 and P(Xn+1 = 1|Xn =
0, Xn1 = 1) = 0.3. Is {Xn , n 0} a Markov chain? Why?

No. It is not a Markov chain, as it does not satisfy Markov Property which
is shown below:
P(Xn+1 = 1 | Xn = 0, Xn1 = kn1 , Xn2 = kn2 , ...) = P(Xn+1 = 1 | Xn = 0)

Markov property tells that the next step n + 1 should only depend on the
current state n. However, in the question, we are given that:
P(Xn+1 = 1 | Xn = 0, Xn1 = 2) , P(Xn+1 = 1 | Xn = 0, Xn1 = 1)
As the next step also depends on Xn1 , it violates the Markov property.
2.(5)
3. Suppose

that
1
P50 = 2


0.5902
0.5902

1
P= 2


0.75
0.36

0.4098
0.4098
0.25
0.64

(a) (2)
1 What is P(Xn+1 = 2|Xn = 1)?
P(Xn+1 = 2 | Xn = 1) = P12 = 0.25

(from the transition matrix P )

50
(b) (2)
1 What is (P1,2 ) ?

(P12 )50 = (0.25)50

50 .
Note that (P12 )50 , P12

(c) (2)
1 What is P(X50 = 1|X0 = 2)?
50
P(X50 = 1 | X0 = 2) = P21
= 0.5902

(from the transition matrix P50 )

(d) (4)
2 What is P(X51 = 2|X0 = 2)?

P(X51 = 2|X0 = 2) = P(X51 = 2|X50 = 1)P(X50 = 1|X0 = 2)

+ P(X51 = 2|X50 = 2)P(X50 = 2|X0 = 2)
50
50
= P12 P21
+ P22 P22

= 0.25 0.5902 + 0.64 0.4098 = 0.4098

As an alternative solution, observe that P50 is a stationary distribution
such that P51 = P P50 = P50 . Thus,
51
50
P(X51 = 2|X0 = 2) = P22
= P22
= 0.4098

4.
3 A store sells a particular product which is classified as durable goods,
3 (6)
i.e., left-over items can be sold next period. The weekly demands for the
product are iid random variables with the following distribution:
d
P(D = d)

0
0.2

1
0.3

2
0.1

3
0.4

The store is closed during the weekends. Unsold items at the end of a week
can be sold again the following week. Each Friday evening if the number of
remaining items is less than or equal to 2, the store orders enough items to
bring the total number of items up to 4 on Monday morning. Otherwise, do
not order. (The ordered items reach the store before the store opens on the
Monday morning.) Assume that any demand is lost when the product is out
of stock. Let Xn be the number of items in stock at the BEGINNING of the
nthe week. Xn is DTMC. What is the state space and transition probability
matrix of Xn ?
Observe that the state space of Xn is
S = {3, 4},
Remember that we order up to 4 only when the inventory each week is < 3. So at
the beginning of a day the minimum number of items is equal to 3.
Next, we find the transition matrix.

Xn
(# of item at the
beginning of week n)
0

Dn
(demand)
0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3
0
1
2
3

inventory
# of item at the
end of week n
0
0
0
0
1
0
0
0
2
1
0
0
3
2
1
0
4
3
2
1

Xn+1
(at the beginning
of week n + 1)
4
4
4
4
4
4
4
4
4
4
4
4
3
4
4
4
4
3
4
4

Note that
P(Xn+1 = 3|Xn = 3) = P(Dn = 0) = 0.2,
P(Xn+1 = 4|Xn = 3) = P(Dn = 1) + P(Dn = 2) + P(Dn = 3) = 0.8,
P(Xn+1 = 3|Xn = 4) = P(Dn = 1) = 0.3,
P(Xn+1 = 4|Xn = 4) = P(Dn = 0) + P(Dn = 2) + P(Dn = 3) = 0.7.
Therefore, the transition matrix as follows:

P=

0.2 0.8
0.3 0.7


.