Question 10.5.P.1: Consider a Markov chain on {1, 2, 3, 4} with transition matr...
Consider a Markov chain on {1, 2, 3, 4} with transition matrix
P=\left[\begin{array}{cccc}1 & 1 / 2 & 0 & 0 \\0 & 1 / 6 & 1 / 2 & 0 \\0 & 1 / 3 & 1 / 6 & 0 \\0 & 0 & 1 / 3 & 1\end{array}\right].
a. If the Markov chain starts at state 2, find the expected number of steps before the chain is absorbed.
b. If the Markov chain starts at state 2, find the probability that the chain is absorbed at state 1.
a. Since states 1 and 4 are absorbing states, reordering the states as {1,4, 2, 3} produces the canonical form
So
The expected number of steps needed when starting at state 2 before the chain is absorbed is the sum of the entries in the column of M corresponding to state 2, which is
\frac{30}{19}+\frac{12}{19}=\frac{42}{19} .
b. Using the canonical form of the transition matrix, we see that
The probability that the chain is absorbed at state 1 given that the Markov chain starts at state 2 is the entry in A whose row corresponds to state 1 and whose column corresponds to state 2; this entry is 15/19.