Question 11.2.27: Consider an absorbing Markov chain with state space S. Let f......

Consider an absorbing Markov chain with state space S. Let f be a function defined on S with the property that

f(i)= \sum\limits_{j\epsilon S}{p_{ij}f(j)},

or in vector form

f = Pf .

Then f is called a harmonic function for P. If you imagine a game in which your fortune is f(i) when you are in state i, then the harmonic condition means that the game is fair in the sense that your expected fortune after one step is the same as it was before the step.

(a) Show that for f harmonic

f = Pⁿf

for all n.

(b) Show, using (a), that for f harmonic

f = P^∞f ,

where

P^∞ = \lim \limits_{n \longrightarrow  \infty } P^n =\left( \begin{array}{c|c}   0    & \mathrm{~B} \\\hline    0   & \mathrm{I}\end{array}\right)

(c) Using (b), prove that when you start in a transient state i your expected final fortune

\sum\limits_{k}{b_{ik}f(k)}

is equal to your starting fortune f(i). In other words, a fair game on a finite state space remains fair to the end. (Fair games in general are called martingales. Fair games on infinite state spaces need not remain fair with an unlimited number of plays allowed. For example, consider the game of Heads or Tails (see Example 1.4). Let Peter start with 1 penny and play until he has 2. Then Peter will be sure to end up 1 penny ahead.)