Home Subscribe

2. Makov Process

The future evolution of the process depends only on the instant observation of the system and not on the past.

2.1. A 2-state Markov chain


Example 1

Assume you have 2 shirts, white and blue. Their state space is \(S = {W,B}\).

- The probability of you wearing a white shirt and continue wearing the white shirt is 0.7.
- The probability of changing the white shirt to a blue shirt is 0.3.
- Once you’re wearing a blue shirt, the probability of you continue wearing the blue shirt is 0.4.
- The probability of changing the blue shirt to the white shirt is 0.6.

\[\text{2 Shirts} = \begin{cases} P(W|W) &= 0.7\\ P(B|W) &= 0.3\\ P(W|B) &= 0.6\\ P(B|B) &= 0.4 \end{cases}\]
  • Suppose you repeat this for several days. Let’s say today is Monday and you’ve already decided that you’ll wear a white shirt. \(S = {1,0}\)

  • What is the probability of you wearing a white or blue shirt on Tuesday through Friday?

(i) At the end of Monday: Mon \(= S \times \text{trans_matrix}\)
(ii) At the end of Tuesday: Tue \(= Mon \times \text{trans_matrix}\)
(iii) At the end of Wednesday: Wed \(= Tue \times \text{trans_matrix}\)

1nodes = ['White', 'Blue']
2trans_matrix = np.array([[0.7,0.3],[0.6,0.4]])
3pd.DataFrame(trans_matrix, columns=nodes, index=nodes)
1S = np.array([[1,0]])
2Mon = np.matmul(S,trans_matrix)
1Tue = np.matmul(Mon,trans_matrix)
  • If you are not decided, you can set \(S = [0.5, 0.5\)] or introduce randomness.

2.2. A 3-state Markov chain


Example 2

There are 3 states:
- Home
- Bar
- Back Home

- You only have one bar you go to if you want to go out.
- From state Home you can only go out. The first assumption is that you have to go out to be in a bar.
- From the Bar, you can stay in the bar or go back home.
- For each time step, you have a 50% probability of going back home and a 50% probability of staying.
- When you are Back Home, the only thing that you could do is stay home. It means you won’t go out again.

 1import numpy as np
 2import pandas as pd
 4nodes = ['Home', 'Bar', 'Back Home']
 5trans_matrix = np.array([[0,1,0],[0,0.5,0.5],[0,0,1]])
 6pd.DataFrame(trans_matrix, columns=nodes, index=nodes)
 8# You're in the bar. Where can you go next?
 9# All nodes: np.arange(0,len(nodes),1)
10# Bar transitions: trans_matrix[1]
11print(np.random.choice(np.arange(0,len(nodes),1), p=trans_matrix[1]))

Add Comment

* Required information
Drag & drop images (max 3)
Enter the word shark backwards.


No comments yet. Be the first!