1
00:00:00,040 --> 00:00:02,460
The following content is
provided under a Creative
2
00:00:02,460 --> 00:00:03,870
Commons license.
3
00:00:03,870 --> 00:00:06,910
Your support will help MIT
OpenCourseWare continue to
4
00:00:06,910 --> 00:00:10,560
offer high quality educational
resources for free.
5
00:00:10,560 --> 00:00:13,460
To make a donation or view
additional materials from
6
00:00:13,460 --> 00:00:17,390
hundreds of MIT courses, visit
MIT OpenCourseWare at
7
00:00:17,390 --> 00:00:18,640
ocw.mit.edu.
8
00:00:23,440 --> 00:00:25,930
JOHN TSITSIKLIS: So what we're
going to do is to review what
9
00:00:25,930 --> 00:00:29,390
we have discussed last time.
10
00:00:29,390 --> 00:00:32,380
Then we're going to talk about
the classic application of
11
00:00:32,380 --> 00:00:35,950
Markov chains to analyze
how do you
12
00:00:35,950 --> 00:00:39,130
dimension a phone system.
13
00:00:39,130 --> 00:00:42,400
And finally, there will be
two new things today.
14
00:00:42,400 --> 00:00:44,940
We will see how we can calculate
certain interesting
15
00:00:44,940 --> 00:00:48,950
quantities that have to
do with Markov chains.
16
00:00:48,950 --> 00:00:50,200
So let us start.
17
00:00:52,860 --> 00:00:56,680
We've got our Markov chain and
let's make the assumption that
18
00:00:56,680 --> 00:00:58,800
our chain is kind of nice.
19
00:00:58,800 --> 00:01:01,550
And by nice we mean that
we've got maybe
20
00:01:01,550 --> 00:01:02,800
some transient states.
21
00:01:06,380 --> 00:01:11,580
And then we've got a single
recurrent class
22
00:01:11,580 --> 00:01:15,450
of recurrent states.
23
00:01:15,450 --> 00:01:18,600
So this is a single recurrent
class in the sense that from
24
00:01:18,600 --> 00:01:22,080
any state in that class you can
get to any other state.
25
00:01:22,080 --> 00:01:24,790
So once you're in here you're
going to circulate and keep
26
00:01:24,790 --> 00:01:26,650
visiting all of those states.
27
00:01:26,650 --> 00:01:28,340
Those states appear transient.
28
00:01:28,340 --> 00:01:32,020
The trajectory may move around
here, but eventually one of
29
00:01:32,020 --> 00:01:34,320
these transitions will happen
and you're going to
30
00:01:34,320 --> 00:01:36,040
end up in this lump.
31
00:01:38,640 --> 00:01:42,680
Let's make the assumption that
the single recurrent class is
32
00:01:42,680 --> 00:01:43,930
not periodic.
33
00:01:48,170 --> 00:01:51,740
These are the nicest kind
of Markov chains.
34
00:01:51,740 --> 00:01:54,500
And they're nicest because
they have the following
35
00:01:54,500 --> 00:01:58,900
property, the probability that
you find yourself at some
36
00:01:58,900 --> 00:02:03,100
particular state j at
the time n when that
37
00:02:03,100 --> 00:02:04,800
time is very large.
38
00:02:04,800 --> 00:02:08,130
That probability settles to a
steady state value that we
39
00:02:08,130 --> 00:02:10,139
denote by pi sub j.
40
00:02:10,139 --> 00:02:12,630
And there are two parts
in the statement.
41
00:02:12,630 --> 00:02:15,470
One part is that this
limit exists.
42
00:02:15,470 --> 00:02:19,010
So the probability of state j
settles to something, and
43
00:02:19,010 --> 00:02:23,080
furthermore that probability
is not affected by i.
44
00:02:23,080 --> 00:02:25,550
It doesn't matter where you
started, no matter where you
45
00:02:25,550 --> 00:02:28,310
started, the probability of
state j is going to be the
46
00:02:28,310 --> 00:02:30,720
same in the long run.
47
00:02:30,720 --> 00:02:34,990
Maybe a clearer notation
could be of this form.
48
00:02:34,990 --> 00:02:37,650
The probability of being at
state j given the initial
49
00:02:37,650 --> 00:02:42,570
state being i is equal to
pi(j) in the limit.
50
00:02:42,570 --> 00:02:47,380
Now, if I don't tell you where
you started and you look at
51
00:02:47,380 --> 00:02:52,520
the unconditional probability
of being at state i, you can
52
00:02:52,520 --> 00:02:56,260
average over the initial
states, use the total
53
00:02:56,260 --> 00:03:01,560
expectation theorem and you're
going to get the same answer
54
00:03:01,560 --> 00:03:05,330
pi(j) in the limit.
55
00:03:05,330 --> 00:03:09,400
So this tells you that to the
conditional probability given
56
00:03:09,400 --> 00:03:11,820
the initial state in the limit
is the same as the
57
00:03:11,820 --> 00:03:13,780
unconditional probability.
58
00:03:13,780 --> 00:03:17,200
And that's a situation that we
recognize as being one where
59
00:03:17,200 --> 00:03:18,890
we have independence.
60
00:03:18,890 --> 00:03:24,970
So what this result tells
us is that Xn and Xi are
61
00:03:24,970 --> 00:03:27,410
approximately independent.
62
00:03:27,410 --> 00:03:32,690
They become independent in the
limit as n goes to infinity.
63
00:03:32,690 --> 00:03:35,130
So that's what the steady
state theorem tells us.
64
00:03:35,130 --> 00:03:38,860
The initial conditions don't
matter, so your state at some
65
00:03:38,860 --> 00:03:43,440
large time n has nothing to do,
is not affected by what
66
00:03:43,440 --> 00:03:45,350
your initial state was.
67
00:03:45,350 --> 00:03:47,790
Knowing the initial state
doesn't tell you anything
68
00:03:47,790 --> 00:03:50,830
about your state at time
n, therefore the
69
00:03:50,830 --> 00:03:52,890
states at the times--
70
00:03:52,890 --> 00:03:56,580
sorry that should be a 1,
or it should be a 0 --
71
00:03:56,580 --> 00:04:02,350
so the state is not affected by
where the process started.
72
00:04:02,350 --> 00:04:05,450
So if the Markov chain is to
operate for a long time and
73
00:04:05,450 --> 00:04:08,520
we're interested in the question
where is the state,
74
00:04:08,520 --> 00:04:11,470
then your answer would be, I
don't know, it's random.
75
00:04:11,470 --> 00:04:14,020
But it's going to be a
particular j with this
76
00:04:14,020 --> 00:04:15,500
particular probability.
77
00:04:15,500 --> 00:04:18,550
So the steady state
probabilities are interesting
78
00:04:18,550 --> 00:04:21,060
to us and that raises
the question of how
79
00:04:21,060 --> 00:04:22,470
do we compute them.
80
00:04:22,470 --> 00:04:25,460
The way we compute them is by
solving a linear system of
81
00:04:25,460 --> 00:04:28,580
equations, which are called
the balance equations,
82
00:04:28,580 --> 00:04:31,510
together with an extra equation,
the normalization
83
00:04:31,510 --> 00:04:33,690
equation that has to be
satisfied by probability,
84
00:04:33,690 --> 00:04:37,940
because probabilities must
always add up to 1.
85
00:04:37,940 --> 00:04:40,660
We talked about the
interpretation of this
86
00:04:40,660 --> 00:04:42,280
equation last time.
87
00:04:42,280 --> 00:04:45,500
It's basically a conservation
of probability
88
00:04:45,500 --> 00:04:47,400
flow in some sense.
89
00:04:47,400 --> 00:04:50,140
What comes in must get out.
90
00:04:50,140 --> 00:04:53,190
The probability of finding
yourself at state j at a
91
00:04:53,190 --> 00:04:58,150
particular time is the total
probability of the last
92
00:04:58,150 --> 00:05:01,390
transition taking
me into state j.
93
00:05:01,390 --> 00:05:04,860
The last transition takes me
into state j in various ways.
94
00:05:04,860 --> 00:05:07,890
It could be that the previous
time I was at the particular
95
00:05:07,890 --> 00:05:12,550
state, j and i made a transition
from k into j.
96
00:05:12,550 --> 00:05:15,610
So this number here, we
interpret as the frequency
97
00:05:15,610 --> 00:05:18,190
with which transitions
of these particular
98
00:05:18,190 --> 00:05:21,900
type k to j, occur.
99
00:05:21,900 --> 00:05:25,840
And then by adding over all k's
we consider transitions of
100
00:05:25,840 --> 00:05:29,770
all types that lead
us inside state j.
101
00:05:29,770 --> 00:05:34,240
So the probability of being at
the j is the sum total of the
102
00:05:34,240 --> 00:05:38,690
probabilities of
getting into j.
103
00:05:38,690 --> 00:05:42,800
What if we had multiple
recurrent classes?
104
00:05:42,800 --> 00:05:48,645
So if we take this picture
and change it to this.
105
00:05:53,420 --> 00:05:55,920
So here we got a secondary
recurrent class.
106
00:05:55,920 --> 00:05:57,750
If you're here, you
cannot get there.
107
00:05:57,750 --> 00:06:00,050
If you are here, you
cannot get there.
108
00:06:00,050 --> 00:06:02,370
What happens in the long run?
109
00:06:02,370 --> 00:06:05,990
Well, in the long run, if you
start from here you're going
110
00:06:05,990 --> 00:06:09,360
to make a transition eventually,
either of this
111
00:06:09,360 --> 00:06:12,540
type and you would end up
here, or you will make a
112
00:06:12,540 --> 00:06:16,210
transitional of that type and
you will end up there.
113
00:06:16,210 --> 00:06:21,740
If you end up here, the long
term statistics of your chain,
114
00:06:21,740 --> 00:06:24,860
that is, the probabilities of
the different states, will be
115
00:06:24,860 --> 00:06:27,080
the steady state probabilities
of this
116
00:06:27,080 --> 00:06:30,230
chain regarded in isolation.
117
00:06:30,230 --> 00:06:34,440
So you go ahead and you solve
this system of equations just
118
00:06:34,440 --> 00:06:37,270
for this chain, and these will
be your steady state
119
00:06:37,270 --> 00:06:38,560
probabilities.
120
00:06:38,560 --> 00:06:41,660
If you happened to
get in here.
121
00:06:41,660 --> 00:06:45,650
If, on the other hand, it
happens that you went there,
122
00:06:45,650 --> 00:06:49,710
given that event, then what
happens in the long run has to
123
00:06:49,710 --> 00:06:53,390
do with just this chain
running by itself.
124
00:06:53,390 --> 00:06:55,610
So you find the steady
state probabilities
125
00:06:55,610 --> 00:06:57,340
inside that sub chain.
126
00:06:57,340 --> 00:06:59,860
So you solve the linear system,
the steady state
127
00:06:59,860 --> 00:07:03,520
equations, for this chain
separately and for that chain
128
00:07:03,520 --> 00:07:04,590
separately.
129
00:07:04,590 --> 00:07:08,160
If you happen to start inside
here then the steady state
130
00:07:08,160 --> 00:07:12,650
probabilities for this sub
chain are going to apply.
131
00:07:12,650 --> 00:07:16,420
Now of course this raises the
question, if I start here, how
132
00:07:16,420 --> 00:07:19,500
do I know whether I'm going
to get here or there?
133
00:07:19,500 --> 00:07:21,360
Well, you don't know,
it's random.
134
00:07:21,360 --> 00:07:23,620
It may turn out that you get to
here, it may turn out that
135
00:07:23,620 --> 00:07:24,760
you get there.
136
00:07:24,760 --> 00:07:27,790
So we will be interested in
calculating the probabilities
137
00:07:27,790 --> 00:07:31,240
that eventually you end up here
versus the probability
138
00:07:31,240 --> 00:07:33,570
that eventually you
end up there.
139
00:07:33,570 --> 00:07:36,590
This is something that we're
going to do towards the end of
140
00:07:36,590 --> 00:07:37,840
today's lecture.
141
00:07:43,730 --> 00:07:48,420
So, as a warm up, just to see
how we interpret those steady
142
00:07:48,420 --> 00:07:52,710
state probabilities, let us look
at our familiar example.
143
00:07:52,710 --> 00:07:54,680
This is a 2-state
Markov chain.
144
00:07:54,680 --> 00:07:57,860
Last time we did write down the
balance equations for this
145
00:07:57,860 --> 00:08:01,760
chain and we found the steady
state probabilities to be 2/7
146
00:08:01,760 --> 00:08:04,250
and 5/7 respectively.
147
00:08:04,250 --> 00:08:07,040
So let us try to calculate
some quantities.
148
00:08:07,040 --> 00:08:10,520
Suppose that you start at
state 1, and you want to
149
00:08:10,520 --> 00:08:12,940
calculate to this particular
probability.
150
00:08:12,940 --> 00:08:15,620
So since we're assuming that
we're starting at state 1,
151
00:08:15,620 --> 00:08:19,010
essentially here we are
conditioning on the initial
152
00:08:19,010 --> 00:08:21,790
state being equal to 1.
153
00:08:21,790 --> 00:08:23,920
Now the conditional probability
of two things
154
00:08:23,920 --> 00:08:30,370
happening is the probability
that the first thing happens.
155
00:08:30,370 --> 00:08:34,090
But we're living in the world
where we said that the initial
156
00:08:34,090 --> 00:08:35,720
state was 1.
157
00:08:35,720 --> 00:08:40,000
And then given that this thing
happened, the probability that
158
00:08:40,000 --> 00:08:43,530
the second thing happens.
159
00:08:43,530 --> 00:08:46,200
But again, we're talking about
conditional probabilities
160
00:08:46,200 --> 00:08:50,150
given that the initial
state was 1.
161
00:08:50,150 --> 00:08:53,080
So what is this quantity?
162
00:08:53,080 --> 00:08:57,040
This one is the transition
probability from state 1 to
163
00:08:57,040 --> 00:09:00,800
state 1, so it's P11.
164
00:09:00,800 --> 00:09:04,730
How about the second
probability?
165
00:09:04,730 --> 00:09:08,130
So given that you started at 1
and the next time you were at
166
00:09:08,130 --> 00:09:11,980
1, what's the probability that
at the time 100 you are at 1?
167
00:09:11,980 --> 00:09:15,160
Now because of the Markov
property, if I tell you that
168
00:09:15,160 --> 00:09:17,520
at this time you are
at 1, it doesn't
169
00:09:17,520 --> 00:09:19,190
matter how you get there.
170
00:09:19,190 --> 00:09:22,560
So this part of the conditioning
doesn't matter.
171
00:09:22,560 --> 00:09:29,650
And what we have is the 99 step
transition probability
172
00:09:29,650 --> 00:09:34,820
from state 1 to state 1.
173
00:09:34,820 --> 00:09:38,910
So the probability that you
get to 1 and then 99 steps
174
00:09:38,910 --> 00:09:42,340
later you find yourself again at
one is the probability that
175
00:09:42,340 --> 00:09:45,350
the first transition takes you
to 1 times the probability
176
00:09:45,350 --> 00:09:51,810
that over the next 99
transitions starting from 1,
177
00:09:51,810 --> 00:09:55,930
after 99 steps you end
up again at state 1.
178
00:09:55,930 --> 00:10:00,720
Now, 99 is possibly a big
number, and so we approximate
179
00:10:00,720 --> 00:10:01,730
this quantity.
180
00:10:01,730 --> 00:10:05,810
We're using the steady state
probability of state 1.
181
00:10:05,810 --> 00:10:08,110
And that gives us an
approximation for this
182
00:10:08,110 --> 00:10:10,510
particular expression.
183
00:10:10,510 --> 00:10:14,790
We can do the same thing
to calculate
184
00:10:14,790 --> 00:10:16,490
something of the same kind.
185
00:10:16,490 --> 00:10:19,260
So you start at state 1.
186
00:10:19,260 --> 00:10:23,230
What's the probability that 100
steps later you are again
187
00:10:23,230 --> 00:10:24,560
at state 1?
188
00:10:24,560 --> 00:10:26,000
So that's going to be P11--
189
00:10:28,970 --> 00:10:29,890
not P --
190
00:10:29,890 --> 00:10:31,140
R11.
191
00:10:34,830 --> 00:10:38,560
The 100 step transition
probability that starting from
192
00:10:38,560 --> 00:10:43,980
1 you get to 1, and then after
you get to 1 at time 100
193
00:10:43,980 --> 00:10:46,120
what's the probability that
the next time you find
194
00:10:46,120 --> 00:10:47,870
yourself at state 2?
195
00:10:47,870 --> 00:10:50,580
This is going to be the
probability P12.
196
00:10:50,580 --> 00:10:55,320
And approximately, since 100
is a large number, this is
197
00:10:55,320 --> 00:10:57,890
approximately pi(1) times P12.
198
00:11:06,230 --> 00:11:07,070
OK.
199
00:11:07,070 --> 00:11:12,180
So that's how we can use steady
state probabilities to
200
00:11:12,180 --> 00:11:14,190
make approximations.
201
00:11:14,190 --> 00:11:19,090
Or you could, for example, if
you continue doing examples of
202
00:11:19,090 --> 00:11:23,250
this kind, you could ask for
what's the probability that X
203
00:11:23,250 --> 00:11:32,170
at time 100 is 1, and also X
at time 200 is equal to 1.
204
00:11:32,170 --> 00:11:36,200
Then this is going to be the
transition probability from 1
205
00:11:36,200 --> 00:11:43,140
to 1 in 100 steps, and then over
the next 100 steps from 1
206
00:11:43,140 --> 00:11:46,510
you get again to 1.
207
00:11:46,510 --> 00:11:48,750
And this is going to be
208
00:11:48,750 --> 00:11:51,400
approximately pi(1) times pi(1).
209
00:11:57,550 --> 00:12:01,090
So we approximate multi-step
transition probabilities by
210
00:12:01,090 --> 00:12:05,120
the steady state probabilities
when the number n that's
211
00:12:05,120 --> 00:12:07,510
involved in here is big.
212
00:12:07,510 --> 00:12:11,270
Now I said that's 99
or 100 is big.
213
00:12:11,270 --> 00:12:15,180
How do we know that it's big
enough so that the limit has
214
00:12:15,180 --> 00:12:19,560
taken effect, and that our
approximation is good?
215
00:12:19,560 --> 00:12:23,640
This has something to do with
the time scale of our Markov
216
00:12:23,640 --> 00:12:27,460
chain, and by time scale, I mean
how long does it take for
217
00:12:27,460 --> 00:12:29,740
the initial states
to be forgotten.
218
00:12:29,740 --> 00:12:35,080
How long does it take for there
to be enough randomness
219
00:12:35,080 --> 00:12:37,700
so that things sort of
mix and it doesn't
220
00:12:37,700 --> 00:12:39,330
matter where you started?
221
00:12:39,330 --> 00:12:43,720
So if you look at this chain, it
takes on the average, let's
222
00:12:43,720 --> 00:12:47,660
say 5 tries to make a transition
of this kind.
223
00:12:47,660 --> 00:12:51,970
It takes on the average 2 tries
for a transition of that
224
00:12:51,970 --> 00:12:53,990
kind to take place.
225
00:12:53,990 --> 00:12:58,840
So every 10 time steps or so
there's a little bit of
226
00:12:58,840 --> 00:12:59,660
randomness.
227
00:12:59,660 --> 00:13:03,000
Over 100 times steps there's
a lot of randomness, so you
228
00:13:03,000 --> 00:13:06,420
expect that the initial state
will have been forgotten.
229
00:13:06,420 --> 00:13:07,820
It doesn't matter.
230
00:13:07,820 --> 00:13:10,540
There's enough mixing and
randomness that happens over
231
00:13:10,540 --> 00:13:11,870
100 time steps.
232
00:13:11,870 --> 00:13:16,100
And so this approximation
is good.
233
00:13:16,100 --> 00:13:19,080
On the other hand, if the
numbers were different, the
234
00:13:19,080 --> 00:13:20,880
story would have
been different.
235
00:13:20,880 --> 00:13:26,620
Suppose that this number is
0.999 and that number is
236
00:13:26,620 --> 00:13:33,570
something like 0.998, so that
this number becomes 0.002, and
237
00:13:33,570 --> 00:13:37,910
that number becomes 0.001.
238
00:13:37,910 --> 00:13:40,620
Suppose that the numbers
were of this kind.
239
00:13:40,620 --> 00:13:44,760
How long does it take to forget
the initial state?
240
00:13:44,760 --> 00:13:48,840
If I start here, there's a
probability of 1 in 1,000 that
241
00:13:48,840 --> 00:13:50,770
next time I'm going
to be there.
242
00:13:50,770 --> 00:13:53,840
So on the average it's going
to take me about a thousand
243
00:13:53,840 --> 00:13:58,330
tries just to leave
that state.
244
00:13:58,330 --> 00:14:03,390
So, over roughly a thousand time
steps my initial state
245
00:14:03,390 --> 00:14:05,210
really does matter.
246
00:14:05,210 --> 00:14:08,550
If I tell you that you started
here, you're pretty certain
247
00:14:08,550 --> 00:14:11,320
that, let's say over the next
100 time steps, you
248
00:14:11,320 --> 00:14:12,710
will still be here.
249
00:14:12,710 --> 00:14:15,580
So the initial state
has a big effect.
250
00:14:15,580 --> 00:14:19,600
In this case we say that this
Markov chain has a much slower
251
00:14:19,600 --> 00:14:21,030
time scale.
252
00:14:21,030 --> 00:14:25,350
It takes a much longer time to
mix, it takes a much longer
253
00:14:25,350 --> 00:14:29,150
time for the initial state to
be forgotten, and this means
254
00:14:29,150 --> 00:14:32,670
that we cannot do this kind of
approximation if the number of
255
00:14:32,670 --> 00:14:34,870
steps is just 99.
256
00:14:34,870 --> 00:14:39,620
Here we might need n to be as
large as, let's say, 10,000 or
257
00:14:39,620 --> 00:14:43,320
so before we can start using
the approximation.
258
00:14:43,320 --> 00:14:46,210
So when one uses that
approximation, one needs to
259
00:14:46,210 --> 00:14:50,830
have some sense of how quickly
does the state move around and
260
00:14:50,830 --> 00:14:52,210
take that into account.
261
00:14:52,210 --> 00:14:56,100
So there's a whole sub-field
that deals with estimating or
262
00:14:56,100 --> 00:15:00,030
figuring out how quickly
different Markov chains mix,
263
00:15:00,030 --> 00:15:02,460
and that's the question of
when can you apply those
264
00:15:02,460 --> 00:15:03,710
steady state approximations.
265
00:15:06,860 --> 00:15:12,180
So now let's get a little closer
to the real world.
266
00:15:12,180 --> 00:15:15,590
We're going to talk about a
famous problem that was posed,
267
00:15:15,590 --> 00:15:19,220
started, and solved by
a Danish engineer
268
00:15:19,220 --> 00:15:21,110
by the name of Erlang.
269
00:15:21,110 --> 00:15:25,180
This is the same person whose
name is given to the Erlang
270
00:15:25,180 --> 00:15:27,080
distribution that we
saw in the context
271
00:15:27,080 --> 00:15:28,940
of the Poisson processes.
272
00:15:28,940 --> 00:15:33,950
So this was more than 100 years
ago, when phones had
273
00:15:33,950 --> 00:15:36,060
just started existing.
274
00:15:36,060 --> 00:15:42,900
And he was trying to figure out
what it would take to set
275
00:15:42,900 --> 00:15:47,260
up a phone system that how many
lines should you set up
276
00:15:47,260 --> 00:15:49,920
for a community to be
able to communicate
277
00:15:49,920 --> 00:15:53,020
to the outside world.
278
00:15:53,020 --> 00:15:54,350
So here's the story.
279
00:15:54,350 --> 00:15:57,730
You've got a village, and that
village has a certain
280
00:15:57,730 --> 00:16:03,705
population, and you want
to set up phone lines.
281
00:16:06,290 --> 00:16:09,720
So you want to set up a number
of phone lines, let's say that
282
00:16:09,720 --> 00:16:13,520
number is B, to the
outside world.
283
00:16:17,710 --> 00:16:19,690
And how do you want
to do that?
284
00:16:19,690 --> 00:16:22,000
Well, you want B to
be kind of small.
285
00:16:22,000 --> 00:16:24,130
You don't want to set
up too many wires
286
00:16:24,130 --> 00:16:25,680
because that's expensive.
287
00:16:25,680 --> 00:16:30,300
On the other hand, you want to
have enough wires so that if a
288
00:16:30,300 --> 00:16:33,000
reasonable number of people
place phone calls
289
00:16:33,000 --> 00:16:38,000
simultaneously, they will all
get a line and they will be
290
00:16:38,000 --> 00:16:39,600
able to talk.
291
00:16:39,600 --> 00:16:45,220
So if B is 10 and 12 people want
to talk at the same time,
292
00:16:45,220 --> 00:16:48,490
then 2 of these people would get
a busy signal, and that's
293
00:16:48,490 --> 00:16:50,220
not something that we like.
294
00:16:50,220 --> 00:16:53,010
We would like B to be large
enough so that there's a
295
00:16:53,010 --> 00:16:57,240
substantial probability, that
there's almost certainty that,
296
00:16:57,240 --> 00:17:00,120
under reasonable conditions,
no one is going
297
00:17:00,120 --> 00:17:02,230
to get a busy signal.
298
00:17:02,230 --> 00:17:06,720
So how do we go about modeling
a situation like this?
299
00:17:06,720 --> 00:17:09,890
Well, to set up a model you
need two pieces, one is to
300
00:17:09,890 --> 00:17:17,000
describe how do phone calls
get initiated, and once a
301
00:17:17,000 --> 00:17:21,819
phone call gets started, how
long does it take until the
302
00:17:21,819 --> 00:17:24,530
phone call is terminated?
303
00:17:24,530 --> 00:17:27,530
So we're going to make the
simplest assumptions possible.
304
00:17:27,530 --> 00:17:29,890
Let's assume that phone
calls originate
305
00:17:29,890 --> 00:17:31,920
as a Poisson process.
306
00:17:31,920 --> 00:17:34,210
That is, out of that population
people do not
307
00:17:34,210 --> 00:17:35,290
really coordinate.
308
00:17:35,290 --> 00:17:38,160
At completely random times,
different people with decide
309
00:17:38,160 --> 00:17:39,740
to pick up the phone.
310
00:17:39,740 --> 00:17:42,320
There's no dependencies between
different people,
311
00:17:42,320 --> 00:17:44,790
there's nothing special about
different times, different
312
00:17:44,790 --> 00:17:46,340
times are independent.
313
00:17:46,340 --> 00:17:50,550
So a Poisson model is a
reasonable way of modeling
314
00:17:50,550 --> 00:17:51,410
this situation.
315
00:17:51,410 --> 00:17:55,870
And it's going to be a Poisson
process with some rate lambda.
316
00:17:55,870 --> 00:17:59,870
Now, the rate lambda would be
easy to estimate in practice.
317
00:17:59,870 --> 00:18:02,760
You observe what happens in
that village just over a
318
00:18:02,760 --> 00:18:06,050
couple of days, and you figure
out what's the rate at which
319
00:18:06,050 --> 00:18:09,550
people attempt to place
phone calls.
320
00:18:09,550 --> 00:18:11,930
Now, about phone calls
themselves, we're going to
321
00:18:11,930 --> 00:18:15,120
make the assumption that the
duration of a phone call is a
322
00:18:15,120 --> 00:18:18,580
random variable that has an
exponential distribution with
323
00:18:18,580 --> 00:18:20,630
a certain parameter mu.
324
00:18:20,630 --> 00:18:24,400
So 1/mu is the mean duration
of a phone call.
325
00:18:24,400 --> 00:18:27,570
So the mean duration, , again,
is easy to estimate.
326
00:18:27,570 --> 00:18:31,240
You just observe what's
happening, see on the average
327
00:18:31,240 --> 00:18:33,680
how long these phone
calls are.
328
00:18:33,680 --> 00:18:36,590
Is the exponential assumption
a good assumption?
329
00:18:36,590 --> 00:18:39,990
Well, it's means that most phone
calls will be kind of
330
00:18:39,990 --> 00:18:43,180
short, but there's going to be a
fraction of phone calls that
331
00:18:43,180 --> 00:18:45,640
are going to be larger, and
then a very small fraction
332
00:18:45,640 --> 00:18:47,900
that are going to
be even larger.
333
00:18:47,900 --> 00:18:49,880
So it sounds plausible.
334
00:18:49,880 --> 00:18:57,960
It's not exactly realistic, that
is, phone calls that last
335
00:18:57,960 --> 00:19:02,250
short of 15 seconds are
not that common.
336
00:19:02,250 --> 00:19:04,890
So either nothing happens
or you have to say a few
337
00:19:04,890 --> 00:19:06,630
sentences and so on.
338
00:19:06,630 --> 00:19:10,200
Also, back into the days when
people used to connect to the
339
00:19:10,200 --> 00:19:15,430
internet using dial up modems,
that assumption was completely
340
00:19:15,430 --> 00:19:20,390
destroyed, because people would
dial up and then keep
341
00:19:20,390 --> 00:19:25,730
their phone line busy for a few
hours, if the phone call
342
00:19:25,730 --> 00:19:26,930
was a free one.
343
00:19:26,930 --> 00:19:30,390
So at those times the
exponential assumption for the
344
00:19:30,390 --> 00:19:33,260
phone call duration was
completely destroyed.
345
00:19:33,260 --> 00:19:36,430
But leaving that detail aside,
it's sort of a reasonable
346
00:19:36,430 --> 00:19:41,000
assumption to just get started
with this problem.
347
00:19:41,000 --> 00:19:43,790
All right, so now that we have
those assumptions, let's try
348
00:19:43,790 --> 00:19:46,510
to come up with the model.
349
00:19:46,510 --> 00:19:49,470
And we're going to set up
a Markov process model.
350
00:19:49,470 --> 00:19:52,990
Now the Poisson process runs in
continuous time, and call
351
00:19:52,990 --> 00:19:56,280
durations being exponential
random variables also are
352
00:19:56,280 --> 00:19:59,330
continuous random variables, so
it seems that we are in a
353
00:19:59,330 --> 00:20:01,020
continuous time universe.
354
00:20:01,020 --> 00:20:03,610
But we have only started
Markov chains for the
355
00:20:03,610 --> 00:20:05,310
discrete time case.
356
00:20:05,310 --> 00:20:07,370
What are we going to do?
357
00:20:07,370 --> 00:20:10,470
We can either develop the theory
of continuous time
358
00:20:10,470 --> 00:20:13,600
Markov chains, which
is possible.
359
00:20:13,600 --> 00:20:16,300
But we are not going to
do that in this class.
360
00:20:16,300 --> 00:20:20,460
Or we can discretize time
and work with a
361
00:20:20,460 --> 00:20:22,070
discrete time model.
362
00:20:22,070 --> 00:20:24,850
So we're going to discretize
time in the familiar way, the
363
00:20:24,850 --> 00:20:27,350
way we did it when we started
the Poisson process.
364
00:20:27,350 --> 00:20:31,270
We're going to take the time
axis and split it into little
365
00:20:31,270 --> 00:20:35,130
discrete mini slots, where
every mini slot
366
00:20:35,130 --> 00:20:36,880
has a duration delta.
367
00:20:36,880 --> 00:20:41,520
So this delta is supposed to
be a very small number.
368
00:20:41,520 --> 00:20:44,760
So what is the state
of the system?
369
00:20:44,760 --> 00:20:47,290
So, you look at the situation
in the system at some
370
00:20:47,290 --> 00:20:51,340
particular time and I ask you
what is going on right now,
371
00:20:51,340 --> 00:20:53,680
what's the information
you would tell me?
372
00:20:53,680 --> 00:20:56,990
Well, you would tell me that
right now out of these capital
373
00:20:56,990 --> 00:21:02,160
B lines, 10 of them are busy,
or 12 of them are busy.
374
00:21:02,160 --> 00:21:04,740
That describes the state of
the system, that tells me
375
00:21:04,740 --> 00:21:06,560
what's happening
at this point.
376
00:21:06,560 --> 00:21:10,800
So we set up our states base by
being the numbers from 0 to
377
00:21:10,800 --> 00:21:15,510
B. 0 corresponds to a state in
which all the phone lines are
378
00:21:15,510 --> 00:21:17,260
free, no one is talking.
379
00:21:17,260 --> 00:21:19,590
Capital B corresponds to
a case where all the
380
00:21:19,590 --> 00:21:21,580
phone lines are busy.
381
00:21:21,580 --> 00:21:24,020
And then you've got
states in between.
382
00:21:24,020 --> 00:21:28,050
And now let's look at the
transition probabilities.
383
00:21:28,050 --> 00:21:32,580
Suppose that right so now we
have i-1 lines that are busy.
384
00:21:36,280 --> 00:21:38,380
Or maybe, let me look here.
385
00:21:38,380 --> 00:21:41,460
Suppose that there's i
lines that are busy.
386
00:21:41,460 --> 00:21:44,770
What can happen the next time?
387
00:21:44,770 --> 00:21:48,180
What can happen is that the new
phone call gets placed, in
388
00:21:48,180 --> 00:21:53,670
which case my state moves up
by 1, or an existing call
389
00:21:53,670 --> 00:21:59,060
terminates, in which case my
state goes down by 1, or none
390
00:21:59,060 --> 00:22:03,530
of the two happens, in which
case I stay at the same state.
391
00:22:03,530 --> 00:22:06,930
Well, it's also possible that
the phone call gets terminated
392
00:22:06,930 --> 00:22:10,510
and a new phone call gets placed
sort of simultaneously.
393
00:22:10,510 --> 00:22:14,080
But when you take your time
slots to be very, very small,
394
00:22:14,080 --> 00:22:16,970
this is going to have a
negligible probability order
395
00:22:16,970 --> 00:22:19,850
of delta squared, so
we ignore this.
396
00:22:19,850 --> 00:22:22,190
So what's the probability of
an upwards transition?
397
00:22:22,190 --> 00:22:25,160
That's the probability that the
Poisson process records an
398
00:22:25,160 --> 00:22:29,250
arrival during a mini slot
of duration delta.
399
00:22:29,250 --> 00:22:31,270
By the definition of the
Poisson process, the
400
00:22:31,270 --> 00:22:35,550
probability of this happening
is just lambda delta.
401
00:22:35,550 --> 00:22:39,200
So each one of these upwards
transitions has the same
402
00:22:39,200 --> 00:22:41,000
probability of lambda delta.
403
00:22:41,000 --> 00:22:45,220
So you've got lambda deltas
everywhere in this diagram.
404
00:22:45,220 --> 00:22:49,180
How about, now, phone
call terminations?
405
00:22:49,180 --> 00:22:53,630
If you had the single call that
was active, so if you
406
00:22:53,630 --> 00:22:56,510
were here, what's the
probability that the phone
407
00:22:56,510 --> 00:22:57,820
call terminates?
408
00:22:57,820 --> 00:23:00,840
So the phone call has an
exponential duration with
409
00:23:00,840 --> 00:23:02,820
parameter mu.
410
00:23:02,820 --> 00:23:05,860
And we discussed before that an
exponential random variable
411
00:23:05,860 --> 00:23:08,940
can be thought of as the
first arrival time
412
00:23:08,940 --> 00:23:10,780
in a Poisson process.
413
00:23:10,780 --> 00:23:14,750
So the probability that you get
this event to happen over
414
00:23:14,750 --> 00:23:18,820
a delta time interval is
just mu times delta.
415
00:23:18,820 --> 00:23:22,280
So if you have a single phone
call that's happening right
416
00:23:22,280 --> 00:23:24,780
now, with probability
mu times delta, that
417
00:23:24,780 --> 00:23:27,070
call is going to terminate.
418
00:23:27,070 --> 00:23:30,020
But suppose that we have
i phone calls that
419
00:23:30,020 --> 00:23:31,750
are currently active.
420
00:23:31,750 --> 00:23:35,010
Each one of them has a
probability of mu delta, of
421
00:23:35,010 --> 00:23:39,200
terminating, but collectively
the probability that one of
422
00:23:39,200 --> 00:23:45,850
them terminates becomes
i times mu delta.
423
00:23:45,850 --> 00:23:48,570
So that's because you get the
mu delta contribution --
424
00:23:48,570 --> 00:23:51,540
the probability of termination
from each one of the different
425
00:23:51,540 --> 00:23:54,410
phone calls.
426
00:23:54,410 --> 00:23:58,290
OK, now this is an approximate
calculation, because it
427
00:23:58,290 --> 00:24:01,850
ignores the possibility that two
phone calls terminate at
428
00:24:01,850 --> 00:24:03,110
the same time.
429
00:24:03,110 --> 00:24:09,130
Again, the way to think of why
this is the correct rate, when
430
00:24:09,130 --> 00:24:13,610
you have i phone calls that are
simultaneously running and
431
00:24:13,610 --> 00:24:17,250
waiting for one of them to
terminate, this is like having
432
00:24:17,250 --> 00:24:21,170
i separate Poisson processes
that are running in parallel,
433
00:24:21,170 --> 00:24:23,720
and you ask for the probability
that one of those
434
00:24:23,720 --> 00:24:25,980
processes records an event.
435
00:24:25,980 --> 00:24:28,470
Now when you put all those
process together, it's like
436
00:24:28,470 --> 00:24:32,970
having a Poisson process with
total rate i times mu, and so
437
00:24:32,970 --> 00:24:36,210
i times mu delta is the overall
probability that
438
00:24:36,210 --> 00:24:39,580
something happens in terms of
phone call terminations at
439
00:24:39,580 --> 00:24:40,760
those times.
440
00:24:40,760 --> 00:24:43,470
So in any case, this is the
transition probability for
441
00:24:43,470 --> 00:24:46,280
downwards transitions.
442
00:24:46,280 --> 00:24:49,190
Now that we've got this, we
can analyze this chain.
443
00:24:49,190 --> 00:24:53,220
This chain has the birth death
form that we discussed towards
444
00:24:53,220 --> 00:24:54,770
the end of last lecture.
445
00:24:54,770 --> 00:24:58,360
And for birth death chains, it's
easy to write it out to
446
00:24:58,360 --> 00:25:00,710
find the steady state
probabilities.
447
00:25:00,710 --> 00:25:03,360
Instead of writing down the
balance equations in the
448
00:25:03,360 --> 00:25:07,950
general form, we think in terms
of a conservation of
449
00:25:07,950 --> 00:25:10,630
probabilities or of transitions
by looking at what
450
00:25:10,630 --> 00:25:14,640
happens across a particular
cut in this diagram.
451
00:25:14,640 --> 00:25:18,590
Number of transitions in the
chain that cross from here to
452
00:25:18,590 --> 00:25:21,310
here has to be approximately
equal to the number of
453
00:25:21,310 --> 00:25:24,850
transitions from here to there
because whatever comes up must
454
00:25:24,850 --> 00:25:27,520
come down and then come
up and so on.
455
00:25:27,520 --> 00:25:31,410
So the frequency with which
transitions of this kind are
456
00:25:31,410 --> 00:25:34,070
observed has to be the same as
the frequency of transitions
457
00:25:34,070 --> 00:25:35,690
of this kind.
458
00:25:35,690 --> 00:25:38,400
What's the frequency of how
often the transitions of this
459
00:25:38,400 --> 00:25:40,280
kind happen?
460
00:25:40,280 --> 00:25:45,350
And by frequency I mean quite
percentage of the mini slots
461
00:25:45,350 --> 00:25:48,120
involve a transition
of this kind?
462
00:25:48,120 --> 00:25:51,420
Well, for a transition of that
kind to happen we need to be
463
00:25:51,420 --> 00:25:55,730
at states i-1, which happens
this much of the time.
464
00:25:55,730 --> 00:25:59,360
And then the probability
lambda delta that the
465
00:25:59,360 --> 00:26:01,360
transition is of this kind.
466
00:26:01,360 --> 00:26:06,040
So the frequency of transitions
of with which this
467
00:26:06,040 --> 00:26:11,460
kind of transition is observed
is lambda delta times pi(i-1).
468
00:26:11,460 --> 00:26:17,680
This is the fraction of time
steps at which a transition
469
00:26:17,680 --> 00:26:20,180
from specifically this
state to specifically
470
00:26:20,180 --> 00:26:22,120
that state are observed.
471
00:26:22,120 --> 00:26:24,880
This has to be the same as
the frequency with which
472
00:26:24,880 --> 00:26:28,360
transitions of that kind are
observed, and that frequency
473
00:26:28,360 --> 00:26:32,310
is going to be i mu delta
times pi(i), and then we
474
00:26:32,310 --> 00:26:37,870
cancel the deltas, and we are
left with this equation here.
475
00:26:37,870 --> 00:26:43,260
So this equation expresses pi(i)
in terms of pi(i-1).
476
00:26:43,260 --> 00:26:47,020
So if we knew pi(0) we
can use that equation
477
00:26:47,020 --> 00:26:48,690
to determine pi(1).
478
00:26:48,690 --> 00:26:52,320
Once we know pi(1), we can use
that equation to determine
479
00:26:52,320 --> 00:26:55,490
pi(2), and so on,
you keep going.
480
00:26:55,490 --> 00:26:59,890
And the general formula that
comes out of this, I will not
481
00:26:59,890 --> 00:27:02,350
do the algebra, it's a
straightforward substitution,
482
00:27:02,350 --> 00:27:04,830
you find that pi(i), the steady
state probability of
483
00:27:04,830 --> 00:27:08,650
state i is given by this
expression, which involves the
484
00:27:08,650 --> 00:27:12,020
pi(0) from which we started.
485
00:27:12,020 --> 00:27:13,500
Now what is pi(0)?
486
00:27:13,500 --> 00:27:17,420
Well, we don't know yet, but
we can find it by using the
487
00:27:17,420 --> 00:27:19,520
normalization equation.
488
00:27:19,520 --> 00:27:22,680
The sum of pi(i) has
to be equal to 1.
489
00:27:22,680 --> 00:27:26,430
So the sum of all of those
numbers has to be equal to 1.
490
00:27:26,430 --> 00:27:30,370
And the only way that this can
happen is by setting pi(0) to
491
00:27:30,370 --> 00:27:33,320
be equal to that particular
number.
492
00:27:33,320 --> 00:27:38,970
So if I tell you the value of
capital B, you can set up this
493
00:27:38,970 --> 00:27:43,690
Markov chain, you can calculate
pi(0), and then you
494
00:27:43,690 --> 00:27:47,790
can calculate pi(i), and so you
know what fraction, you
495
00:27:47,790 --> 00:27:51,150
know the steady state
probabilities of this chain,
496
00:27:51,150 --> 00:27:53,920
so you can answer
the question.
497
00:27:53,920 --> 00:27:57,240
If I drop in at a random time,
how likely is it that I'm
498
00:27:57,240 --> 00:28:00,200
going to find the states
to be here, or the
499
00:28:00,200 --> 00:28:01,930
states to be there?
500
00:28:01,930 --> 00:28:03,790
So the steady state
probabilities are
501
00:28:03,790 --> 00:28:07,390
probabilities, but we also
interpret them as frequencies.
502
00:28:07,390 --> 00:28:12,000
So once I find pi(i), it also
tells me what fraction of the
503
00:28:12,000 --> 00:28:16,870
time is the state equal to i.
504
00:28:16,870 --> 00:28:20,030
And you can answer that question
for every possible i.
505
00:28:20,030 --> 00:28:22,180
Now, why did we do
this exercise?
506
00:28:22,180 --> 00:28:25,240
We're interested in
the probability of
507
00:28:25,240 --> 00:28:27,660
the system is busy.
508
00:28:27,660 --> 00:28:31,960
So if a person, a new phone
call gets placed, it just
509
00:28:31,960 --> 00:28:33,020
drops out of the sky.
510
00:28:33,020 --> 00:28:37,560
According to that Poisson
process, that new phone call
511
00:28:37,560 --> 00:28:41,770
is going to find the system
at a random state.
512
00:28:41,770 --> 00:28:45,290
That random state is described
in steady state by the
513
00:28:45,290 --> 00:28:47,440
probabilities pi(i)'s.
514
00:28:47,440 --> 00:28:51,940
And the probability that you
find the system to be busy is
515
00:28:51,940 --> 00:28:55,110
the probability that when you
drop in the state happens to
516
00:28:55,110 --> 00:29:01,380
be that particular number B. So
i sub b is the probability
517
00:29:01,380 --> 00:29:02,680
of being busy.
518
00:29:06,496 --> 00:29:09,470
And this is the probability
that you would like to be
519
00:29:09,470 --> 00:29:11,970
small in a well engineered
system.
520
00:29:11,970 --> 00:29:16,890
So you ask the question, how
should, given my lambda and
521
00:29:16,890 --> 00:29:22,570
mu, my design question is to
determine capital B the number
522
00:29:22,570 --> 00:29:25,720
of phone lines so that
this number is small.
523
00:29:31,290 --> 00:29:36,040
Could we have done, could we
figure out a good value for B
524
00:29:36,040 --> 00:29:39,670
by doing a back of the
envelope calculation?
525
00:29:39,670 --> 00:29:44,410
Let's suppose that lambda is
30 and that mu is 1/3.
526
00:29:49,190 --> 00:29:53,640
So I guess that's, let
us these rates to
527
00:29:53,640 --> 00:29:55,135
be calls per minute.
528
00:29:58,400 --> 00:30:03,100
And this mu, again, is
a rate per minute.
529
00:30:03,100 --> 00:30:07,860
Again, the units of mu are going
to be calls per minute.
530
00:30:07,860 --> 00:30:11,310
So since our time unit is
minutes, the mean duration of
531
00:30:11,310 --> 00:30:13,660
calls is 1/mu minutes.
532
00:30:13,660 --> 00:30:16,930
So a typical call, or
on the average a
533
00:30:16,930 --> 00:30:19,185
call lasts for 3 minutes.
534
00:30:23,900 --> 00:30:27,860
So you get 30 calls
per minute.
535
00:30:27,860 --> 00:30:31,540
Each call lasts for 3 minutes
on the average.
536
00:30:31,540 --> 00:30:36,290
So on the average, if
B was infinite,
537
00:30:36,290 --> 00:30:38,100
every call goes through.
538
00:30:38,100 --> 00:30:42,750
How many calls would be
active on the average?
539
00:30:42,750 --> 00:30:44,570
So you get 30 per minute.
540
00:30:44,570 --> 00:30:48,490
If a call lasted exactly 1
minute, then at any time you
541
00:30:48,490 --> 00:30:51,210
would have 30 calls
being active.
542
00:30:51,210 --> 00:30:54,850
Now a call lasts on the
average for 3 minutes.
543
00:30:54,850 --> 00:30:58,120
So during each minute
you generate 90
544
00:30:58,120 --> 00:31:00,460
minutes of talking time.
545
00:31:00,460 --> 00:31:05,400
So by thinking in terms of
averages you would expect that
546
00:31:05,400 --> 00:31:08,070
at any time there would
be about 90
547
00:31:08,070 --> 00:31:10,540
calls that are active.
548
00:31:10,540 --> 00:31:14,210
And if 90 calls are active on
the average, you could say OK,
549
00:31:14,210 --> 00:31:18,600
I'm going to set up my
capital B to be 90.
550
00:31:18,600 --> 00:31:22,250
But that's not very good,
because if the average number
551
00:31:22,250 --> 00:31:25,950
of phone calls that want to
happen is if the average
552
00:31:25,950 --> 00:31:29,900
number is 90, sometimes you're
going to have 85, sometimes
553
00:31:29,900 --> 00:31:31,460
you will have 95.
554
00:31:31,460 --> 00:31:34,300
And to be sure that the phone
calls will go through you
555
00:31:34,300 --> 00:31:37,790
probably want to choose your
capital B to be a number a
556
00:31:37,790 --> 00:31:40,460
little larger than 90.
557
00:31:40,460 --> 00:31:42,600
How much larger than 90?
558
00:31:42,600 --> 00:31:47,070
Well, this is a question that
you can answer numerically.
559
00:31:47,070 --> 00:31:50,280
So you go through the
following procedure.
560
00:31:50,280 --> 00:31:55,590
I tried different values of
capital B. For any given value
561
00:31:55,590 --> 00:31:59,230
of capital B, I do this
numerical calculation, I find
562
00:31:59,230 --> 00:32:03,530
the probability that the system
is busy, and then I ask
563
00:32:03,530 --> 00:32:07,060
what's the value of B that makes
my probability of being
564
00:32:07,060 --> 00:32:10,510
busy to be, let's say,
roughly 1 %.
565
00:32:10,510 --> 00:32:13,110
And if you do that calculation
with the parameters that they
566
00:32:13,110 --> 00:32:18,620
gave you, you find that B would
be something like 106.
567
00:32:18,620 --> 00:32:21,490
So with the parameters they gave
where you have, on the
568
00:32:21,490 --> 00:32:26,550
average, 90 phone calls being
active, you actually need some
569
00:32:26,550 --> 00:32:30,410
margin to protect against the
[?] fluctuation, if suddenly
570
00:32:30,410 --> 00:32:34,010
by chance more people want to
talk, and if you want to have
571
00:32:34,010 --> 00:32:37,890
a good guarantee that an
incoming person will have a
572
00:32:37,890 --> 00:32:40,860
very small probability of
finding a busy system, then
573
00:32:40,860 --> 00:32:44,860
you will need about
106 phone lines.
574
00:32:44,860 --> 00:32:49,740
So that's the calculation and
the argument that the Erlang
575
00:32:49,740 --> 00:32:52,250
went through a long time ago.
576
00:32:52,250 --> 00:32:55,610
It's actually interesting that
Erlang did this calculation
577
00:32:55,610 --> 00:32:58,590
before Markov chains
were invented.
578
00:32:58,590 --> 00:33:02,250
So Markov's work, and the
beginning of work on Markov
579
00:33:02,250 --> 00:33:06,770
chains, happens about 10-15
years after Erlang.
580
00:33:06,770 --> 00:33:10,180
So obviously he didn't call
that a Markov chain.
581
00:33:10,180 --> 00:33:13,350
But it was something that he
could study from first
582
00:33:13,350 --> 00:33:15,900
principles.
583
00:33:15,900 --> 00:33:19,160
So this is a pretty
useful thing.
584
00:33:19,160 --> 00:33:24,530
These probabilities that come
out of that model, at least in
585
00:33:24,530 --> 00:33:28,230
the old days, they would all
be very well tabulated in
586
00:33:28,230 --> 00:33:32,670
handbooks that every decent
phone company engineer would
587
00:33:32,670 --> 00:33:34,530
sort of have with them.
588
00:33:34,530 --> 00:33:38,890
So this is about as practical
as it gets.
589
00:33:38,890 --> 00:33:42,340
It's one of the sort of
standard real world
590
00:33:42,340 --> 00:33:43,810
applications of Markov chains.
591
00:33:47,040 --> 00:33:53,950
So now to close our subjects,
we're going to consider a
592
00:33:53,950 --> 00:33:57,310
couple of new skills and see how
we can calculate the few
593
00:33:57,310 --> 00:33:59,660
additional interesting
quantities that have to do
594
00:33:59,660 --> 00:34:01,340
with the Markov chain.
595
00:34:01,340 --> 00:34:04,800
So the problem we're going to
deal with here is the one I
596
00:34:04,800 --> 00:34:07,630
hinted that when I was talking
about this picture.
597
00:34:07,630 --> 00:34:09,870
You start at a transient
state, you're going to
598
00:34:09,870 --> 00:34:12,239
eventually end up
here or there.
599
00:34:12,239 --> 00:34:16,389
We want to find the
probabilities of one option of
600
00:34:16,389 --> 00:34:19,210
the two happening or the
other happening.
601
00:34:19,210 --> 00:34:23,880
So in this picture we
have a class of
602
00:34:23,880 --> 00:34:25,680
states that's are transient.
603
00:34:29,130 --> 00:34:34,219
These are transient because
you're going to move around
604
00:34:34,219 --> 00:34:37,170
those states, but there's a
transition that you can make,
605
00:34:37,170 --> 00:34:40,260
and you go to a state from
which you cannot escape
606
00:34:40,260 --> 00:34:41,560
afterwards.
607
00:34:41,560 --> 00:34:43,710
Are you going to end
up here or are you
608
00:34:43,710 --> 00:34:45,370
going to end up there?
609
00:34:45,370 --> 00:34:46,130
You don't know.
610
00:34:46,130 --> 00:34:47,280
It's random.
611
00:34:47,280 --> 00:34:50,940
Let's try to calculate the
probability that you
612
00:34:50,940 --> 00:34:54,800
end up at state 4.
613
00:34:54,800 --> 00:34:59,990
Now, the probability that you
end up at state 4 will depend
614
00:34:59,990 --> 00:35:02,200
on where you start.
615
00:35:02,200 --> 00:35:06,170
Because if you start here, you
probably have more chances of
616
00:35:06,170 --> 00:35:09,720
getting to 4 because you get
that chance immediately,
617
00:35:09,720 --> 00:35:12,390
whereas if you start here
there's more chances that
618
00:35:12,390 --> 00:35:16,260
you're going to escape that way
because it kind of takes
619
00:35:16,260 --> 00:35:17,770
you time to get there.
620
00:35:17,770 --> 00:35:20,810
It's more likely that
you exit right away.
621
00:35:20,810 --> 00:35:26,040
So the probability of exiting
and ending up at state 4 will
622
00:35:26,040 --> 00:35:28,190
depend on the initial state.
623
00:35:28,190 --> 00:35:33,900
That's why when we talk about
these absorption probability
624
00:35:33,900 --> 00:35:38,270
we include an index i that
tells us what the
625
00:35:38,270 --> 00:35:40,310
initial state is.
626
00:35:40,310 --> 00:35:44,350
And we want to find this
absorption probability, the
627
00:35:44,350 --> 00:35:46,820
probability that we end
up here for the
628
00:35:46,820 --> 00:35:48,660
different initial states.
629
00:35:48,660 --> 00:35:52,100
Now for some initial states this
is very easy to answer.
630
00:35:52,100 --> 00:35:55,280
If you start at state 4, what's
the probability that
631
00:35:55,280 --> 00:35:59,160
eventually you end up in
this part of the chain?
632
00:35:59,160 --> 00:35:59,820
It's 1.
633
00:35:59,820 --> 00:36:02,585
You're certain to be there,
that's where you started.
634
00:36:02,585 --> 00:36:06,140
If you start at state 5, what's
the probability that
635
00:36:06,140 --> 00:36:08,880
you end up eventually
at state 4?
636
00:36:08,880 --> 00:36:12,580
It's probability 0, there's
no way to get there.
637
00:36:12,580 --> 00:36:17,930
Now, how about if you start
at a state like state 2?
638
00:36:20,840 --> 00:36:26,350
If you start at state 2 then
there's a few different things
639
00:36:26,350 --> 00:36:27,840
that can happen.
640
00:36:27,840 --> 00:36:32,900
Either you end up at state 4
right away and this happens
641
00:36:32,900 --> 00:36:41,150
with probability 0.2, or you
end up at state 1, and this
642
00:36:41,150 --> 00:36:46,730
happens with probability 0.6.
643
00:36:46,730 --> 00:36:50,200
So if you end up at state
4, you are done.
644
00:36:50,200 --> 00:36:51,540
We are there.
645
00:36:51,540 --> 00:36:56,850
If you end up at state
1, then what?
646
00:36:56,850 --> 00:37:00,980
Starting from state 1 there's
two possibilities.
647
00:37:00,980 --> 00:37:05,360
Either eventually you're going
to end up at state 4, or
648
00:37:05,360 --> 00:37:10,010
eventually you're going
to end up at state 5.
649
00:37:10,010 --> 00:37:14,310
What's the probability
of this happening?
650
00:37:14,310 --> 00:37:19,680
We don't know what it is, but
it's what we defined to be a1.
651
00:37:19,680 --> 00:37:21,130
This is the probability --
652
00:37:21,130 --> 00:37:22,650
a1 is the probability --
653
00:37:22,650 --> 00:37:26,650
that eventually you settle in
state 4 given that the initial
654
00:37:26,650 --> 00:37:28,090
state was 1.
655
00:37:28,090 --> 00:37:30,580
So this probability is a1.
656
00:37:30,580 --> 00:37:34,510
So our event of interest
can happen in two ways.
657
00:37:34,510 --> 00:37:38,350
Either I go there directly,
or I go here
658
00:37:38,350 --> 00:37:39,710
with probability 0.6.
659
00:37:39,710 --> 00:37:43,730
And given that I go there,
eventually I end up at state
660
00:37:43,730 --> 00:37:46,540
4, which happens with
probability a1.
661
00:37:46,540 --> 00:37:52,140
So the total probability of
ending up at state 4 is going
662
00:37:52,140 --> 00:37:54,660
to be the sum of the
probabilities of the different
663
00:37:54,660 --> 00:37:57,230
ways that this event
can happen.
664
00:37:57,230 --> 00:38:04,250
So our equation, in this case,
is going to be, that's a2, is
665
00:38:04,250 --> 00:38:07,420
going to be 0.2 (that's the
probability of going there
666
00:38:07,420 --> 00:38:11,210
directly) plus with probability
0.8 I end up at
667
00:38:11,210 --> 00:38:17,160
state 1, and then from state 1
I will end up at state 4 with
668
00:38:17,160 --> 00:38:19,320
probability a1.
669
00:38:19,320 --> 00:38:25,330
So this is one particular
equation that we've got for
670
00:38:25,330 --> 00:38:28,560
what happens if we start
from this state.
671
00:38:28,560 --> 00:38:32,450
We can do a similar argument
starting from any other state.
672
00:38:32,450 --> 00:38:35,790
Starting from state i the
probability that eventually I
673
00:38:35,790 --> 00:38:39,960
end up at state 4 is, we
consider the different
674
00:38:39,960 --> 00:38:43,350
possible scenarios of where do
I go next, which is my state
675
00:38:43,350 --> 00:38:46,440
j, with probability Pij.
676
00:38:46,440 --> 00:38:50,980
Next time I go to j, and given
that I started at j, this is
677
00:38:50,980 --> 00:38:53,630
the probability that I
end up at state 4.
678
00:38:53,630 --> 00:38:56,920
So this equation that we have
here is just an abstract
679
00:38:56,920 --> 00:39:02,540
version in symbols of what we
wrote down for the particular
680
00:39:02,540 --> 00:39:04,540
case where the initial
state was 2.
681
00:39:04,540 --> 00:39:08,880
So you write down an equation
of this type for every state
682
00:39:08,880 --> 00:39:09,580
inside here.
683
00:39:09,580 --> 00:39:15,520
You'll have a separate equation
for a1, a2, and a3.
684
00:39:15,520 --> 00:39:19,200
And that's going to be a system
of 3 equations with 3
685
00:39:19,200 --> 00:39:25,250
unknowns, the a's inside
the transient states.
686
00:39:25,250 --> 00:39:29,190
So you can solve that 3 by
3 system of equations.
687
00:39:29,190 --> 00:39:33,590
Fortunately, it turns out to
have a unique solution, and so
688
00:39:33,590 --> 00:39:36,070
once you solve it you have found
the probabilities of
689
00:39:36,070 --> 00:39:40,880
absorption and the probability
that eventually you get
690
00:39:40,880 --> 00:39:42,610
absorbed at state 4.
691
00:39:51,540 --> 00:39:56,790
Now, in the picture that we had
here, this was a single
692
00:39:56,790 --> 00:39:59,570
state, and that one was
a single state.
693
00:39:59,570 --> 00:40:06,500
How do things change if our
recurrent, or trapping sets
694
00:40:06,500 --> 00:40:09,600
consist of multiple states?
695
00:40:09,600 --> 00:40:16,820
Well, it doesn't really matter
that we have multiple states.
696
00:40:16,820 --> 00:40:21,470
All that matters is that this
is one lump and once we get
697
00:40:21,470 --> 00:40:24,150
there we are stuck in there.
698
00:40:24,150 --> 00:40:33,170
So if the picture was, let's
say, like this, 0.1 and 0.2,
699
00:40:33,170 --> 00:40:36,960
that basically means that
whenever you are in that state
700
00:40:36,960 --> 00:40:42,620
there's a total probability of
0.3 of ending in that lump and
701
00:40:42,620 --> 00:40:45,100
getting stuck inside
that lump.
702
00:40:45,100 --> 00:40:50,660
So you would take that picture
and change it and make it
703
00:40:50,660 --> 00:40:57,330
instead a total probability of
0.3, of ending somewhere
704
00:40:57,330 --> 00:41:00,270
inside that lump.
705
00:41:00,270 --> 00:41:03,670
And similarly, you take this
lump and you view it as just
706
00:41:03,670 --> 00:41:07,570
one entity, and from any state
you record the total
707
00:41:07,570 --> 00:41:10,090
probability that given
that I'm here I
708
00:41:10,090 --> 00:41:12,140
end up in that entity.
709
00:41:12,140 --> 00:41:15,540
So basically, if the only
thing you care is the
710
00:41:15,540 --> 00:41:18,760
probability that you're going
to end up in this lump, you
711
00:41:18,760 --> 00:41:22,830
can replace that lump with a
single state, view it as a
712
00:41:22,830 --> 00:41:24,600
single state, and calculate
713
00:41:24,600 --> 00:41:26,180
probabilities using this formula.
714
00:41:33,830 --> 00:41:36,780
All right, so now we know
where the chain is
715
00:41:36,780 --> 00:41:37,860
going to get to.
716
00:41:37,860 --> 00:41:39,850
At least we know
probabilistically.
717
00:41:39,850 --> 00:41:42,690
We know with what probability
it is going to go here, and
718
00:41:42,690 --> 00:41:45,160
that also tells us the
probability that eventually
719
00:41:45,160 --> 00:41:47,160
it's going to get there.
720
00:41:47,160 --> 00:41:55,780
Other question, how long is it
going to take until we get to
721
00:41:55,780 --> 00:41:58,300
either this state
or that state?
722
00:41:58,300 --> 00:42:02,160
We can call that event
absorption, meaning that the
723
00:42:02,160 --> 00:42:05,130
state got somewhere into a
recurrent class from which it
724
00:42:05,130 --> 00:42:06,380
could not get out.
725
00:42:12,340 --> 00:42:12,510
Okay.
726
00:42:12,510 --> 00:42:15,570
Let's deal with that question
for the case where we have
727
00:42:15,570 --> 00:42:19,320
only 1 absorbing state.
728
00:42:19,320 --> 00:42:22,410
So here our Markov chain is a
little simpler than the one in
729
00:42:22,410 --> 00:42:23,580
the previous slide.
730
00:42:23,580 --> 00:42:25,870
We've got our transient
states, we've got our
731
00:42:25,870 --> 00:42:28,930
recurrent state, and once you
get into the recurrent state
732
00:42:28,930 --> 00:42:31,770
you just stay there.
733
00:42:31,770 --> 00:42:35,330
So here we're certain that no
matter where we start we're
734
00:42:35,330 --> 00:42:37,270
going to end up here.
735
00:42:37,270 --> 00:42:39,170
How long is it going to take?
736
00:42:39,170 --> 00:42:40,830
Well, we don't know.
737
00:42:40,830 --> 00:42:42,540
It's a random variable.
738
00:42:42,540 --> 00:42:44,660
The expected value of that
random variable,
739
00:42:44,660 --> 00:42:46,890
let's call it mu.
740
00:42:46,890 --> 00:42:50,510
But how long it takes to get
there certainly depends on
741
00:42:50,510 --> 00:42:52,470
where we start.
742
00:42:52,470 --> 00:42:56,390
So let's put in our notation
again this index i that
743
00:42:56,390 --> 00:42:59,220
indicates where we
started from.
744
00:42:59,220 --> 00:43:03,300
And now the argument is going to
be of the same type as the
745
00:43:03,300 --> 00:43:05,480
one we used before.
746
00:43:05,480 --> 00:43:11,620
We can think in terms of a tree
once more, that considers
747
00:43:11,620 --> 00:43:13,930
all the possible options.
748
00:43:13,930 --> 00:43:16,300
So suppose that you
start at state 1.
749
00:43:19,060 --> 00:43:24,110
Starting from state 1, the
expected time until you end up
750
00:43:24,110 --> 00:43:27,100
dropping states is mu1.
751
00:43:27,100 --> 00:43:30,700
Now, starting from state 1, what
are the possibilities?
752
00:43:30,700 --> 00:43:33,090
You make your first transition,
and that first
753
00:43:33,090 --> 00:43:36,550
transition is going to take
you either to state
754
00:43:36,550 --> 00:43:38,610
2 or to state 3.
755
00:43:38,610 --> 00:43:42,010
It takes you to state 2 with
probability 0.6, it takes you
756
00:43:42,010 --> 00:43:48,320
to state 3 with probability
0.4.
757
00:43:48,320 --> 00:43:52,490
Starting from state 2,
eventually you're going to get
758
00:43:52,490 --> 00:43:54,410
to state 4.
759
00:43:54,410 --> 00:43:55,990
How long does it take?
760
00:43:55,990 --> 00:43:58,480
We don't know, it's
a random variable.
761
00:43:58,480 --> 00:44:02,325
But the expected time until
this happens is mu2.
762
00:44:05,100 --> 00:44:08,250
Starting from state 2, how long
does it take you to get
763
00:44:08,250 --> 00:44:10,300
to state 4.
764
00:44:10,300 --> 00:44:13,870
And similarly starting from
state 3, it's going to take
765
00:44:13,870 --> 00:44:17,850
you on the average mu3
time steps until you
766
00:44:17,850 --> 00:44:20,070
get to state 4.
767
00:44:20,070 --> 00:44:24,450
So what's the expected value
of the time until I
768
00:44:24,450 --> 00:44:25,790
end at state 4?
769
00:44:31,070 --> 00:44:37,910
So with probability 0.6, I'm
going to end up at state 2 and
770
00:44:37,910 --> 00:44:42,650
from there on it's going to be
expected time mu2, and with
771
00:44:42,650 --> 00:44:45,850
probability 0.4 I'm going to
end up at state 3, and from
772
00:44:45,850 --> 00:44:48,730
there it's going to take
me so much time.
773
00:44:48,730 --> 00:44:52,790
So this is the expected time
it's going to take me after
774
00:44:52,790 --> 00:44:54,860
the first transition.
775
00:44:54,860 --> 00:44:59,420
But we also spent 1 time step
for the first transition.
776
00:44:59,420 --> 00:45:02,610
The total time to get there
is the time of the first
777
00:45:02,610 --> 00:45:06,330
transition, which is 1, plus
the expected time starting
778
00:45:06,330 --> 00:45:07,600
from the next state.
779
00:45:07,600 --> 00:45:10,210
This expression here is the
expected time starting from
780
00:45:10,210 --> 00:45:13,640
the next state, but we also need
to account for the first
781
00:45:13,640 --> 00:45:15,840
transition, so we add 1.
782
00:45:15,840 --> 00:45:18,010
And this is going
to be our mu1.
783
00:45:21,150 --> 00:45:25,370
So once more we have a linear
equation that ties together
784
00:45:25,370 --> 00:45:27,930
the different mu's.
785
00:45:27,930 --> 00:45:32,070
And the equation starting from
state 4 in this case, of
786
00:45:32,070 --> 00:45:35,620
course is going to be simple,
starting from that state the
787
00:45:35,620 --> 00:45:38,280
expected number of steps it
takes you to get there for the
788
00:45:38,280 --> 00:45:42,220
first time is of course, 0
because you're already there.
789
00:45:42,220 --> 00:45:45,640
So for that state this is fine,
and for all the other
790
00:45:45,640 --> 00:45:48,280
states you get an equation
of this form.
791
00:45:48,280 --> 00:45:51,410
Now we're going to have an
equation for every state.
792
00:45:51,410 --> 00:45:53,760
It's a system of linear
equations, once more we can
793
00:45:53,760 --> 00:45:57,290
solve them, and this gives us
the expected times until our
794
00:45:57,290 --> 00:46:03,890
chain gets absorbed in
this absorbing state.
795
00:46:03,890 --> 00:46:06,650
And it's nice to know that
this system of equations
796
00:46:06,650 --> 00:46:09,510
always has a unique solution.
797
00:46:09,510 --> 00:46:13,220
OK so this was the expected
time to absorption.
798
00:46:13,220 --> 00:46:16,720
For this case where we had this
scene absorbing state.
799
00:46:16,720 --> 00:46:23,740
Suppose that we have our
transient states and that we
800
00:46:23,740 --> 00:46:27,020
have multiple recurrent
classes, or
801
00:46:27,020 --> 00:46:28,590
multiple absorbing states.
802
00:46:33,850 --> 00:46:37,470
Suppose you've got the
picture like this.
803
00:46:37,470 --> 00:46:42,040
And we want to calculate the
expected time until we get
804
00:46:42,040 --> 00:46:44,110
here or there.
805
00:46:44,110 --> 00:46:47,830
Expected time until we get
to an absorbing state.
806
00:46:47,830 --> 00:46:50,320
What's the trick?
807
00:46:50,320 --> 00:46:55,010
Well, we can lump both of these
states together and
808
00:46:55,010 --> 00:47:00,660
think of them as just one bad
state, one place for which
809
00:47:00,660 --> 00:47:03,730
we're interested in how long
it takes us to get there.
810
00:47:03,730 --> 00:47:10,250
So lump them as one state, and
accordingly kind of merge all
811
00:47:10,250 --> 00:47:11,570
of those probabilities.
812
00:47:11,570 --> 00:47:15,470
So starting from here, my
probability that the next I
813
00:47:15,470 --> 00:47:18,690
end up in this lump and they
get absorbed is going to be
814
00:47:18,690 --> 00:47:21,630
this probability plus
that probability.
815
00:47:21,630 --> 00:47:24,010
So we would change
that picture.
816
00:47:24,010 --> 00:47:30,080
Think of this as being
just one big state.
817
00:47:30,080 --> 00:47:36,080
And sort of add those two
probabilities together to come
818
00:47:36,080 --> 00:47:39,360
up with a single probability,
which is the probability that
819
00:47:39,360 --> 00:47:42,590
starting from here next
time I find myself at
820
00:47:42,590 --> 00:47:44,440
some absorbing state.
821
00:47:44,440 --> 00:47:48,510
So once you know how to deal
with a situation like this,
822
00:47:48,510 --> 00:47:52,030
you can also find expected times
to absorption for the
823
00:47:52,030 --> 00:47:55,190
case where you've got multiple
absorbing states.
824
00:47:55,190 --> 00:47:57,990
You just lump all of those
multiple absorbing states into
825
00:47:57,990 --> 00:47:59,240
a single one.
826
00:48:01,550 --> 00:48:05,040
Finally, there's a
kind of related
827
00:48:05,040 --> 00:48:09,600
quantity that's of interest.
828
00:48:09,600 --> 00:48:13,480
The question is almost the same
as in the previous slide,
829
00:48:13,480 --> 00:48:16,950
except that here we do not have
any absorbing states.
830
00:48:16,950 --> 00:48:20,335
Rather, we have a single
recurrent class of states.
831
00:48:25,320 --> 00:48:27,840
You start at some state i.
832
00:48:27,840 --> 00:48:31,780
You have a special state,
that is state s.
833
00:48:31,780 --> 00:48:35,600
And you ask the question, how
long is it going to take me
834
00:48:35,600 --> 00:48:39,040
until I get to s for
the first time?
835
00:48:39,040 --> 00:48:41,140
It's a single recurrent
class of states.
836
00:48:41,140 --> 00:48:43,730
So you know that the state keeps
circulating here and it
837
00:48:43,730 --> 00:48:46,610
keeps visiting all of
the possible states.
838
00:48:46,610 --> 00:48:49,510
So eventually this state
will be visited.
839
00:48:49,510 --> 00:48:52,070
How long does it take
for this to happen?
840
00:48:55,290 --> 00:48:55,425
Ok.
841
00:48:55,425 --> 00:48:58,820
So we're interested in how
long it takes for this to
842
00:48:58,820 --> 00:49:02,080
happen, how long it takes until
we get to s for the
843
00:49:02,080 --> 00:49:03,000
first time.
844
00:49:03,000 --> 00:49:06,480
And we don't care about what
happens afterwards.
845
00:49:06,480 --> 00:49:10,110
So we might as well change this
picture and remove the
846
00:49:10,110 --> 00:49:15,470
transitions out of s and to make
them self transitions.
847
00:49:15,470 --> 00:49:18,140
Is the answer going to change?
848
00:49:18,140 --> 00:49:19,280
No.
849
00:49:19,280 --> 00:49:22,890
The only thing that we changed
was what happens
850
00:49:22,890 --> 00:49:25,760
after you get to s.
851
00:49:25,760 --> 00:49:28,810
But what happens after you
get to s doesn't matter.
852
00:49:28,810 --> 00:49:31,730
The question we're dealing with
is how long does it take
853
00:49:31,730 --> 00:49:33,900
us to get to s.
854
00:49:33,900 --> 00:49:37,010
So essentially, it's after we
do this transformation --
855
00:49:37,010 --> 00:49:40,740
it's the same question as
before, what's the time it
856
00:49:40,740 --> 00:49:43,990
takes until eventually
we hit this state.
857
00:49:43,990 --> 00:49:46,390
And it's now in this new
picture, this state is an
858
00:49:46,390 --> 00:49:48,410
absorbing state.
859
00:49:48,410 --> 00:49:50,710
Or you can just think from
first principles.
860
00:49:50,710 --> 00:49:55,720
Starting from the state itself,
s, it takes you 0 time
861
00:49:55,720 --> 00:49:57,430
steps until you get to s.
862
00:49:57,430 --> 00:50:01,610
Starting from anywhere else,
you need one transition and
863
00:50:01,610 --> 00:50:05,050
then after the first transition
you find yourself
864
00:50:05,050 --> 00:50:09,560
at state j with probability Pij
and from then on you are
865
00:50:09,560 --> 00:50:13,350
going to take expected time
Tj until you get to that
866
00:50:13,350 --> 00:50:14,710
terminal state s.
867
00:50:14,710 --> 00:50:17,340
So once more these equations
have a unique solution, you
868
00:50:17,340 --> 00:50:19,510
can solve them and
find the answer.
869
00:50:19,510 --> 00:50:22,530
And finally, there's a related
question, which is the mean
870
00:50:22,530 --> 00:50:24,210
recurrence time of s.
871
00:50:24,210 --> 00:50:30,490
In that question you start
at s, the chain will move
872
00:50:30,490 --> 00:50:34,090
randomly, and you ask how long
is it going to take until I
873
00:50:34,090 --> 00:50:37,160
come back to s for
the next time.
874
00:50:37,160 --> 00:50:38,390
So notice the difference.
875
00:50:38,390 --> 00:50:42,670
Here we're talking the first
time after time 0, whereas
876
00:50:42,670 --> 00:50:45,500
here it's just the first
time anywhere.
877
00:50:45,500 --> 00:50:51,250
So here if you start from
s, Ts* is not 0.
878
00:50:51,250 --> 00:50:54,450
You want to do at least one
transition and that's how long
879
00:50:54,450 --> 00:50:57,360
it's going to take me until
it gets back to s.
880
00:50:57,360 --> 00:51:00,900
Well, how long does it take
me until I get back to s?
881
00:51:00,900 --> 00:51:06,060
I do my first transition, and
then after my first transition
882
00:51:06,060 --> 00:51:11,920
I calculate the expected time
from the next state how long
883
00:51:11,920 --> 00:51:15,350
it's going to take me until
I come back to s.
884
00:51:15,350 --> 00:51:20,200
So all of these equations that I
wrote down, they all kind of
885
00:51:20,200 --> 00:51:22,500
look the same.
886
00:51:22,500 --> 00:51:23,620
But they are different.
887
00:51:23,620 --> 00:51:27,210
So you can either memorize all
of these equations, or instead
888
00:51:27,210 --> 00:51:30,580
what's better is to just
to get the basic idea.
889
00:51:30,580 --> 00:51:33,140
That is, to calculate
probabilities or expected
890
00:51:33,140 --> 00:51:35,830
values you use the total
probability or total
891
00:51:35,830 --> 00:51:38,260
expectation theorem and
conditional the first
892
00:51:38,260 --> 00:51:40,810
transition and take
it from there.
893
00:51:40,810 --> 00:51:43,230
So you're going to get a little
bit of practice with
894
00:51:43,230 --> 00:51:47,276
these skills in recitation
tomorrow, and of course it's
895
00:51:47,276 --> 00:51:48,730
in your problem set as well.