The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: I want to review a little bit the things we did
last time, and carry them just a little bit
further, in some cases.
Remember, we're talking about the Markov processes now.
We defined a countable state Markov process in terms of an
embedded Markov chain and in each state an
embedded Markov chain.
There's a parameter nu sub i, which is the rate of an
exponential process for leaving that state.
And that describes what the Markov process does because
knowing what the Markov chain does and knowing what these
exponential random variables do that gives you everything
you need, if you will, to simulate this process.
It gives you, in principle, everything you need to
calculate everything you want to know about what's going on
in the process.
So if you remember, we found out that the--
we essentially defined steady state probabilities for the
process in terms of the steady state probabilities for the
Markov chain.
And what we were doing there is to first, restrict
ourselves to the case where the Markov chain is positive--
well, at least, is recurrent.
But let's imagine this positive recurrence so we
don't have to worry about those distinctions for the
time being.
If it's positive recurrent there's a set of pi sub j,
which gives the steady state probability that a transition,
over the long term, is going to go into state j.
If you look at the process overall time, if you look at a
sample path of it, with probability one pi sub j, is a
fraction of transitions that go into state j.
Since when you're in state j, the expected amount of time
that you stay in state j is 1 over nu sub j.
Nu sub j is the rate of an exponential random variable so
the expected value of that random variable is 1 over nu
sub j, so the fraction of time that you should be in state j
should be proportional to pi j over nu sub j.
But since these fractions have to add up to 1, we normalize
it by dividing by the sum of pi k over--
pi sub k over nu sub k.
That's the formula that we into it.
We showed last time that this formula made sense.
What we showed is that if you look at the--
What's that peculiar thing doing there?
If you look at the rate at which transitions are
occurring in this Markov chain, which is determined by
everything going on, but this rate at which transitions are
occurring, assuming we start in state i is equal to 1 over
the sum over k of pi sub k over nu sub k.
Namely, it's the same denominator over here, and you
can interpret this in the same way as you interpret this.
This is independent of the state that you start in.
These probabilities here, these fractions of time in
each state, are independent of where you start.
So Mi of t is a sample path, average rate at which
transitions occur with probability 1.
Namely, no matter where you start, and no matter what
sample path you're looking at with probability one, what you
wind up with is a number of transitions up to time t
divided by t, which goes to a limit of probability one, and
it's independent of the starting state.
OK, so if this sum here is infinite then something
peculiar is happening because when this sum is infinite,
even though the pi sub j's are positive, even though the
embedded Markov chain is positive recurrent, these
fractions of time in each state don't
make any sense anymore.
So that in fact, what you have is a Markov process which is
getting slower and slower and slower.
That's what this equals infinity means.
It means that these pi sub k's, the probability of being
in certain states with very high--
If the probability is being high of states where you spend
a long time in those states and that sum adds up to
infinity, this transition rate is equal to zero.
It means that the transitions rates are getting slower and
slower and slower, we showed an example of that of an mm1q.
Well, it wasn't quite mm1, but it was like an mm1q, but it
had the property that as you moved up to higher and higher
states, both arrivals got slower and slower, and also
the service time got slower and slower.
We called this a rattled server, and
also discouraged arrival.
So with the two of those things, as you move up to
higher and higher states, what happens is transitions get
slower and slower and eventually, you settle down in
a situation where nothing is getting done.
And that's because of the sum here.
It's not because of anything peculiar in
any one given state.
Let's see, where we're we?
OK, if this is equal to infinity, the transition rate
goes to 0 and the process has no meaningful steady state.
Otherwise, the steady state uniquely
satisfies this equation.
That's what we showed less time.
This says that the rate in equals the rate out for each
state, that's what this equation says.
This quantity over here p sub j is the probability that
you're in state j, at any given time, nu sub j is
expected amount of time you stay in there.
So this is the, in a sense, this product here corresponds
to the rate in state j.
The sum over here corresponds to the rate at which you're
going into state j.
All of these P's are greater than 0, and the sum of the P's
are equal to 1.
This is the case though we normally call a positive
recurrent process.
I notice I didn't define that in the notes.
I now notice, thinking about it even more, that I'm not
sure I want to define it that way, but anyway that's what
it's usually called.
It's called a positive recurrent process.
It requires both that this is less than infinity and the
steady state probabilities pi sub j have a solution.
If you have a birth/death process, you can deal with the
birth/death process in exactly the same way as we dealt with
birth/death Markov chains.
Namely, what goes up must eventually come down.
And what that means is a P sub j, this is the fraction of
time you're in state j.
This is the rate at which you move up, so this combination
here is the overall rate at which you're moving up, not
conditional on being in state j.
This is the overall probability of moving down.
So the transition rate up, in any given state, equals the
transition rate down.
OK, so that's a useful equation.
That's the way you usually solve
for birth/death processes.
The same as you use for solving for change, you can
get it from the chain equation very easily.
If you have an irreducible process, an irreducible
process is one where every state communicates with every
other state, remember?
Same as the definition for change.
If there is a solution to these equations here, mainly,
these are the equations where you directly solved for the
average time in each state.
The time spent in blah, blah, blah--
What this is saying again, is the rate at which you leave
state j is what this is, is equal to the rate at which you
enter state j, which is what this is.
So this again, is saying what comes in must go out.
And if this equation is satisfied, then the embedded
chain is positive recurrent and instead of solving for the
P's from the pi's, you're solving for the pi's from the
P. It's the same kind of equation.
Notice before P sub j was equal to pi sub j over nu sub
j normalized.
Now, pi sub j is equal to a P sub j, nu sub
j normalized again.
And you normalize this because the rate at which transitions
are taking place is normally not 1.
We also found that this rate at which transitions are
taking place, 1 over this quantity, is also equal to
this sum here.
So the sum of Pj nu sub j is the overall rate at which
transitions are taking place.
Now, if the sum of nu i times p sub i is infinite then,
according to this formula, each pi sub j is equal to 0,
which says that the embedded Markov chain has to be either
transient or is null-recurrent, and something
very, very strange is going on.
Because here we've solved these equations, we found
positive P sub j's, positive probabilities of being in each
state or, at least, that's the way we try to interpret it.
And then we find that as far as the transition rates are
concerned all of the transition rates
are equal to 0.
So what's going on?
I must admit frankly, I don't know what's going on.
Because, I mean, I know mathematically
what's going on.
This is what the equations say.
There's a nice example of it, which I'll look at in the next
slide, but to interpret this in any very satisfactory way
seems to be very hard.
OK, here's the example, and we talked about this a
little bit last time.
If we look at the embedded chain, we can call this a
hyperactive birth/death chain.
It's hyperactive in the sense that the higher the state gets
the faster the thing runs.
So as you get more customers into this queue, you could
imagine then the server runs faster, the
customers arrive faster.
And as it builds up, this keeps on going faster and
faster and faster.
For this chain, it looks like it's stable, doesn't it?
Because the probability of going up--
No, it looks like it's not stable.
The probability of going up is 0.6.
The probability of going down this 0.4.
We have that in each state.
This is what we've seen often for mm1q's, where things come
in at a faster rate than they go out, and the only thing
that could happen is the queue builds up.
Namely, you start out down here, and you keep moving up
and you move up forever, so something bazaar is happening.
But then, we solve these equations for the steady state
fraction of time we spend in each state,
and what do we get?
Because this is the probability of going up, and
this is the probability of going down from state 1.
And the rate at which things happen from state 1 is 2.
And the rate at which things happen in state 2 is 4,
what's going on?
Since the rate at which things happen here is 4, this rate of
going down is 2 times 1.6.
Here we are in--
If we're in state 1 and moving up, the rate at which
transitions occur in state 1 is 2.
The probability that the transition is up is 0.6, so
the rate of an upper transition is
0.6 times 2, 1.2.
If we're in this state, higher rate of transitions there
twice as high, you're moving down with probability 0.4, but
since the rate is twice as high, the overall rate at
which things are going down is 1.6.
This looks like a stable mm1q.
OK, bizarre.
We can solve this.
We can solve this with the formula for solving for the
average time in each state.
And what do we get using p sub j times q sub j, j plus 1. p
sub j is the fraction of time in state j.
q sub j, j plus 1, is the rate at which we
move out of that state.
That's equal to the fraction of time or in state j plus 1
times the rate at which we're moving down from j plus 1 to
j, if these equations make any sense.
But at any rate, we can solve these equations p sub j plus 1
has to equal 3/4 p sub j.
That ratio there is 3/4.
And the average time in each state, according to these
equations, which we can see there's something funny about
it, but it's 1/4 times 3/4 to the j because of this
relationship here.
But the sum of p sub j times nu sub j is equal to infinity,
which is why we can't get any steady state embedded chain
probabilities.
OK, well, in an effort to understand this, what you can
do is you can truncate this as mm1 Markov
process to just k states.
And it's very easy to truncate the Markov process because all
you have to do is just cut off all the transition rates that
go into these higher rate states.
When you cut off everything going beyond here, what you
get is just this Markov process shown here.
Well, Markov chain shown here, Markov process shown here,
and, at this point, we have a finite number of states.
Nothing funny can happen.
And what we get if we actually go through all the
calculations is that the fraction of time in state j is
1/4 times 1 minus 3/4 to the k.
This term goes to 0 with k, so this whole term here is
unimportant.
Times 3/4 to the j.
This 1/4 times 3/4 to the j was the result we had here
when we looked at all the states there.
Pi sub j, looking at this chain here, which is an
unstable chain, you go up with higher
probability than you go down.
What you get is pi sub j is 1/3 times 1 minus 2/3 to the
k, this term is going to 0 as k increases, times 2/3
to the k minus j.
In other words, when you truncate this change here,
which is unstable, what's going to happen is that you're
going to tend to stay in the higher states.
And you're going to dribble down to the lower states with
very little probability in the lower states.
As you increase k and increase the number of states that
you're dealing with what's happening is that you're
increasingly spending most of your time and these higher
ordered states.
So as you increase k, you just move up one on which states
are most likely.
So you get this kind of result here 2/3 to the k minus j,
which is decreasing rapidly with k.
Which says, as k goes to infinity, this goes to 0 for
all j, and this goes to a sensible quantity for each j.
When you sum up p sub j times nu sub j, this is giving you
the rate at which transitions occur, what you get is this
term, which doesn't amount to anything times 1/2 times 3/2
to the k minus 1.
This term here is approaching infinity exponentially, which
says that something doesn't make any sense here.
What's happening in this queue is that the rate at which
transitions occur is going to infinity.
If you start out at state 0, very rapidly the
state builds up.
The higher the state goes up, the faster
the transitions become.
So the transition rate is approaching infinity, and this
solution here, which looks like it makes perfect sense,
doesn't make any sense at all.
Because in fact, what's happening is the transition
rate is increasing as time goes on, whether the number of
transitions in a finite time is infinite are not with
probability 1.
I don't know.
I can't figure out how to solve that problem.
If I figure out how to do it, I will let you know.
But anyway, the point is, for this embedded chain in this
same process, the idea of steady state is totally
meaningless.
So the caution there is as you deal with Markov processes
more and more, and any time you deal with killing a great
deal, you deal with this kind of process all the time.
What you normally do then is you start solving for these
probabilities.
You simulate something, you figure out what these
probabilities are from the simulation, everything looks
fine until you look at the embedded chain.
And then when you look at the embedded chain, you realize
that this is all nonsense.
I wish I could say more about this but I can't.
But anyway, it's a note of caution when you're dealing
with Markov processes.
Check what the embedded chain is doing because it might not
be doing something very nice.
Let's go onto reversibility for Markov processes.
We talked about reversibility for Markov chains, sort of
half understood what was going on there.
Fortunately, for Markov processes, I think it's a
little easier to see what's going on than it was for
Markov chains.
So if you almost understand reversibility for Markov
chains then I'll be easy to get the extra things
that you need here.
For any Markov chain in steady state, the backward transition
probabilities were defined as pi sub i times Pi(j) star is
equal to pi j times P(j)i star.
In other words, the transition from i to j, the probability
of being in state i and going to state j, which is this
expression right here.
You can write it in two different ways.
And there's nothing magical or sophisticated here.
It's the probability that Xn plus 1 is equal to i times the
probability that Xn is equal to j, given that the next
state is equal to i.
We can do that.
There's nothing wrong with talking about the probability
that the state now as j, given that the state one time from
now, the state i.
And that's also equal to the probability that Xn is equal
to j times the probability.
You go from j to i.
This is just base law written in a particularly simple form.
This also holds for the embed chain of a Markov process.
So to draw a picture for this, you're sitting here in state
i, eventually, at some time t1, there's a transition.
You go to state j.
In state j, there's a transition rate nu sub j, so
after some time, whose expected value is 1 over nu
sub j at time t2, you go to another state.
That new state say is state k, so you start out in state i,
you go to state j, you stick there for a time, nu sub j,
and then you go on to state k.
OK, so if we look at this picture again, and we look at
it in terms of the sample time Markov chain, what's going on?
If you're moving right, in other words,
moving up in time.
And I again, urge you if you have trouble thinking of time
running backwards, think of left and right, because you
will have no trouble thinking of things going to the left
and things going to the right.
So moving to the right, which is the normal way to move
after entering state j, the exit rate is nu sub j.
In other words, we exit in each delta with a probability
nu sub j times delta.
The same holds moving left.
In other words, if you're at time t2, if moving this way
you move into state j, what happens?
You leave state j going leftward in each delta unit
time with--
There's this same rate here, which is the rate when you
look at it either way, is this rate here at which you leave
this state j, which is delta nu sub j, and all we're using
here is the memorylessness of the exponential distribution.
That's the only thing going on here.
So Poisson process is clearly reversible from the
incremental definition, and that's what we're using here.
And that what this means is that the steady state
probabilities, the pi sub i's, and also, the nu sub i's,
which are the rates at which transitions occur, are the
same going left as going right.
So this is the same kind of thing we had before, but I
think now, you can see it more easily because there's a
finite amount of time that you're sticking in state j,
and it's a random amount of time.
And moving in one direction, you're moving out of state j
with this constant rate delta, delta times nu j.
When you're going the other way, it's the same probability
that you're going out, going backwards in time, over a
period delta of delta times nu sub j.
It's the same both ways.
And I think this is easier to see than the
thing we did before.
OK, so the probability of having a right transition from
j to k in a little period of time as p sub j, fraction of
time in state j, times q sub jk transition
rating k times delta.
Similarly, if q star sub kj is the left going process
transition rate, the probability of having the same
transition as pk times q star sub kj, thus
we have this equation.
And this equation, turns into this.
The rate going backwards is going from k to j is equal to
nu sub k times the rate for the embedding Markov chain
going backwards.
So we define a Markov process as being reversible if q sub i
j star, if the backward rate is equal to the forward rate
for all i and j.
And if we assume positive recurrence and we assume that
pi sub i over nu sub i is less than infinity, which is the
stability equation, which says that the rate of transitions
has to be finite, then the Markov process is reversible,
if and only if, the embedded chain is.
OK, so this gives you a nice easy condition to talk about
reversibility.
You can either show that the chain is reversible or the
process is reversible.
They both work the same way.
And if you understand chains then you can use that to
understand processes.
If you understand processes, you can use that
to understand chains.
OK, so from that we get what I like to call
the guessing theorem.
Suppose a Markov process is irreducible, this means every
state communicates with every other state, it's easy to
verify irreducibility.
It's sometimes hard to verify whether a chain is positive
recurrent or not and whether the process is positive
recurrent or not.
So this guessing theorem says if a Markov process is
irreducible and if p sub i is a set of probabilities that
satisfy this equation here--
these are the reversibility equations--
p sub i times qij, rate at which you're going up is equal
to the rate at which you're going down for all i and j.
And when you find that p sub i that satisfies these equations
here, if it also satisfies this equation, then, in fact,
you're in business.
And what the theorem says is first, all of these average
probabilities are greater than zero for all i, 2, these
average probabilities, this p sub i, is the sample-path
fraction of time in state 1 with probability 1, 3, the
process is reversible, and 4, the embedded
chain is positive recurrent.
You get, all at once--
all you've got to do is find the
solution to those equations.
If you can find the solution to those equations and it
satisfies this, then you're done.
You don't need a birth/death chain or anything else.
All you need to do is guess a solution to those equations.
If you get one, then that establishes
everything you need.
OK, useful application.
All birth/death processes which satisfy this finite
number of transition conditions are reversible.
Remember, we talked about trees
with the Markov condition.
If you have a Markov graph which is a tree, and it
satisfies this condition, then that process has to be
reversible.
You get it from this theorem also because, again, you have
this condition that what goes over a branch in a tree, you
can only go up on that branch one more time,
then you can go down.
So you've got the same equality and the argument is
exactly the same.
OK.
So what do we get out of this?
What we get is Burke's theorem.
And what this Burke's theorem says--
Burke's theorem here, for processes, is what you usually
think of as Burke's theorem.
It was the original statement of Burke's Theorem before
people started to look at sample time chains.
And what it says is, given an M/M/1 queue in steady-state
with arrival rate lambda less than mu--
if you look at that condition, that's exactly what you need
for the stability condition we've been talking about--
the departure process is Poisson with rate lambda.
Now you would think the process--
and when we're thinking about Markov chains, you all thought
when you saw this that the departure process had rate mu
because the server was operating at rate and mu, who
said no, that's not right.
Because the service rate is mu anytime the server is active.
But is lambda is less than mu the server
is not always active.
The server is sometimes idle and therefore, what comes in
is the same as what goes out.
The only possibility is that this queue is gradually
building up over time, and then what goes out is less
than what comes in.
But otherwise, what comes in is what goes out.
And the rate at which the server is working is not mu,
that the server is working a rate lambda.
Mainly, the server is working at rate mu when it has things
to do but every time it doesn't have things to do it
takes a coffee break.
It's not doing anything.
And at that point, when you amortized over the time the
server is taking coffee breaks and the time the server is
working the rate is lambda because that's the rate at
which things are coming in.
You can only get one thing out for each thing in.
So either the queue builds up forever, in which case, what's
going out is less than what's coming in or the queue is not
building up and what goes out is exactly what's coming in.
I hope looking at it that way convinces you that what's
coming out here is the same rate as what's going in.
OK, the next thing is that the state x of t is independent of
the departures before t.
This was the same as the condition we had when we were
looking at Markov chains.
The state x of t is independent of the
departure before t.
Whether you look at this thing forward or backward, the state
is just the height of that graph there.
So when you're looking at it backwards, the state that you
go through over the process is the same as the set of states
when you're looking at it the other way.
It's just that you have to flip it around.
And it says that the state is independent of
departures before t.
The reason for that is that when you look at the backward
process, every departure here is an arrival here.
These arrivals here--
This is an arrival, this is an arrival,
this is a third arrival.
This is the fourth arrival and those depart later.
When you're looking at it that way, what's going on?
These things that are called departures up in the top graph
are called arrivals when you're looking at it going
this way like.
What this is saying is-- when you're looking at this as an
M/M/1 process, what it's saying is for the backward
M/M/1 process, the arrivals that come in later than me do
not affect my service time.
This is because we have first come first serve service, and
if somebody with an enormous service
requirement comes after me--
If I'm at the supermarket and somebody comes in with three
big barrels of food, I breath a sigh of relief because I'm
there first.
And I get out quickly, and I don't have to
wait for this person.
And that's essentially, what this is saying.
OK, the third one for first come first serve service, a
customer's arrival time, given that it departs at time t, is
independent of the departures before time t.
That's almost the same as two, but it's not
quite the same as two.
It only applies if you have first
come first serve service.
This is not quite true for Markov chains.
You remember when we stated Burke's
theorem for Markov chains.
We only had these first two statements, and now we have
this third statement.
Let me try to explain in the next slide why that's true.
If I can explain this to you and make you believe it on the
first time I feel very proud of myself.
Because this is the kind of thing you have to have
explained to you three times.
You have to sit down and think about it.
The first three times you think about it, you say,
that's all baloney.
The fourth time you think about it,
you say, that's false.
And maybe the fifth time you say, it's false again.
And the sixth time you look at it just the right way, and it
turns out to be true.
And then you go back a week later, and it's false again.
But anyway, let me try.
We're going to first look at the right moving
sample path, OK?
So departure at t up here is an arrival [? in ?] the
departure in the right moving sample path is an arrival in
the M/M/1 left moving sample path.
For first come first service, looking at
the left moving process--
that's the left moving process--
the departure time of an arrival at time t--
here's an arrival at time t--
the departure of that arrival, in this case, is that
departure right there.
That departure time depends on the arrivals and the service
requirements back over here because when we're looking at
things going this way, this is what happens first.
This is what happens later.
So we have these arrivals coming in this way, this
arrival here has nothing to do--
Well, this arrival here, in general, does have something
to do with this arrival, because if this arrival were
not finished by this time, we would have to wait for it.
The waiting time of this arrival here depends on what
happened before, which is over here.
It does not depend on the arrivals after it because it's
first come first serve service.
And therefore, anything that comes after is
sitting behind us.
It doesn't bother us at all.
OK, for the corresponding right moving process then
coming back the other way, the arrival time of that departure
is independent as the departures before t, and
that's exactly what this theorem is saying.
In order to make sense of this you have to do something that
we've been doing all along.
We don't always talk about is very much, but I hope you're
getting used to doing it.
You talk about sample paths.
When you're talking about sample paths, you're talking
about one particular instantiation of the process.
You prove a result for that sample path, then you say,
that's true for all sample paths.
And then, you say, a ha, then it must be true for the random
variables of which these sample paths are sample cases.
That's an argument we go through so many times.
In fact, when you take probability the first time,
pretty soon it becomes second nature to you, and
every once in while--
In fact it becomes so much second nature, that most
people don't even distinguish between random variables and
actual sample path numbers.
They use the same symbol for both of them.
Every once in a while I get confused by this, but
eventually, they get it straightened out, and that's
part of what this argument is here.
OK, let's test our understanding now.
Let's look at tandem M/M/1 queues.
First, we don't quite know what that means, but let's
consider two queues, one sitting right after the other.
Departures from the first queue move directly into the
second queue.
The partition here moved directly into here.
I've shown these departures at rate lambda because what
Burke's theorem says, is that these departures are, in fact,
a Poisson process, we rate lambda.
You think you're through at this point, but you're not.
You still have some difficult thinking to go through.
So you're assuming originally, that the input is Poisson.
You're assuming that the first queue has exponential
services, the service times are independent of each other,
they're all iid, and they're independent of the
inter-arrival times.
So this first thing here is just an M/M/1 queue like
you're used to.
We now know that what's coming out of that first queue is a
Poisson process.
It's a little hard to look at as a Poisson
process but it is.
It is not a Poisson process if you're conditioning on what's
going on in the first queue.
It's not a Poisson process conditional on these arrival
times here or on these service times here.
It's only a Poisson process if we're looking at the departure
times unconditional on anything else.
Conditional only on the other departure times or iid, That's
what we've proven.
But conditional on what's going on in the first queues
of those departures, are sure as hell not even
close to doing iid.
I mean, if you've got a lot of arrivals and you've got a lot
of long service times, the queue is going to get busy.
And what's coming out of that queue is going to be at rate
mu, which is what all of you believe until you see this
theorem, and think it through.
And that's why you believe it, because you look normally, at
time moving from left to right.
We're going to assume the service time at rates mu 1 and
mu 2 are independent from queue to queue.
And they're independent--
Ah, I didn't change that.
This doesn't make any sense.
The service rates at mu1 and mu2 are independent from this
queue to this queue, and they're independent of the
inter-arrival times back there.
Now, why do I have to change that?
Why can't I say that the service times at mu1 and mu2
are independent of both the arrivals to the first queue
and the arrivals to the second queue?
Because first, I can't assume things that are not given by
the model of the problem.
The things going from queue 1 into queue 2, I know they're
Poisson now, but I don't know that they're Poisson
independent of what's going into the first queue, and
independent of the service times in the first queue, and
they certainly aren't.
So what I have to say here is they're independent from queue
to queue and independent of the arrivals
at the first queue.
OK, we know that the arrivals at queue 2 are
Poisson at rate lambda.
That's true by Burke's theorem.
And they're independent of the service times at 2.
That alone makes the second queue M/M/1.
Because the second queue has arrivals.
It has service times.
The arrivals are independent of the service times.
It is, by definition, what an M/M/1 queue is.
It might depend on what's going on in the first queue to
a certain extent.
Burke's theorem tells us, in one sense, it doesn't depend
on it, and that's the next thing we have to deal with.
The states of the two systems are independent and the time
of a customer in system one is independent of
that in state two.
I'm not going to try to convince you of that here.
The text does, I think, do a pretty good job of
convincing you of it.
It's a more complicated argument than this.
And s something which you just have to either say, I believe
it because I'm too tired to do anything else.
Or you have to go through it several times, sometimes
disbelieving and sometimes believing until finally,
saying, I guess it must be true.
OK, so enough of that.
Let's go on to random walks, which is where we should have
been 45 minutes ago.
You see our problem here is that we would like to talk
about martingales a little bit in this course because
martingales are extremely important things.
You can prove all sorts of things from them.
An enormous amount of modern research in the field uses
martingales, and you can do a great deal with martingales
without knowing anything about measure theory.
And almost all the books that talk about martingales use
measure theory, and therefore, require you to take a couple
of extra terms of math courses before you can understand
what's going on.
So it's important to be able to say something about them,
in this course, so you have enough--
so you know when you have to learn more
about them at least.
So I want to get to that point, which is why I've been
speeding up a little bit from what I would like to do.
If you noticed, if you've been reading the notes, I have
skipped the [INAUDIBLE]
equations.
I have skipped semi-Markov processes.
We're not going to deal with them.
The [INAUDIBLE]
equations are quite important.
You use them quite a bit.
They play the same role for Markov processes that all the
business with eigenvalues we went through
play with Markov chains.
The trouble is we don't have time to do it.
And the other trouble is all the time that we would spend
understanding that would be time understanding linear
systems of differential equations.
And if you study that in math and you have a good feel for
it, fine, you can read the section and notes.
If you're not familiar with that, then you have to learn
something about it.
It's not something we should waste our time on because it's
not something we will use at any other time in this course.
OK, so we want to get onto random walks.
A random walk is, in fact, a very simple kind of animal.
Let x sub i, i greater than or equal to 1, be a sequence of
independent identically distributed random variables.
And let s sub n be x1 plus x2 plus x sub n for n greater
than or equal to 1.
Is this something that we spent a 1/4 of the term
talking about, or 1/2 the term talking about, or 3/4 of the
term talking about?
I don't know, but we've certainly spent an awful lot
of time talking about the sums of iid random variables.
What is different here is that instead of being interested in
these random variables s sub n, what we're interested in is
this process of the s sub n's.
I mean, we're interested in looking at the whole sequence
of them, and saying things about that.
What we would like to be able to do is say
questions of this sort.
You pick some alpha bigger than 0, doesn't
matter what it is, 1.
What's the probability that s sub n is greater than or equal
to alpha for at least one in greater than or equal to 0?
So we're not asking, what's the probability that s sub 10
is bigger than alpha.
We're asking, if we look at this sequence, s1, s2, s3, s4,
what's the probability that any one of
those terms cross alpha?
In other words, when we look at the entire sample path,
does the entire sample path lie below alpha or does the
sample path, at some point, cross alpha, then perhaps,
come back down again or continue to go up or do
whatever it wants to do.
What the question is, does it cross it at least once?
The questions connected with that are, if it does cross the
threshold, when does it cross the threshold?
That's important.
If it crosses the threshold, what's the overshoot with
which it crosses the threshold?
Does it go shooting way above the threshold
when it crosses it?
If you have one of these random variables that has very
long tails, then it's very likely that when you cross the
threshold, you've crossed a threshold because of some
sample value which is humongous.
And therefore, you typically are going way above the
threshold when you cross it.
That makes the study of random walks very much more
complicated than it would be otherwise.
These overshoot problems are extremely difficult, extremely
tedious, and unfortunately, very often, very important.
So that's a bad combination, to be tedious and important.
You would rather avoid things like that.
Next one is given two thresholds for a given alpha
bigger than 0, and a given beta less than 0--
here we are, starting out at 0.
We have a threshold up here and a threshold down here.
And we want to know what's the probability that we ever cross
one of the two thresholds.
That, I hope, you can answer right away.
Can you?
s sub n, after a long period of time, is
going to look Gaussian.
It's going to look Gaussian with a standard deviation,
which is growing gradually.
And as a standard deviation grows gradually, and we have
these two finite limits, eventually, we're going to
cross one of those thresholds.
So the question is not do you cross one of the thresholds?
The question is, which threshold do you cross?
What's the probability of crossing each of them?
At what end do you cross the one that you cross?
And what's the overshoot when you cross it?
So those are the questions there.
These threshold crossing problems are important.
They're important in almost everything that you use
stochastic processes for.
They're familiar to me for studying errors in digital
communications systems because that's sort of the basis of
all of that study.
They're important in studying the overflowing queues.
Anytime you build a queue, I mean, what are you building?
You're building a waiting room for customers, and you're
building a service facility.
And one of the first questions is, how big do I have to make
the waiting room?
How much storage do I need, is the question.
So you really want to be able to answer the question, what's
the probability that the queue will ever overflow the queue?
That's a threshold problem.
In hypothesis testing, when you want to find out which of
two hypotheses is true, this is important.
When you want to look at, what's often a more important
question, is you test for one or two hypotheses, and if
you're smart, and if you can do it, you keep the test
running until you're pretty sure that you
have the right answer.
If you've got the right answer right away, then you can stop,
and save time from there on.
If you have to go for a long time before you get the
answer, then you go for a long time
before you get the answer.
That's called sequential hypothesis
testing, and that's important.
There are these questions or ruin and other catastrophes,
which, as you've observed in this paper over the last few
years, most people don't know how to deal with them.
If they do know how to deal with them, they're too
affected by short term profits and things like that that do
what they should do.
But part of the trouble is people don't know how to deal
with those problems.
And therefore, if you're trying to maximize your
profits and you can say, I've talked to 10 experts, they've
told me five different things.
It gives you a perfectly good excuse for maximizing your
profits rather than doing something reasonably sensible
or reasonably safe.
So these are important problems that
we're dealing with.
I'm going to start with a brief discussion of three
simple cases.
First one is called simple random walks.
The next one is called integer random walks, and the third
one is called renewal processes.
These random walks we're talking about sound like very
simple minded things, but if you think about it for a
minute, all of these renewal processes we've spent all this
time studying and struggling with, are just a special case
of random walks.
I mean, a renewal process has non-negative random variables.
We're adding up those non-negative random variables
to see what's going on.
So a renewal process is just dealing with sums of iid
non-negative random variables, where this is dealing with a
case where the random variables can be either
positive or negative.
OK, so we're going to start with a little bit about them.
A random walk is called simple if the underlying random
variable xn n is the simplest random random variable of all.
Mainly, it's binary.
The probability that it's 1, that it's equal to 1, and the
probability is minus 1 is equal to q.
You could make it just 0 or 1, namely, make it Bernoulli, but
that makes it too special.
I mean, since we've been studying renewal processes,
where all these random variables are non-negative,
let's take the plunge and study the simplest case of
random variables, which can be either positive or negative.
The simplest case is this binary case, where the random
variables can be 1 or minus 1.
That's what a simple random walk is.
OK, it's just a scaling variation on our Bernoulli
process though.
The probability that xi is equal to 1 for m
out of the n trials.
Mainly, you're going to flip this coin x sub i, heads is 1,
tails is minus 1.
The probability that you get heads for m out of the n
trials is n factorial over m factorial
times n minus m factorial.
Number of combinations of n things taken m at a time times
p to the m times 1 minus p to the n minus m.
We've known that forever.
You probably knew in high school.
You relearned it when you were taking elementary probability.
We've talked about it in this course.
We can also view this as a Markov chain.
You start off in state 0 with probability p.
You go to state 1 with probability 1 minus p.
You go to state minus 1.
On the second trial, you either go to state 2 or to 1,
2 or to 0, or to minus 2, and so forth.
That's just another way of analyzing the same simple
random walk.
Just like in the Stop When You're Ahead game that we
talked about in class, the probability that we ever cross
a threshold at k equal to p over 1 minus p to the k.
Let me remind you of why this is true, remind you of
something else.
I want to find the probability that any s sub n is greater
than or equal to k, so I need this union here.
Many people write that as a maximum over n as the
probability that the maximum over the s sub n's is greater
than or equal to k.
Other people write it as the probability that the supremum
of this sum is greater than or equal to k.
Why do I like to use the union, other than the fact
that I like set theory more than algebra?
Anybody know what some simple reasons for that are?
Yeah?
AUDIENCE: It might not be the max [INAUDIBLE].
PROFESSOR: What?
AUDIENCE: You were just trying to decide whether it crosses--
the sum-- that k right now, whether it-- the maximum
implied that was as high as it [INAUDIBLE].
PROFESSOR: Not necessarily, no.
I mean, I might not be able to find the maximum.
If I'm not dealing with a simple random walk, and I look
at the sequence S1, S2, S3, S4, and I ask what's the
maximum of that sequence, the maximum might not exist.
The supremum always exists if you include infinity as a
possible value for it.
But the supremum is not very nice either.
Because suppose I had a random walk where it was possible for
the random variables to take on arbitrarily small values.
And suppose I had a sequence which went crawling up towards
1, as n increases, it keeps climbing up towards 1, but it
never quite gets there.
What's the supremum of that set of numbers?
1.
So the supremum is greater than or equal to 1.
But that's not what I'm interested in.
I'm interested in the question, what's the
probability that any one of these random variables is
greater than or equal to 1?
So that's the straightforward way to write it.
Now does anybody care about these minor differences of a
maximum or a supremum or what have you?
No.
The only reason you care about it is when you write a
maximum, after you become a little bit careful with your
mathematics, you say, is that a maximum?
Or is it a supremum?
So you write it as a supremum and then you go through the
argument I just went through.
And the point is, you're not interested in that at all.
All you're interested in is do you cross the
threshold or don't you?
And this is the, natural way to write it.
OK.
Now the next thing is, how do we get this formula here?
Well, if we're going to cross a threshold at k, and k is an
integer, there's not going to be any overshoot.
We're either going to get there and hit it, and then we
might go on beyond there.
We'll either hit it on [INAUDIBLE]
or we won't hit it.
Now if we're going to hit it at [INAUDIBLE], how are we
going to do that?
Well we have to hit 1 at some point.
Because you can only move up or down by one at a time.
So you're going to have to hit 1 at some point.
Given that you've hit 1, you have to go on and hit 2 at
some point.
Given that you've hit 2, you have to move up and hit 3 at
some point.
Those events are independent of each other.
So the probability that you ever get to k is the
probability that you can stop when you're ahead.
Namely, it's the probability that you go from 0 to 1 up to
the k-th power.
So I'm saying that p over 1 minus p is the probability you
ever move from state 0 up to state 1.
And when we were looking at Stop When You're Ahead, it was
exactly this game here.
We moved up with probability p, down with probability q.
We were looking at the probability we'd ever be ahead
in this gambling game where we kept betting money, and we
always had an infinite capital.
So we could keep on betting forever.
And we stopped as soon as we hit 1.
So it's exactly this question here.
The way we solved it at that point was to write an
equation for it.
On the first step, you either move up, and you're there.
Or you move down.
And if you move down, you have to move up twice.
So the probability you ever get from here to here is p
plus q times the probability you get from here squared.
You solve that quadratic equation.
You might remember solving that before.
And that's where you get to p over 1 minus p.
OK, so that's not terribly important.
What is important is that this is the only problem in random
walks I know which has a trivial solution.
Most problems have a harder solution than that.
But there is at least one problem which is easy.
Now for those of you worried about a final exam, final
exams have to find problems which are easy.
So that tells you something.
I mean, we have to take the easy results in this course,
and we have to use them in someway.
And since I'm telling you that, I probably
will include it, so--
OK.
Integer random walks.
That's the next kind of random walk to talk about.
x is an integer random variable.
So similarly, s sub n is an integer random variable.
So when we're moving up, we move up in integer values,
which makes things a little bit easier.
This means, again, that we can model this as a Markov chain.
In the Markov chain, we start at 0, we move up whatever
number of values x sub n can have, or we move down whatever
number of values it can have.
And then from there, we keep moving again.
So it's this Markov chain with a very regular structure,
where for each state, the set of various states of up
transition and set of down transitions are all the same
for all the states.
So that's a simple Markov chain to deal with.
OK, and then we said that renewal processes are special
cases of random walks where x is a positive random variable.
When you're sketching sample paths, the axes are usually
reversed from random processes to random walks.
I'm just pointing this out because every time you try to
go from a random walk to a two-way renewal process, you
will be happily writing equations and drawing pictures
of what you're doing.
And you will suddenly come up against this problem, which is
that whenever you draw a figure for a renewal process,
you draw it this way.
OK?
In other words, what you're looking at is the time at
which the n-th arrival occurs.
Time is going this way.
The time at which the first arrival occurs is S1.
The time at which the second arrival occurs is
S2, S3, and so forth.
These intervals here--
x1, x2, x3, x4, x5, and so forth--
whenever you're dealing with a random walk, what you're
interested in directly these s's and x's.
And you always draw it the opposite way
with the axes reversed.
And what's confusing to you at first is that when you see
this picture, it doesn't look like this picture at all.
These two pictures are identical
with the axes reversed.
One is the way you draw renewal processes.
The other is the way you draw random walks--
which also suggests that sometimes when you're dealing
with random walks, you want to draw a picture like the
picture you draw for renewal processes and vice versa.
So the two are very closely related.
OK, so I want to do a little bit about the queuing delay in
a G/G/1 queue.
We've talked about G/G/1 queues a little bit in terms
of Little's Theorem and various other things.
And there's one very simple thing about G/G/1 queues that
follows from looking at random walks.
And there's a nice part of the problem you can solve dealing
with random walks.
Let's let x sub i be the IID random variables, which are
inter-arrival intervals for the G/G/1 queue.
Here's the first inter-arrival interval.
Here's the second.
Here's the third.
Here's the fourth up here, and so forth.
The departure occurs--
there's an inherent arrival that we visualize at time 0.
y 0 is a service time that that arrival requires.
y1 is the service time that this first arrival requires.
y2 is the service time of the third arrival.
These service times are all independent of each other.
We've looked at this picture before.
What we haven't done is to draw the obvious
conclusion from it.
If you try to talk about what is the waiting time in queue
of the n-th customer, how can we draw that in a figure?
Well here it is in this figure.
The 0-th customer has no waiting time at all, because
there's nothing in the queue.
It goes directly into the server.
I'm just talking about queueing
time, not system time.
So 0 goes directly into the server, takes some long
service time, ends here.
The next arrival occurs at this time here after this
inter-arrival time.
And what's it have to do?
It has to wait until this customer finishes.
And then it has to wait for its own service time before it
gets out of the system.
The next customer comes in x2 units of time after the first
customer comes in.
So it comes in here.
It has to wait in queue until after the first customer gets
finished, and so forth.
Now look at the time from the arrival of the second customer
until the second customer goes into service.
What is that?
It can be written in two different ways.
And this is the observation that makes it possible to
analyze G/G/1 queues.
The inter-arrival time of the second customer plus the
queueing time of the second customer is equal to the
queuing time of the first customer plus the service time
of the first customer.
OK?
This time is equal to that time.
Put little ovals around them so you can spot it.
So in general, if the arrival n is queued, then xn plus wn,
the inter-arrival time for the n-th customer plus its waiting
time in queue is equal to the service time of the previous
customer plus the waiting time of the
previous customer in queue.
If arrival n sees an empty queue, then wn equals 0.
OK?
This is easy.
If you didn't draw this picture, it
would be very hard.
If you tried to do this by algebra, you
would never get done.
But when you draw a picture, there's nothing to it.
OK, so that says w sub n is equal to y sub n minus 1 minus
xn plus w sub n minus 1 if w sub n minus 1 plus yn minus 1
is greater than or equal to xn, namely, if xn
gets queued at all.
And otherwise, w sub n is equal to 0.
In other words, if the n-th customer enters a queue which
is empty, it doesn't wait in queue at all.
It goes right into service if that condition is satisfied.
So we can write this whole thing as w sub n is equal to
the maximum of w sub n minus 1 plus yn minus 1 minus xn or 0.
Namely, what's happening is that the amount of time that
each customer has to wait in the queue, not in the system--
waiting time in the queue is either this quantity here
that's related to the waiting time of the previous customer
added to the service time of the previous customer minus
the n arrival time of the customer before.
If you define u sub n as y sub n minus 1 minus plus xn--
y sub n minus 1 minus xn--
namely, the service time of the n minus first customer
minus the inter-arrival time for the n-th customer, this is
independent over n.
So this is a IID random variable.
Each of the arrivals x sub n are IID.
Each of the departures and each of the
service times are IID.
Arrivals and services are independent of each other.
So this u sub n random variable is a sequence of IID
random variables.
And what this is saying then, is that w sub n is equal to
the maximum of w sub n minus 1 plus un or 0.
If for the time being, ignore that maximum.
I mean, suppose the queue is very, very long.
The queue is very, very long.
You don't have 0.
Everybody has some service time.
This would be a random walk then. w sub n, the waiting
time for the n-th customer, is equal to the waiting time of
the n minus first customer plus u sub n.
u sub n is playing the role of the inter-arrival in the
random walk.
That's the x's that we had in the random walk.
w sub n minus 1 is the sum.
w sub n is equal to the previous sum plus the new
random variable coming in.
So without the maximum, this is just a random walk.
The w sub n's are the random walk based on these peculiar
random variables u sub i, which are the peculiar random
variables of service times minus inter-arrival times.
OK.
With the max, this says w sub n is like a random walk.
What does it do?
It keeps going up for a while you have a queue.
Then it might start dropping.
But it can't go negative.
Whenever it gets down to 0, it gets down below 0, the next
time you start at 0 again.
And you start going again.
The next time it goes negative, you get
bumped up to 0 again.
It's like a naughty kid who goes through all sorts of
problems, but as soon as he goes negative, somebody picks
him up and starts him back up again.
So it's a--
well, it's a different kind of process.
But anyway, it's like a random walk, except it has this
peculiar characteristic that any time it crosses the 0
threshold, it gets bumped back up, and you start over again.
So you have these segments which make it a renewal
process also.
You have renewals whenever you get to 0 and
you start over again.
And we'll talk about how those processes work later.
The text has another way of looking at it, where you look
at this random walk going backwards in time rather than
forward in time.
But we don't want to spend a lot of time on that.
We just want to see this does have something to do with
random walks.
I want to spend some real time studying detection and
decisions and hypothesis testing, because each one of
these things are very important in a number of
different fields.
If you study radar or any kind of military problem, detection
is very important.
You want to see whether something has happened or not.
And often you have to see whether it's happened in the
presence of a great deal of noise or something.
So you're not sure of whether it's happened or not.
So you make many observations to see whether it's happened.
And then you try to make a decision on the basis of all
of those things.
Decisions--
that's what control people think about all the time.
Control freaks always want to make decisions.
They don't want you to make decisions.
They want to make the decisions themselves.
But that's part of what studying random
processes is all about.
How do you make sensible decisions?
You generally--
when you make a decision, you make it with some uncertainty
connected to it.
I mean, people respond to this by pretending there isn't any
uncertainty.
They pretend when they made a decision that
they must be right.
But actually, they could be wrong.
And sensible people face the fact that they might be wrong.
And therefore, they try to analyze what's the probability
of being wrong, what's the probability of being right,
and all of these things.
Hypothesis testing--
all scientists deal with this.
There are competing theories in some field, and in these
competing theories, you want to find out which is the right
hypothesis.
So you do a lot of tests.
And after all those tests, you try to decide which hypothesis
you believe in.
Why is it important to make a choice?
Well it's important to make a choice in all sorts of reasons
and all sorts of areas.
Even in a scientific area, I mean-- the
field has to move somehow.
If the field has all sorts of open questions and if
everybody says, well, there might be quantum theory, or
there might not be quantum theory.
It might all be wrong.
It might all be right.
I don't know.
I mean there's noise in the system,
we can't tell anything.
So all they're dealing with is a set of very complicated
probabilities.
Instead of that, people do make decisions.
They say, let's proceed on the basis of what they think is
the best course of action.
You're voting for a candidate for public office.
Well you can say, I don't care.
But you can't vote for both.
You have to choose one or the other.
And you have to make a decision,
what's the best choice?
So these problems arise everywhere.
And if you leave these problems out of the study of
random processes, what are you left with?
A purely academic exercise.
OK?
All you're doing is you're establishing probabilities,
but you're not using them.
So this is sort of where the rubber hits the road, when you
study decision making or hypothesis testing or
detection, whichever one you want to call it.
But you absolutely need to make these hypothesis tests at
some point.
We're going to call them hypothesis tests because the
language used there seems to be easier to live with than
the language in other places.
Not that I like the way the statisticians do these things.
Statisticians talk about errors of the first kind and
errors of the second kind.
And I and everyone else I know never knows what's the first
kind and what's the second kind?
But they never talk about giving names to these things.
It's always first kind and seconds kind and so forth.
But we won't bother ourselves about that.
What we're going to do is we'll consider only problems
where you have two possible decisions.
So it's a binary decision problem, binary hypothesis
testing problem.
We're going to consider a sample space which has a
random variable called h in it.
The random variable h can have two possible values, 0 or 1.
And you're going to have the multiple observations that
you're going to make, Y1, Y2, Y3, Y4, up to y sub n, say.
To make the problem easy and to make it correspond to
random walks, we're going to assume that the observations
are IID conditional on h equals 0, and they're also ID
conditional on h equals 1.
If you're used to studying noise problems, a nice example
of this is you send one of two binary values over some
communication channel.
There's additive noise added to it.
At the receiver, what you see is what you sent plus some
additive noise.
You try to figure out from that sum of signal plus noise
was a 0 sent, or was a 1 sent?
And you don't know which was sent, and you
have to take a guess.
And usually if the noise is not too big, you can make a
very good guess, so you don't make many errors.
This is exactly the problem that we're
concerned with here.
If you have multiple observations, you can think of
sending that same binary digit n times.
And then on the basis of all of those observations, see
what you want to guess.
I mean scientific experiments--
does a scientific theory ever get resolved by one
experiment?
I mean sometimes you read in textbooks that it does.
But it never does.
Somebody does an experiment.
They come up with some conclusion.
And immediately 10 other groups around the world are
all doing the same experiment to validate it or
not validate it.
Maybe they don't all publish papers about it.
But pretty soon, that experiment is done so many
times with so many variations on it that you can think of
having multiple observations of the same thing.
OK.
So when we got all done with it, we're going to assume that
the observations will have probability density functions.
They're analog and the hypothesis is binary.
Doesn't make any difference, you can have observations that
are discrete also.
It's just whether you want to use probability mass functions
or probability density functions.
It looks a little bit easier to do it this way.
So the probability density of having n observations given
that the hypothesis l is the correct one is
this product here.
And that's true for both l equals 1 and l equals 0.
Let me go just a little bit further.
Baye's law then says that the probability that h is equal to
the hypothesis l is equal to the a priori probability that
the alt hypothesis is correct times this density divided by
the sum of the two densities.
Nothing fancy here at all.
If you compare these two probabilities--
probability that 0 is a correct hypothesis with the
probability that 1 is the correct hypothesis--
you take the ratio of these.
What you get this p0 over p1 times this density divided by
that density.
Well this was thought of well over 100 years ago, probably
150 years ago.
People fought about it terribly, terrible fights
about this.
One of the worst fights in science that's ever happened.
Then we had Bayesian statisticians and non-Bayesian
statisticians.
And you know, these people were all perfectly willing to
talk about conditional probabilities so long as they
were looking forward in time.
When they started looking backward in time, namely
looking at these conditional probabilities going backwards,
and saying, we have hypotheses in our model.
These have probabilities.
We will talk about these probabilities
of everything involved.
We have a complete probabilistic model.
We will talk about-- in this probabilistic model, when you
get a certain observation, what's the probability the one
hypothesis is correct?
What's the probability the other hypothesis is correct?
People suddenly lost everything they'd learned
about probability, and said this can't be right.
So there were enormous fights about it.
Anyway, I wanted to get to that point.
Think about that.
Next time we're going to start out and do a
little bit with this.
And suddenly a random walk is going to emerge.
So we will do that next time.