Lecture 15: The Last Renewal

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu

ROBERT GALLAGER: OK so today we're going to review a little bit Little's theorem-- we're going to review it a little bit, but say a few new things about it. I want to say something about Markov chains and renewal processes, because one of the most valuable things about understanding both is that you can use renewal theory to solve an extraordinary number of Markov chain problems, and you can use Markov chains to solve an awful lot of renewal problems.

And I want to make that clear today because it's a trick that you have perhaps seen in the homework, or perhaps you've missed it. If you missed it you've probably done a lot of extra work that you wouldn't have had to do otherwise. But it's a very, useful thing. So it's worth understanding it.

Finally we'll talk a little bit about delayed renewal processes at the end. What we will say essentially is there's a long section on delayed renewal processes, which goes through and does everything we did for ordinary renewal processes, and as far as almost all the asymptotic properties are concerned, they're exactly the same.

It's just modifying a few of the ideas a little bit. The essential thing there is that when you're looking at something asymptotically, and the limit as t goes to infinity, what happens in that first little burst of time doesn't really make much difference anymore.

So we will talk about that if we get that far. OK, one of the main reasons why convergence with probability 1 is so important, you've probably wondered why we're spending so much time talking about this, why we use it so often?

I would like you get some idea of why it is often much easier to use than the convergence of probability. So I'm going to give you two examples of that. One of them is this initial thing which we talked about in the notes also and I talked about on lecture before.

There's this nice theorem which says that if the sequence of random variables converges to some number-- alpha-- with probability 1, and if f of x is a real valued function of a real variable, that's continuous, that x equals alpha. In other words, as you start converging, as you get close to alpha, this function is continuous there. And since it's continuous there, you have to get closer and closer. That's the essence of that.

So It says that then it's function of zn, a function of a random variable, a real valued function of a random variable is also a random variable. It converges with probability 1 to f of alpha. That was the thing we use to get the strong law for renewals, which says it's a probability that the limit as t goes to infinity of n of t of omega over t is equal to 1 over x-bar with probability 1.

In other words, the probability of this set of sample points for which this limit exists is equal to 1 anytime you get confused by one of these statements, that says with probability 1, and by now you're probably writing this just as an add-on at the end, and you often forget that there's an awful lot tucked into that statement.

And I tried to put a little of it there. Initially when we talked about it we put more in, saying the probability of the set of omega such that this limit exists is equal to 1. We state it in all sorts of different ways.

But always go back, and think a little bit about what it's really saying. Random variables are not like numbers. Random variables are far more complicated things. Because of that they have many more ways they can approach limits. They have many more peculiar features about them.

But anyway, the fact that this theorem holds true it is a result of a little bit of monkeying around with n divided by the sum of n random variables, and associating that with an n of t over t. But it's also associated with this function here. So you have the two things. The thing which is difficult conceptually is this one here. So that's one place where we used the strong law, where if we try to state a weak law of large numbers for renewals, without being able to go from this strong law to the weak law, it'd really be quite hard to prove it.

You can sit down and try to prove it if you want to, and I think you'll see that it really isn't very easy. So strong law of renewals also holds if the expected value of x is equal to infinity.

In this case, understanding why this is true really requires you to think pretty deeply about random variables-- and have an infinite expectation of what that means. But the idea here is since x is a random variable-- in other words, it can't take on infinite values-- except with probability 0. So it's always finite.

So when you add a bunch of them you get something which is still finite. So that s sub n is a random variable. In other words, if you look at the probability that s sub n is less than or equal to t, and then you let t go off to infinity, the fact that s sub n is a random variable means that the probability that sn is less than or equal to t goes to 1.

And it does that for sample values with probability 1 is a better way to say it. Here I'm actually stating the convergence in probability 2, because it follows from the convergence with probability 1. Since you have convergence with probability 1, n of t over t also converges in probability, which says that the probability as t goes to infinity that n of t over t minus 1 over x-bar magnitude is greater than epsilon, is equal to 0.

That's this funny theorem we proved about convergence with probability 1. Set of random variables implies convergence in probability.

Here's another theorem about convergence which is called the Elementary Renewal Theorem. We talked about that, we proved half of it. After we talked about talked about Wald's equality. And we said the other half really wasn't very interesting. And I hope some of you at least looked at that. After you look at it, it's a bunch of mathematics, and a bunch of equations.

And it says this-- so we have three limits theorems about n of t. About what happens when t gets large there're this number of renewals that occur, and the time n of t. One of them is a strong law, which is really a sample path average. I'm trying to start using the words sample path average instead of time average because I think it gives you a better idea of what's actually going on.

But the strong law is really a sample path argument. The weak law-- this thing here about convergence and probability-- it still tells you quite a bit, because it tells you that as t it gets large, the probability that n of t over t can be significantly different than one over x-bar is going to 0.

This in a sense tells you even less. I mean why does this tell you less than this does? What's significant thing does this tell you that this doesn't tell you?

Suppose we had a situation where half the time with probability 1/2 n of t over t is equal to 2 over x4, and the other half of the time it's equal to 0. That can't happen, But according to this it could happen. The expected value of n of t over t would still be 1 over x4.

But this statement doesn't tell you when you think about whether n of t over t is really squeezing down on 1 of x-bar, it just tells you that the expected value of it is squeezing down on 1 over x-bar. So this is really a pretty weak theorem, and you wonder why people spend so much time analyzing it?

I'll tell you why in just a minute. And it's not it's not a pretty story. We talked about residual life. I want to use this which I think you all understand-- I mean for residual life, and for duration and age you draw this picture, and that is perfectly clear what's going on.

So I don't think there's any possibility of confusion with that. Here's the original picture of a sample path picture of a rival apex, of the number of arrivals up until time t climbing up. And then we look at residual life, the amount of time at any time until the next arrival comes.

This is strictly a sample path idea, for a particular sample path, from 0 to infinity, you look at the whole thing. In other words, think of setting up an experiment. And this experiment you view with the entire sample path for this particular sample point that you're talking about.

You don't stop at any time, you just keep on going. Obviously you can't keep on going forever, but you keep on going long enough that you get totally bored, and say well I'm not interested in anything after 20 years. And nobody will be interested in my results if i wait more than 20 years, and I'll be dead if I wait much slower than that.

So you say we will take this sample path for a very long time. This is the sample path that we get. We then argue that the integral of y of t over t as a sum of terms, this is a random variable here. y of t is a random variable. It's a number. If I put in a particular sample point, each of these terms here are random variables. The sum of them is a random variable.

And if I put in a particular sample point, it's a sum of numbers. Now, we did the following thing with that-- I think it was pretty straightforward. You look at what the sum is up to n of t. In other words, for a particular time that you're looking at, the experiment that you do is you integrate y of t-- which is this residual life function-- from 0 to t.

At the same time, at time t there's a certain number of renewals that have occurred, and you look at 1 over 2t times the sum of x of n squared, up to that point, not counting this last little bit of stuff here, and then you upper bound it by this sum, counting this little bit of extra stuff here.

And we pretty much proved in class and in the notes that this little extra thing at the end doesn't make any difference even if it's very big. Because you're summing over such a long period of time. That's one argument involved there.

The other argument involved is really very hidden, and you don't see it unless you write things down very carefully. But it tells you why the strong law of numbers are so important. So I wanted to talk about here a little bit.

What that says-- I mean the thing that is kind of fishy here, is here we're summing up to n of t, and we don't really know what n of t is. It depends on how many arrivals have occurred. And if you write this out carefully as a sample path statement what is it saying? Let's go into the next slide, and we'll see what it's saying.

For the sample point omega-- let's assume for the moment that this limit exists-- what you're talking about is the sum from n equals 1 up to the number of arrivals that have taken place up until time t for this particular sample point.

And it's the sum of these squares of these inner renewal times, and we're dividing by 2t, because we want to find the rate at which this is all going on. We write it out then as a limit. of x of n squared over omega divided by n of t of omega, times n of t of omega, divided by 2 2t.

In other words, we simply multiply and divide by n of t in omega. Why do we want to do that? Because this expression here looks very familiar. It's a sum of random variables. With the sum of n of t of omega random variables, but we know that as t gets large, n of t of omega gets large also.

So that we know that with probability 1, as t approaches infinity, this sum here-- if we forget about that term-- this sum here by the strong law of large numbers is equal to-- well in fact by the weak law of large numbers-- it is equal to the expected value of x squared.

If we take the limit of this term, we get the limit of n of t of omega over 2t. The strong law for renewals tells us that is equal to 1 over the expected value of x. So that gives us our answer.

Now why can we take a limit over this times this, and say that that's equal to the limit of this times the limit of that? If you're dealing with random variables, that's not correct in general.

But here what we're dealing with-- as soon as we put this omega in, we're dealing with a sum of numbers. And here we're dealing with a number also. For every value of t, this is the number.

In other words, this is a function of t numerical function of t, a real valued function of t. This is a real valued function of t. And what do you know about the limit of a product of two sequences of real numbers?

If you know a little bit of analysis, then you know that you're going to take the limit of a product, and take the sequence of that limit, then the answer that you get is the first limit times the second limit. OK?

I mean you might not recognize the statement in that generality, but if I ask you what it is the sum of a n times b n? If we know that the limit of a n is equal to say, a? And the limit of b n is equal to b? Then we know that this sum here, in the limit the limit is n to infinity, it's just going to be a times b. And you're going to sit down and argue that for yourselves looking at the definition of what a limit is. So it's not a complicated thing.

But you can't do that if you're not dealing with a sample path notion of convergence here. You can't make that connection. If you only want to deal with the weak law of large numbers, if you want to say infinity doesn't really make any sense because it doesn't exist, I can't wait that long. And therefore the strong law of large numbers doesn't make any sense. You can't go through that argument, and you can't get this very useful result which says-- What was I trying to prove? I was trying to prove that the expected value of residual life as a time average is equal to the expected value of x squared divided by 2 times the expected value of x.

This makes sense over finite times. Yes?

AUDIENCE: [INAUDIBLE]?

ROBERT GALLAGER: Yeah. No, I want to divide by n. And then I think that makes it all right, OK? Something like that. Let's see, is that right the way I have it now?

AUDIENCE: [INAUDIBLE].

ROBERT GALLAGER: Yeah I want to do that, it's right too, but if I want to take the summation-- oh no, the summation makes it messier, you're right.

The thing I'm trying to state is the limit as n goes to infinity of an times bn, is a times b. And there are restrictions there, you can't have limit of an going to 0 or something, and the limit of bn going to infinity, or strange things like that.

But what I'm arguing here-- I don't want you getting involved with this because I haven't really thought about it. What I want you to think about is the fact that you can use the laws of analysis for real numbers and whether you've studied analysis or not, you're all familiar with those because you all use them all the time.

And when you're dealing with the strong law of large numbers, you can convert everything down to a sample path notion, and then you're simply dealing with limits of real numbers at that point. So you don't have to do anything fancy.

So this result would be hard to do in terms of ensemble averages. If you look at the end of Chapter 4 in the notes, you see that the arguments there get very tricky, and very involved. It does the same thing eventually, but in a much harder way.

OK, for the sample point omega, oh, we did that.

OK, Residual life and duration are examples of renewal reward functions, so this is just saying what we've already said so let's not dwell on it anymore.

OK stopping trials. Stopping trials will be on the quiz. The Wald equality will be on the quiz. We will get solutions to problem set seven back to you, although we won't get your graded solutions back to you before the quiz. I hope we'll get them out tomorrow. I hope. Yes, we will. They will be on the web.

Stopping trial is a positive integer valued random variable such as for each n, the indicator random variable indicator of j equals n. In other words, the random variable which takes the value 1 if this random variable j is equal to 1, takes the value of 0 otherwise. If that random variable is a function of x1 up to x of n. If you look at x1 to x of n, and you can tell from just looking at that and not looking at the future at all whether you're going to stop at time n. Then you call that a stopping trial.

And we generalize that to look at a sequence x sub n, x sub 1 to x sub n and some other set of random variables-- v sub 1 to v sub n. And the same argument. If the rule to stop is a rule which is based on only what you've seen up until time n, then it's called the stopping trial.

Possibly the effect of stopping trial it's the same except that j might be a defective random variable. In other words, there might be some small probability that you never stop, that you just keep on going forever. When you look at one of these problems, you use stopping rules on, it's not immediately evident before you start to analyze it what do you ever stop or not. So you have to analyze it somewhat before you know whether you're going to stop.

So it's nice to do things in terms of defective stopping rules, because what you can do there holds true whether or not stop. Wald's equality then says if these random variables or a sequence, IID sequence they each have a mean x-bar, and if j is a stopping trial, and if the expected value of j is less than infinity-- in other words, if it exists-- then the sum at the stopping trial satisfied-- the expected value of the sum equals expected value of x times the expected value of j.

Those of you who did the homework this week noticed three examples of where this is used not to find the expected value of s sub j, but where it's used to find the expected value of j.

And I guess 90% percent of the examples I've seen do exactly that you can find the expected value of s of j very easily. I mean you have an experiment where you keep going until something happens. Something happens, this is the sum of these random variables reaches some limit. And when they reach the limit you stop.

If when they reach the limit you stop, you know what that limit is. You know what the expected value of s sub j is, because that's where you stop. And from that, if what x-bar is you then know what the expected value of j is. So we should really state it as expected value of j is equal to the expected value of s sub j divided by the expected value of x, because that's where you usually use it.

OK this question of whether the expected value of j has to be less than infinity or not. If the random variable x you're dealing with, is a positive random variable, then you don't need to worry about that restriction. The only time when you have to worry about this restriction is where x can be both positive and negative. And then you have to worry about it a little bit. If you don't understand what I just said, go back and look at that example of stop when you're ahead. Because the example of stop when you're ahead, you can't use Wald's equality there, it doesn't apply. Because the expected amount of time until you stop is equal to infinity, and the random variable has both positive and negative values. And because of that, the whole thing breaks down.

OK. Let's talk a little bit about Little's theorem. As we said last time, Little's theorem is essentially an accounting trick. I should tell you something about how I got into teaching this course. I got into teaching it because I was working on networks at the time. Queuing was essential in networks. And mathematicians have taken over the queuing field.

And the results were so complicated, I couldn't understand them. So I started teaching it as a way of trying to understand them. And I looked at Little's theorem, and like any engineer, I said, aha. What's going on here is that the sum of these waiting times is equal to the integral of L of t, the difference between A of t and D of t, as you proceed.

So there's this equality here. If I look at this next busy period, I have that same equality. If I look at the next busy period, I have the same in equality. And anybody with any smidgen of common sense knows that that little amount of business at the end, about that final period, can't make any difference. And because of that, Little's theorem is just this accounting equality.

It says that the sum of the w's is equal to the integral of L. And that's all there is to it. When you look at this more, and you look at funny queuing situations, you start to realize that these busy periods can take very long periods of time. They might be infinite. All sorts of strange things could happen. So you would like to be able to prove something.

Now, what happens when you try to prove it, this is the other reason why I ignored it when I started teaching this course. Because I didn't understand the strong law of large numbers. I didn't understand what it was. Nobody had ever told me that this was a theorem about sample values.

So I tried to prove it. And I said the following thing. The expected value of L by definition is 1 over t times the limit. L is the number of customers in the system at time t. So we're going to integrate the number of customers in a system over all this period t. And I'm going to divide by 1 over t.

Oh my God. Would you please interchange that limit in the 1 over t? I mean it's obvious when you look at it, that has to be what it is. And by this accounting identity, this is equal to the limit of this sum from I equals zero to N of t or w sub I over t. With this question of having omitted this last little busy period, whatever part of it you're in when you get the t.

I mean, that's the part that's common sense. You know you can do that. Now lambda is equal to the limit of 1 over t times A of t. A of t is just the number of arrivals up until time t. A of t, when we're doing Little's theorem, counts this fictitious arrival at time zero. Or I should say that the renewal theory omits as fictitious the real arrival at time zero, which is what's going on in Little's theorem.

So then the expected value of w is going to be the limit as t approaches infinity of 1 over a of t times the sum of w sub I. I'm going to break that up in the same way I broke this thing here up. It's the limit of t over a of t times the limit of w sub I from I equals 1 to a of t, 1 over t. Breaking up this limit here requires taking this sample pass view again.

In other words, you look at a particular sample point omega. And for that particular sample point omega, you simply had the same thing as I was saying here. And I don't know whether I said it right or not. But anyway, that's what we're using. It does work for real numbers. And therefore, what we wind up with is this, which is 1 over lambda, and this, which is the expected value of L, from this, from the accounting identity.

OK. So again, you're using the strong law as a way to get from random variables to numbers. And you understand how numbers work. OK, one more example of this same idea. One of the problems in the homework, problem set six are problems I thought-- yes.

AUDIENCE: About the previous slide, that's from [INAUDIBLE] right?

PROFESSOR: Yes, yes. Sorry.

AUDIENCE: Then how-- is there an easy way to go from there to the [INAUDIBLE] distributions?

PROFESSOR: Oh, to the ensemble average. Yes, there is, but not if you don't read the rest of Chapter 4. And it's not something I'm going to dwell on in class. It's--- it's something which is mathematically messy and fairly intricate. And in terms of common sense, you realize it has to be true.

I mean, if the ensemble average, up until time t, is approaching the limit, then you must have the situation that that limit is equal to the sample path average. The question is whether it's approaching a limit or not. And that's not too hard to prove. But then you have all this mathematics of going through the details of it, which in fact is tricky.

OK. So back to mark up change and renewal processes. You remember you went through a rather tedious problem where you were supposed to use Chebyshev's inequality to prove something. And maybe half of you recognized that it would be far easier to use Markov's inequality. And if you did that, this is what I'm trying to do here.

The question is, if you look at the expected amount of time, from state I in a Markov chain, until you return to state I in the Markov chain again, and you can do that by this theory of attaching rewards to states in a Markov chain. What you wind up with is this result that the expected renewal time in an ergodic Markov chain, is exactly equal to 1 over the steady state probability of that state.

And if any of you find a simple and obvious way to prove that from the theory of Markov chains, I would be delighted to find it. Because I've never been able to find any way of doing that without going into renewal theory. And renewal theory lets you do it almost immediately. And it's a very useful resolve.

So the argument is the following. You're going to let Y1, Y2, and so forth be the inter-renewal periods. Here we're looking at a sample path point of view again. No, we're not. Y1, Y2 are the random variables that are the inter-renewal periods. The elementary renewal theorem is something we talked about. It says that the expected value of n sub I of t divided by t is equal to 1 over the expected value of y.

That's the elementary renewal theorem for renewal theory. So we've stopped talking about Markov chains now. We've said, for this Markov chain, you can look at recurrences from successive this is to state I. That forms a renewal process. And according to the elementary renewal theorem for renewal processes, this is equal to 1 over y bar.

Now we go back to Markov chains again. Let's look at the probability of being in state I at time t, given that we were in state I at time zero. That's the probability that n sub I of t, minus n sub I of t minus 1 is equal to 1. That's the probability that there was an arrival at time t, which in terms of this renewal process means there was a visit to state I at time t.

Every time you get to state I, you call it a renewal. We've defined a renewal process which gives you a reward of 1 every time you hit state I, and reward of zero all the rest of the time. So this is equal to probability of Ni of t minus Ni of t minus 1. The probability that that's equal to 1. So its expected value of N sub I of t minus N sub I of t minus 1. That's the expected value.

This is always greater than or equal to this. This is either 1 or it's zero. You can't have two arrivals. You can't have two visits at the same time. So you add up all of these things. You sum this from n equals 1 up to t, and what do you get? You sum this, and it's a telescoping series.

So you add expected value of n, 1 of t minus n 0 of t, which is 0, plus n2 of t minus n1 of t, plus n3 of t minus n2 of t. And everything cancels out except the n sub I of t. The I here is just the state we're looking at. We could've left it out. So p sub I, I of t, approaches pi sub I exponentially. Because down there, very fast, it stays very close. If we sum up over n of them, and these are quantities which are approaching this limit, pi sub I, exponentially fast, then the sum divided by the number of terms we're summing over is just pi sub I also.

So what we have is a pi sub I that's equal to this limit. That's equal to the expected value of n sub I over t. The elementary renewal theorem reads out, which is equal to 1 over y bar. So the expected recurrence time for state I is equal to pi sub I.

If you look carefully at what I've done there, I have assumed that-- well, when I did it, I was assuming it was an ergodic Markov chain. I don't think I have to assume that. I think it can be periodic, and this is still true. You can sort that out for yourselves.

That is the first slide I'll use which says, whenever you see a problem trying to prove something about Markov chains and you don't know how to prove it right away, think about renewal theory. The other one will say, whenever you're thinking about a problem in renewal theory, and you don't see how to deal with it immediately, think about Markov chains. You can go back and forth between the two of them.

It's just like when we were dealing with Poisson processes. Again, in terms of solving problems with Poisson processes, what was most useful? It was this idea that you could look at a Poisson process in three different ways. You could look at it as a sum of exponential [INAUDIBLE] arrival times. You could look at it as somebody throwing darts on a line. And you could look at it as a Bernoulli process which is shrunk down to time zero with more and more arrivals.

By being able to look at each of the three ways, you can solve pieces of the problem using whichever one of these things is most convenient. This is the same thing too. For renewal processes and Markov chains, you can go back and forth between what you know about each of them, and find things about the other. So it's a useful thing.

Expected number of renewals. Expected number of renewals is so important in renewal theory that most people call it m of t, which is the expected value, of n of t. n of t, by definition, is the number of renewals that have occurred by time t. The elementary renewal theorem says the limit, as t goes to infinity, of expected value of n of t over t is equal to 1 over x bar.

Now, what happens here? That's a very nice limits theorem. But if you look at trying to calculate the expected value of n of t for finite t, there are situations where you get a real bloody mess. And one example of that is supposedly in our arrival interval, is 1 or the square root of 2.

Now, these are not rationally related. So you start looking at the times at which renewals can occur. And it's any integer times 1, plus any integer times the square root of 2. So the number of possibilities within a particular range is growing with the square of that range. So what you find, as t gets very large, is possible arrival instance are getting more and more dense. There are more and more times when possible arrivals can occur.

There's less and less structure to the time between those possible arrivals. And the magnitude of how much the jump is at that possible time, there's no nice structure to that. Sometimes it's big, sometime it's little. So m of t is going to look like an enormously jagged function.

When you get out to some large t, it's increasing. And you know by the elementary renewal theory that this is going to get close to a straight line here. m of t over t is going to be constant. But you have no idea of what the fine structure of this is.

The fine structure can be extraordinarily complicated. And this bothered people, of course, because it's always bothersome when you start out with a problem that looks very simple, and you try to ask a very simple question about it. And you get an enormously complicated answer.

So an enormous amount of work has been done on this problem, much more than it's worth. But we can't ignore it all because it impacts on a lot of other things. So if we forget about this kind of situation here, where the inter-arrival interval is either 1 or something which is irrational. If you look at an inter-arrival interval, which is continuous where you have a probability density.

And as a probability density is very nicely defined, then you can do things much more easily. And what you do to do that is you invent something called the renewal equation. Or else you look in a textbook and you find the renewal equation. And you see how that's derived. It's not hard. I'm not going to go through it. Because it has very little to do with what we're trying to accomplish here.

But what the renewal equation says is that the expected number of renewals up until time t satisfies the equation. But it's the probability that x is less than or equal to t. Plus a convolution here of m of t minus x times d f of x of x.

If you state this in terms of densities, it's easier to make sense out of it. It's the interval from zero to t, of 1 plus m of t minus x, all times the density of x dx This first term here just goes to 1 after t gets very large. So it's not very important. It's a transient term The important part is this convolution here, which tells you how m of t is increasing.

You look at that, and you say, that looks very familiar. For electrical engineering students who have ever studied linear systems, that kind of equation is what you spend your life studying. At least it's what I used to spend my life studying back when they didn't know so much about electric engineering. Now, there are too many things to learn. So you might not know very much about this.

But this is a very common linear equation in m of t. It's a differential. It's an integral equation from which you can find out what m of t is. You can think conceptually of starting out at m of 0, which you know. And then you use this equation to build yourself up gradually.

So you started m of t equals 0. Then you look at m of t with epsilon, where you get this term here a little bit. And you break this all up into intervals. And pretty soon, you find something very messy happening, which is why people were interested in it.

But it can be solved if this density has a rational or plus transform. Now I don't care about how to solve that equation. That has nothing to do with the rest of the course. But the solution has the following form. The expected value of n of t is going up with t as 1 over x bar. I mean, you know it has to be doing that.

Because we know from the elementary renewal theorem that eventually the expected value of n of t over t has to look like 1 over x bar. So that's that term here, m of t over t looks like 1 over x bar. There's this next term, which is a constant term, which is sigma squared of x divided by 2 times x bar squared.

That looks sort of satisfying because it's dimensionless. Minus 1/2, plus some function here which is just a transient, which goes away as t gets large. Now you have to go through some work to get this. But it's worthwhile to try to interpret what this is saying a little bit.

This epsilon of t, it's all this mess we were visualizing here. Except of course, this result doesn't apply to messy things like this. That only applies to those very simple functions that circuit theory people used to study. Because they were relevant for inductors and capacitors and resistors and that kind of stuff.

So we had this most important term. We have this term, which asymptotically goes away. And we have this term here, which looks rather strange. Because what this is saying is that this asymptotic form here, m of t, as a function of t, has this slope here, which is 1 over x bar as the slope. Then it has something added to it.

This is 1 over x bar plus some constant. And in this case, it's sigma squared over 2 x bar squared minus 1/2. If you notice what that sigma squared over 2 x bar squared minus 1/2 is for an exponential random variable, it's zero. So for an exponential random variable which has no memory, you're right back to this curve here, which makes a certain amount of sense. Because this is some kind of transient at zero, which says where you start out makes a difference.

Now, if you look at a-- I mean, I'm using this answer for simple random variables that I can understand just to get some insight about this. But suppose you look at a random variable, which is deterministic. x equals 1 with probability 1. What are these renewals look like then?

Start out here. No renewals for a while. Then you go up here. You're always underneath this curve here. You look at a very heavy tailed distribution like this thing we-- you remember this distribution? Where you have x is equal to epsilon with probability 1 minus epsilon, and that's equal to 1 over epsilon with probability epsilon.

So that the sample functions look like a whole bunch of very quick. And then there's this very long one. And then a lot of little quick ones. We can look at what that's doing as far as expected value of n of t is concerned. You're going to get an enormous number of arrivals. right at the beginning. And then you're going to go for this long period of time with nothing. So you're going to have this term here, which is sticking-- let's put you way up here. OK? So that now you're starting out because of this transient that you're starting out at a particular instant when you don't know whether you're going to get an epsilon or a 1 minus epsilon. You're not in one of these periods where you're waiting for this 1/epsilon period to end. Then you get that term there. Then you get this epsilon of t, which is just a term that goes with y.

OK so that's what this formula is telling you. Want to talk a little bit about Blackwell's theorem also. Mostly talking about things today that we're not going to prove. I mean, it's not necessary to prove everything in your life. It's important to prove those things which really give you insight about the result. I'm actually going to prove one form of Blackwell's theorem today. And I'll prove it-- guess how I'm going to prove it? I mean, the theme of the lecture today is if something is puzzling in renewal theory, use Markov chain theory. That's what I'm going to do. OK.

What does Blackwell's theorem say? It says that the expected renewal rate for large t is 1 over x bar. OK, it says that if I look at a little tiny increment here instead of this crazy curve here, if I look way out after years and years have gone by, it says that I'm going to be increasing at a rate which is very, very close to nothing. I might be lifted up a little bit or I might be lifted down a little bit, but the amount of change in some very tiny increment here of t to t plus epsilon.

Blackwell's theorem tries to say that the expected change in this very tiny increment here of size epsilon is equal to 1 over x bar times epsilon. The expectation equals epsilon over expected value of x. OK?

So it's saying what the elementary renewal theorem says, but something a lot more. The elementary renewal theorem says you take n of t, you divide it by t. When you divide it by t, you lose all the structure. That's why the elementary renewal theorem is a fairly simple statement to understand. Blackwell's theorem is not getting rid of all the structure, he's just looking at what happens in this very tiny increment here and saying, it behaves in this way.

Now, you think about this and you say, that's not possible. And it's not possible because I've only stated half of Blackwell's theorem. The other part of Blackwell's theorem says if you have a process that can only take jumps at, say, integer times, then this can only change at integer times. And since it can only change at integer time, I can't look at things very close too, I can't look at epsilon very small. I can only look at intervals which are multiples of that change time.

OK. So that's what he's trying to say. But the other thing that he's not saying, when I look at this very tiny interval here between t and t plus epsilon, it looks like he's saying that m of t has a density and that this density is 1 over x bar. And it can't have a density, either.

If I had any discrete random variable at all, that discrete random variable can only take jumps at discrete times. So you can never have a density here. If you have a density to start with, then maybe you have a density after you're through. But you can't claim it. So all you can claim is that for very small intervals, you have this kind of change. You'll see that that's exactly what his theorem says.

But when you make this distinction between densities and discrete, we still haven't captured the whole thing. Because if the interarrival interval is an integer random variable, namely it can only change at integer times, then you know what has to happen here. You can only have changes at integer times.

You can generalize that a little bit by saying if every possible value of the inter-renewal interval is a multiple of some constant, then you just scale the integers to be less or greater. So the same thing happens. When you have these random variables like we'll take one value 1 or value square root of 2, then that's where this thing gets very ugly and nothing very nice happens.

So Blackwell said fundamentally, there are two kinds of distribution functions-- arithmetic and non-arithmetic. I would say there are two kinds-- discrete and continuous. But he was a better mathematician than that and he thought this problem through more. So he knew that he wanted to lump all of the non-arithmetic things together.

A random variable has an arithmetic distribution if its set of possible sample values are integer multiples of some number, say lambda. In other words, if it's an integer valued distribution with some scaling on it. That's what he's saying there. All the values are integers, but you can scale the integers bigger or less by multiplying them by some number lambda.

And the largest such choice of lambda is called the "span of the distribution." So that when you look at m of t, what you're going to find is when t gets larger, only going to get changes at this span value and nothing else. So each time you have an integer times lambda, you will get a jump. And each time you don't have an integer times lambda, it has to stay constant. OK.

So if x is arithmetic with span lambda greater than 0, then every sum of random variables has to be arithmetic with a span either lambda or an integer multiple of lambda. So n of t can increase only at multiples of lambda. If you have a non-arithmetic discrete distribution like 1 and pi, the points at which n of t can increase become dense as t approaches infinity.

Well, what we're doing here is separating life into three different kinds of things. Into arithmetic distributions, which are like integer distributions. These awful things which are like two possible values as discrete, but they take values which are not rational compared with each other. Points at which n of t can increase become dense. And finally, the third one is if you have a density and then you have something very nice again. So what Blackwell's theorem says is that limit as n goes to infinity of m of t plus lambda minus m of t is equal to lambda divided by x bar. OK?

This is if you have an arithmetic x and a span of lambda. This is what we were saying should be the thing that happens and Blackwell proved that that is what happens. And he said that as t gets very large, in fact, what happens here is that this becomes very, very regular. Every span you get a little jump which is equal to the span size times the expected increase. And then you go level for that span interval and you go up a little more.

So that what you get in a limit is a staircase function. And this is lambda here. And this is lambda times 1 over x bar here. OK? And this might have this added or subtracted thing that we were talking about before. But in this interval, that's the way it behaves.

What he was saying for non-arithmetic random variables, even for this awful thing like pi with probability half and 1 with probability half, he was saying pick any delta that you want, 10 to the minus 6, if the limit is t goes to infinity of m of t plus delta minus m of t is equal to delta over x bar. At density has this behavior, but these awful things, like this example we're looking at, which gets very-- I mean, you have these points of increase which are getting more and more dense and increases which are more and more random. Yes.

AUDIENCE: Is this true only for x bar finite or is this true even if x bar is infinite?

PROFESSOR: If x bar is infinite? I would guess it's true if x bar is infinite because then you're saying that you hardly ever get increases. But I have to look at it more carefully. I mean, if x bar is infinite, it says that the limit is t goes to infinity if this difference here goes to 0. And it does. So yes, it should hold true then, but I certainly wouldn't know how to prove it. And if you ask me what odds, I would bet you $10 to $1 that it's true. OK? [CHUCKLE].

OK. Blackwell's theorem uses very difficult analysis and doesn't lead to much insight. If you could code that properly, I tried to read proofs of Blackwell's theorem. I've tried to read Blackwell's proof, and Blackwell is a guy who writes extraordinarily well, I've tried to read other people's proof of it, and I've never managed to get through one of those proofs and say, yes, I agree with that. But maybe you can go through them. It's hard to know. But I wouldn't recommend it to anyone except my worst enemies.

The hard case here is this non-arithmetic but discrete distributions. What I'm going to do is prove it for you now as returns to a given state in a Markov chain. In other words, if you have a renewal interval which is integer, you can only get renewals at times 1, 2, 3, 4, 5, up to some finite limit. Then I claim you can always draw a Markov chain for this. And if I can draw a Markov chain for it, then I can solve the problem. And the answer that I get is a surprisingly familiar result for Markov chains and it proves Blackwell's theorem for that special case.

OK, so for any renewal process with inter-renewals at a finite set of integer times, there's a corresponding Markov chain which models returns to state 0. I'm just going to pick an arbitrary state 0. I want to find the intervals between successive returns to state 0. And what I'm doing here in the Markov chain is I start off at state 0. The next thing that happens is I might come back to state 0 in the next interval.

One thinks it got passed out to you, the self-loop is not there. The self-loop really corresponds to returns in time 1, so it should be there. If you don't return in time 1, then you're going to go off to state 1, as we'll call it. From state 1, you can return in one more time interval, which means you get back in time 2. Here you get back in time 1, here you get back in time 2, here you get back in time 3, here you get back in time 4, and so forth. So you can always draw a chain like this.

And the transition probabilities-- nice homework problem would be to show that the probability of starting at i and going to i plus 1 is exactly this. Why is it this? Well, multiply this probability by this probability by this probability by this probability, and what you get is the probability that the return takes five steps or more. You multiply this by this by this by this. Multiply this thing for different values of i, and what happens? Successive terms-- this all cancels out. So you wind up with 1 minus piece of x of i when you're all done. Or i plus 1. OK?

Now here's the interesting thing. For a lazy person like me, it doesn't make any difference whether I've gotten this formula right or not. I think I have it right, but I don't care. I've only done it right because I know that some of you would be worried about it and some of you would think I was ignorant if I didn't show it to you. But it doesn't make any difference.

When I get done writing down that this is the Markov chain that I'm interested in, I look at this and I say, this is ergodic. I can get from any state here to any other state. I will also assume that it's aperiodic because if it weren't aperiodic, I would just leave out the states 1, 3, 5, and so forth for that period 2 and so forth. OK.

So then we know that the limit is n goes to infinity of p sub 00n. In other words, the probability of being in state 0 at time n given that you were in state 0 at time 0 is pi 0. pi 0 I can calculate. If I'm careful enough calculating this, I can also calculate the steady state probabilities here. Whether I'm careful here or not, I know that after I get rid of the periodicity here, that I have something which is ergodic here. So I know I can find those pi's.

Now, so we know that. pi 0, we already saw earlier today, is equal to 1 over the expected renewal time between visits to state 0. So with pi 0 equal to 1 over x bar and this equal to pi 0, the expected difference between the probability renewal of time n and the probability renewal of time n minus 1 is exactly 1 over x bar, which is exactly what Blackwell said. Yes.

AUDIENCE: Can you please explain why this proves the Blackwell theorem? I don't really see it.

PROFESSOR: Oh, I proved the Blackwell theorem because what I've shown here is that as n gets large, the probability that you will be in state 0 at time t given that you're in state 0 at time 0-- in other words, I'm starting off this renewal process in state 0. So the probability of being in state 0 at time n is really exactly this thing that Blackwell was talking about. OK? Blackwell was saying-- I mean, lambda here is 1 because I've just gotten rid of that. So the limit of m of t plus t plus 1 minus m of t is the expectation of a renewal at time t plus 1. OK? And that's 1 over x bar.

AUDIENCE: So why is this renewal process-- why is this Markov chain model exactly what we have in renewal? So you're claiming that we have a renewal if and only if we return to state 0 in this Markov chain.

PROFESSOR: Yeah.

AUDIENCE: So that's the thing I don't see. Is it supposed to be obvious?

PROFESSOR: Oh, you don't see why that's true? Let me try to do that. I thought that at least was obvious, but as I found as I try to develop this course, things which are obvious are the things which are often not obvious.

If I have a random variable, let's say, which takes on the value 1 with probability 1/2 and the probability 2 with probability 1/2 and I use that as the inter-renewal time for a renewal process, then starting off in time 0 with probability 1/2, I will have a renewal in time 1 and I will have a renewal in time 2 with probability 1/2 also.

AUDIENCE: Right.

PROFESSOR: That's exactly what this says if I draw it for-- I don't need that.

AUDIENCE: I see.

PROFESSOR: OK?

AUDIENCE: OK. Do you mind doing it for a slightly more complicated example just so it's easier to see in full generality? So it looks like [INAUDIBLE] values or something.

PROFESSOR: OK. And then this won't be-- let's make this 1/2. And this 1/2. OK. And this-- what is this going to be? I mean, this has to be 1 at this point.

AUDIENCE: So then this would take on 1 with probability 1/2, 2 with probability 1/4, and 3 with probability [INAUDIBLE]?

PROFESSOR: I think so, yes.

AUDIENCE: Good. Thanks.

PROFESSOR: I mean, this is a question of whether I've calculated these numbers right or not. And looking at this example, I'm not at all sure I have. But as I say, it doesn't make any difference. I mean, so long as you buy the fact that if I don't return in time 0, then I'm in some situation where it's already taken me one unit of time, I'm not through, I have to continue and I keep continuing and that's-- OK?

OK. And there's-- oh. I already explained delayed renewal processes. I will explain it again. A delayed renewal process is a modification of a renewal process for which the first inter-renewal interval x1 has a different distribution than the others. And the intervals are all independent of each other. So that the first interarrival period might do anything. After that, they all do the same thing.

And the argument here is if you're looking at a limit theorem that how long any-- if you're looking at the limit of how many arrivals occur over a very long period of time, the amount of time it takes this first arrival to occur doesn't make any difference. It occurs at some time. And after that, it gets amortized over an enormously long time which is going to infinity.

So if it takes a year for the first arrival to occur, I look at 1,000 years. If it only takes me six months for the first arrival to occur, well, I still look at 1,000 years, but I mean, you see the point. This first interval becomes unimportant compared with everything else. And because of that, the strong law still is going to hold. That says that convergencing probability also occurs. All it does as far as m of t is concerned, the expected value of m of t, it moves it up or moves it down, but doesn't change the slope of it and so forth.

Even if the expected time for the first renewal is infinite. And that sounds very strange, but that still is true and it's true by essentially the same argument. You wait until the first arrival occurs. It has to occur at some point. And after that, you can amortize that over as long as you want, you're just looking at a limit. When you look at a limit, you can take as long as you want to, and you take long enough that you wash out the stuff at the beginning.

I mean, if you told me that, I would say you're waving your arms. But if you read the last section of the notes and you summarize it, that's exactly how you will summarize it. OK.