18. Recurrent Networks

• Download English-US transcript (PDF)

MICHALE FEE: All right, let's go ahead and get started. So we're starting a new topic today. This is actually one of my favorite lectures, one of my favorite subjects in computational neuroscience. All right, so brief recap of what we've been doing. So we've been working on circuit models of neural networks. And we've been working on what we call a rate model, in which we replaced all the spikes of a neuron with, essentially, a single number that characterizes the rate at which a neuron fires.

We introduced a simple network in which we have an input neuron and an output neuron with a synaptic connection of weight w between them. And that synaptic connection leads to a synaptic input that's proportional to w times the firing rate of the input neuron. And then we talked about how we can characterize the output, the firing rate of the output neuron, as some nonlinear function of the total input to this output neuron.

We've talked about different F-I curves. We've talked about having what's called a binary threshold unit, which has zero firing below some threshold. And then actually, there are different versions of the binary threshold unit. Sometimes the firing rate is zero for inputs below the threshold. And in other models, we use a minus 1. And then a constant firing rate of one above that threshold.

And we also talked about linear neurons, where we can write down the firing rate of the output neuron just as a weighted sum of the inputs. And remember that these neurons are kind of special in that they can have negative firing rates, which is not really biophysically plausible, but mathematically, it's very convenient to have neurons like this.

So we took this simple model and we expanded it to the case where we have many input neurons and many output neurons. So now we have a vector of input firing rates, u, and a vector of output firing rates, u. And for the case of linear neurons, we talked about how you can write down the vector of firing rates of the output neuron simply as a matrix product of a weight matrix times the vector of input firing rates. And we talked about how this can produce transformations of this vector of input firing rates.

So in this high-dimensional space of inputs, we can imagine stretching that input vector along different directions to amplify certain directions that may be more important than others. We talked about how you can do that, stretch in arbitrary directions, not just along the axes. And we talked about how that vector of-- that, sorry, matrix of weights can produce a rotation.

So we can have some set of inputs where, let's say, we have clusters of different input values corresponding to different things. And you can rotate that to put certain features in particular output neurons. So now you can discriminate one class of objects from another class of objects by looking at just one dimension and not the whole high-dimensional space.

So today, we're going to look at a new kind of network called a recurrent neural network, where not only do we have inputs to our output neurons from an input layer, but we also have connections between the neurons in the output layer. So these neurons in a recurrent network talk to each other. And that imbues some really cool properties onto these networks.

So we're going to develop the math and describe how these things work to develop an intuition for how recurrent networks respond to their inputs. We're going to get into some of the computations that recurrent networks can do. They can act as amplifiers in particular directions. They can act as integrators, so they can accumulate information over time. They can generate sequences. They can act as short-term memories of either continuous variables or discrete variables. It's a very powerful kind of circuit architecture.

And on top of that, in order to describe these mathematically, we're going to use all of the linear algebra tools that we've been developing so far. So, hopefully, a bunch of things will kind of connect together. OK, so mathematical description of recurrent networks. We're going to talk about dynamics in these recurrent networks, and we're going to start with the very simplest kind of recurrent network called an autapse network. Then we're going to extend that to the general case of recurrent connectivity.

And then we're going to talk about how recurrent networks store memories. So we'll start talking about a specific circuit models for storing short-term memories. And I'll touch on recurrent networks for decision-making. And this will kind of lead into the last few lectures of the class, where we get into how sort of specific cases of looking at how networks can store memories.

OK, mathematical description. All right, so the first thing that we need to do is-- the really cool thing about recurrent networks is that their activity can evolve over time. So we need to talk about dynamics, all right? The feed-forward networks that we've been talking about, we just put in an input. It gets weighted by synaptic strength, and we get a firing rate in the output, just sort of instantaneously. We've been thinking of you put an input, and you get an output.

In general, neural networks don't do that. You put an input, and things change over time until you settle at some output, maybe, or it starts doing something interesting, all right? So the time course of the activity becomes very important, all right? So neurons don't respond instantaneously to inputs. There are synaptic delays. There are integration of membrane potential. Things change over time.

And a specific example of this that we saw in the past is that if you have an input spike, you can produce a postsynaptic current that jumps up abruptly as the synaptic conductance turns on. And then the synaptic conductance decays away as the neurotransmitter unbinds from the neurotransmitter receptor, and you get a synaptic current that decays away over time, OK? So that's a simple kind of time dependence that you would get.

And that could lead to time dependence in the firing rate of the output neuron. OK, dendritic propagation, membrane time constant, other examples of how things can take time in a neural network. All right, so we're going to model the firing rate of our output neuron in the following way. If we have an input firing rate that's zero and then steps up to some constant and then steps down, we're going to model the output, the firing rate of the output neuron, using exactly the same kind of first order linear differential equation that we've been using all along for the membrane potential, for the Hodgkin-Huxley gating variables. The same kind of differential equation that you've seen over and over again.

So that's the differential equation we're going to use. We're going to say that the time derivative of the firing rate of the output neuron times the time constant is just equal to minus the firing rate of the output non plus v infinity. And so you know that the solution to this equation is that the firing rate of the output neuron will just relax exponentially to some new v infinity.

And the v infinity that we're going to use is just this non-linear function times the weighted input to our neuron. So we're going to take the formalism that we developed for our feed-forward networks to say, what is the firing rate of the output neuron as a function of the inputs? And we're going to use that firing rate that we've been using before as the v infinity for our network with dynamics. Any questions about that?

All right, so that becomes our differential equation now for this recurrent network, all right? So it's just a first order linear differential equation, where the v infinity, the steady state firing rate of the output neuron, is just this nonlinear function times the weighted sum of all the inputs. All right, and actually, for most of what we do today, we're going to just take the case of a linear neuron.

All right. So this I've already said. This I've already said. And actually, what I'm doing here is just extending this. So this was the case for a single output neuron and a single input neuron. What we're doing now is we're just extending this to the case where we have a vector of input neurons with a firing rate represented by a firing rate vector u, and a vector of output neurons with a fine rate vector v. And we're just going to use this same differential equation, but we're going to write it in vector notation.

So each one of these output neurons has an equation like this, and we're going to combine them all together into a single vector. Does that make sense? All right, so there is our vector notation of the activity in this recurrent network. Sorry, I forgot to put the recurrent connections in there.

So the time dependence is really simple in this feed-forward network, right? So in a feed-forward network, the dynamics just look like this. But in a recurrent network, this thing can get really interesting and start doing interesting stuff.

All right, so let's add recurrent connections now and add these recurrent connections to our equation. So in addition to this weight matrix w that describes the connections from the input layer to the output layer, we're going to have another weight matrix that describes the connections between the neurons in the output layer. And this weight matrix, of course, has to be able to describe a connection from any one of these neurons to any other of these neurons.

And so this weight matrix is going to be a function of the postsynaptic neuron, the weight-- the synaptic strength is going to be a function of the postsynaptic neuron and the presynaptic-- the identity of the postsynaptic neuron and the identity of the presynaptic neuron. Does that make sense?

OK, so there are two kinds of input-- a feed-forward input from the input layer and a recurrent input due to connections within the output layer. Any questions about that? OK, so there is the equation now that describes the time rate of change of the firing rates in the output layer. It's just this first order linear differential equation. And the infinity is just this non-linear function of the inputs, of the net input to this neuron, to each neuron.

And the net input to this set of neurons is a contribution from the feed-forward inputs, given by this weight matrix w, and this contribution from the recurrent inputs, given by this weight matrix, m. So that is the crux of it, all right? So I want to make sure that we understand where we are. Does anybody have any questions about that? No? All right, then I'll push ahead.

All right, so what is this? So we've seen this before. This product of this weight matrix times this vector of input firing rates just looks like this. You can see that the input to this neuron, this first output neuron, is just the dot product of these weights onto the first neuron and the dot product of that vector of weights, that row of the weight matrix, with the vector of input firing rates.

And the feed-forward contribution to this neuron is just the dot product of that row weight of this input weight matrix with the vector of input firing rates, and so on. If we look at the recurrent input to these neurons, the recurrent input to this first neuron is just going to be the dot product of this row of the recurrent weight matrix and the vector of firing rates in the output layer.

The recurrent inputs to the second neuron is going to be the dot product of this row of the weight matrix and the vector of firing rates. Yes?

AUDIENCE: So I guess I'm a little confused, because I thought it was from A. Oh, to A. OK.

MICHALE FEE: Yeah, it's always post, pre. Post, pre in a weight matrix. That's because we're usually writing down these vectors the way that I'm defining this notation. This vector is a column matrix, a column vector.

All right, so we're going to make one simplification to this. When we work with the recurrent networks, we're usually going to simplify this input. And rather than write down this complex feed-forward component, writing this out as this matrix product, we're just going to simplify the math. And rather than carry around this w times u, we're just going to replace that with a vector of inputs onto each one of those neurons, OK?

So we're just going to pretend that the input to this neuron is just coming from one input, OK? And the input to this neuron is coming from another single input. And so we're just going to replace that feed-forward input onto this network with this vector h. So that's the equation that we're going to use moving forward, all right? Just simplifies things a little bit so we're not carrying around this w u.

So now, that's our equation that we're going to use to describe this recurrent network. This is a system of coupled equations. What does that mean? You can see that the time derivative of the firing rate of this first neuron is given by a contribution from the input layer and a contribution from other neurons in the output layer. So the time rate of change of this neuron depends on the activity in all the other neurons in the network. And the time rate of change in this neuron depends on the activity of all the other neurons in the network.

So that's a set of coupled equations. And that, in general, can be-- you know, it's not obvious, when you look at it, what the solution is, all right? So we're going to develop the tools to solve this equation and get some intuition about how networks like this behave in response to their inputs.

So the first thing we're going to do is to simplify this network to the case of linear neurons. So we don't have-- so the neurons just fire. Their firing rate is just linear with their input. And so that's the equation for the linear case. All we've done is we've just gotten rid of this non-linear function f.

All right, so now let's take a very simple case of a recurrent network and use this equation to see how it behaves, all right? So the simplest case of a recurrent network is the case where the recurrent connections within this layer are given by-- the weight matrix is given by a diagonal matrix. Now, what does that correspond to? What that corresponds to is this neuron making a connection onto itself with a synapse of weight lambda one, right there.

And that kind of recurrent connection of a neuron onto itself is called an autapse, like an auto synapse. And we're going to put one of those autapses on each one of these neurons in our output layer, in our recurrent layer. So now we can write down the equation for this network, all right? And what we're going to do is simply replace-- sorry, let me just bring up that equation again. Sorry, there's the equation.

And we're simply going to replace this weight matrix m, this recurrent weight matrix, with that diagonal matrix that I just showed you. So there it is. So that time rate of change of this vector of output neurons is just minus v plus this diagonal matrix times [INAUDIBLE] plus the inputs.

So now you can see that if we write out the equation separately for each one of these output neurons-- so here it is in vector notation. We can just write that out for each one of our output neurons. So there's a separate equation like this for each one of these neurons. But you can see that these are all uncoupled. So we can understand how this network responds just by studying this equation for one of those neurons.

OK, so let's do that. We have an independent equation. The firing rate change-- the time derivative of the firing rate of neuron one depends only on the firing rate of neuron one. It doesn't depend on any other neurons. As you can see, it's not connected to any of the other neurons.

OK, so let's write this equation. And let's see what that equation looks like. So we're going to rewrite this a little bit. We're just going to factor out the va all right here. This parameter, 1 minus lambda a, controls what kind of solutions this equation has. And there are three different cases that we need to consider. We need to consider the case where 1 minus lambda is greater than zero, equal to zero, or less than zero.

Those three different values of that parameter 1 minus lambda give three different kinds of solutions to this equation. We're going to start with the case where lambda is less than one. And if lambda is less than 1, then this term right here is greater than zero. If we do that, then we can rewrite this equation as follows.

We're going to divide both sides of this equation by 1 minus lambda, and that's what we have here. And you can see that this equation starts looking very familiar, very simple. We have a first order linear differential equation, where we have a time constant here, tau over 1 minus lambda, and a v infinity here, which is the input, the effective input onto that neuron, divided by 1 minus lambda. So that's tau dv dt equals minus v plus v infinity.

But now you can see that the time constant and the v infinity depend on lambda, depend on the strength of that connection, all right? And the solution to that we've seen before, to this equation. It's just exponential relaxation toward v infinity.

OK, so here's our v infinity. There's our tau. True for the case of lambda between-- let's just look at these solutions for the case of lambda between zero and one. So I'm going to plot v as a function of time when we have an input that goes from zero and then steps up and then is held constant.

All right, so let's look at the case of lambda equals zero. So this lambda zero means there's no autapse. It's just not connected. So you can see that, in this case, the solution is very simple. It's just exponential relaxation toward infinity. v infinity is just given by h, the input, and tau is just the original tau, 1 minus 0, right? So it's just exponential relaxation to h. That make sense?

And it relaxes with a time constant tau, tau m. We're going to now turn up the synapse a little bit so that it has a little bit of strength. You see that what happens when lambda is 0.5, that v infinity gets bigger. v infinity goes to 2h. Why? Because it's h divided by 1 minus 0.5. So it's h over 0.5, so 2h. And what happens to the time constant? Well, it becomes two tau.

All right, and if we make lambda equal to 0.3-- sorry, 0.66. We turn it up a little bit. You can see that the response of this neuron gets even bigger. So you can see that what's happening is that when we start letting this neuron feed back to itself, positive feedback, the response of the neuron to a fixed input-- the input is the same for all of those. The response of the neuron gets bigger. And so having positive feedback of that neuron onto itself through an autapse just amplifies the response of this neuron to its input.

Now, let's consider the case where-- so positive feedback amplifies the response. And what also does it do? It slows the response down. The time constants are getting longer, which means the response is slower. All right, let's look at what happens when the lambdas are less than zero. What does lambda less than zero correspond to here?

AUDIENCE: [INAUDIBLE]

MICHALE FEE: Yeah, which is, in neurons, what does that correspond to?

AUDIENCE: [INAUDIBLE]

MICHALE FEE: Inhibition. So this neuron, when you put an input in, it tries to activate the neuron. But that neuron inhibits itself. So what do you think's going to happen? So positive feedback made the response bigger. Here, the neuron is kind of inhibiting itself. So what's going to happen? You put in that same h that we had before, what's going to happen when we have inhibition?

AUDIENCE: Response is [INAUDIBLE].

MICHALE FEE: What's that?

AUDIENCE: The response is going to be smaller.

MICHALE FEE: The response will just be smaller, that's right. So let's look at that. So here's firing rate of this neuron is a function of time for a step input. You can see for a lambda equals zero, we're going to respond with an amount h. But if we put in-- in a time constant tau. If we put in a lambda of negative one-- that means you put this input in-- that neuron starts inhibiting itself, and you can see the response is smaller.

But another thing that's real interesting is that you can see that the response of the neuron is actually faster. So if the feedback-- if the lambda is minus one, you can see that v infinity is h over 1 minus negative 1. So it's h over 2. All right, and so on. The more we turn up that inhibition, the more suppressed the neuron is, the weaker the response that neuron is to its input, but the faster it is. So negative feedback suppresses the response of the neuron and speeds up the response.

OK, now, there's one other really important thing about recurrent networks in this regime, where this lambda is less than one. And that is that the activity always relaxes back to zero when you turn the input off. OK, so you put a step input in, the neuron responds, relaxing exponentially to sum of v infinity. But when you turn the input off, the network relaxes back to zero, OK?

So now let's go to the more general case of recurrent connections. Oh, and first, I just want to show you how we actually show graphically how a neuron responds-- sorry, how one of these networks respond. And a typical way that we do that is we plot the firing rate of one neuron versus the firing rate of another neuron. That's called a state-space trajectory. And we plot that response as a function of time after we put in an input.

So we can put an input in described as some vector. So we put in some h1 and h2, and we then plot the response of the neuron-- the response of the network in this output state space. So let me show you an example of what that looks like. So here is the output of this little network for different kinds of inputs. So Daniel made this nice little movie for us.

Here, you can see that if you put an input into neuron one, neuron one responds. If you put a negative input into neuron one, the neuron goes negative. If you put an input into neuron two, the neuron responds. And if you put a negative input into neuron two, it responds. Now, why did it respond bigger in this direction than in this direction?

AUDIENCE: That's [INAUDIBLE].

MICHALE FEE: Good. Because neuron one had--

AUDIENCE: Positive?

MICHALE FEE: Positive feedback. And neuron two had negative feedback. So neuron one, this neuron one, amplified its input and gave a big response. Neuron two suppressed the response to its input, and so it had a weak response.

Let's look at another interesting case. Let's put an input into these neurons-- not one at a time, but simultaneously. So now we're going to put an input into both neurons one and two simultaneously. It's like Spirograph. Did you guys play with Spirograph? It's kind of weird, right? It's like making little butterflies for spring.

So why does the output-- why does the response of this neuron to an input, positive input to both h1 and h2, look like this? Let's just break this down into one of these little branches. We start at zero. We put an input into h1 and h2, and the response goes quickly like this and then relaxes up to here. So why is that? Lena?

AUDIENCE: [INAUDIBLE] so there was [INAUDIBLE] and then because it's negative, it's shorter.

MICHALE FEE: Yup. The response in the v2 direction is weak but fast.

AUDIENCE: Yeah.

MICHALE FEE: So it goes up quickly. And then the response in the v1 direction is?

AUDIENCE: Slow, but [INAUDIBLE].

MICHALE FEE: Good. That's it. It's slow, but [AUDIO OUT]. It's amplified in this direction, suppressed in this direction. But the response is fast this way and slow this way. So it traces this out. Now, when you turn the input off, again, it relaxes. v2 relaxes quickly back to zero, and v1 relaxes slowly back to zero. So it kind of traces out this kind of hysteretic loop. It's not really hysteresis.

Then it's exactly mirror image when you put in a negative input. And when you put in h1 positive and v1 negative, it just looks like a mirror image. All right, so any questions about that? Yes, Lena?

AUDIENCE: If there was nothing, like no kind of amplified or [INAUDIBLE], would it just be like a [INAUDIBLE]?

MICHALE FEE: Yeah, so if you took out the recurrent connections, what would what would it look like?

AUDIENCE: An x?

MICHALE FEE: Yeah, the output-- so let's say that you just literally set those to zero. Then the response will be the identity matrix, right? You get the output as a function of input. Let's just go back to the equation. Can always, always get the answer by looking at the equation. Too many animations.

No, it's a very good question. Here we go. There it is right there. So you're asking about-- let's just ask about the steady state response. So we can set dv dt equal to zero. And you're asking, what is v? And you're saying, let's set lambda to zero, right? We're going to set all these diagonal elements to zero. And so now v equals h.

OK, great question. Now, let's go to the case of fully recurrent networks. We've been working with this simplified case of just having neurons have autapses. And the reason we've been doing that is because the answer you get for the autapse kind of captures almost all the intuition that you need to have. What we're going to do is we're going to take a fully recurrent neural network, and we're going to do a mathematical trick that just turns it into an autapse network.

And the answer for the fully recurrent network is just going to be just as simple as what you saw here. All right, so let's do that. Let's take this fully recurrent network. Our weight matrix m now, instead of just having diagonal elements, also has off-diagonal elements.

And I'll say that one of the things that we're going to do today is just consider the simplest case of this fully recurrent network, where the connections are symmetric, where a connection from v1 to v2 is equal to the connection from v2 to v1, all right? We're going to do that because that's the next thing to do to build our intuition, and it's also mathematically simpler than the fully general case, OK?

So we saw how the behavior of this network is very simple if m is diagonal. So what we're going to do is we're going to take this arbitrary matrix m, and we're going to just make it diagonal. So let's do that. So we're going to rewrite our weight matrix m as-- so we're going to rewrite m in this form, where this phi-- sorry, where this lambda is a diagonal matrix.

So we're going to take this network with recurrent connections between different neurons in the network, and we're going to transform it into sort of an equivalent network that just has autapses. So how do we write m in this form, with a rotation matrix times a diagonal matrix times a rotation matrix? We just solve this eigenvalue equation, OK? Does that make sense?

We're just going to do exactly the same thing we did in PCA, where we find the covariance matrix. And we rewrote the covariance matrix like this. Now we're going to take a weight matrix of this recurrent network, and we're going to rewrite it in exactly the same way. So that process is called diagonalizing the weight matrix.

So the elements of lambda here are the eigenvalues of m. And the columns of the phi are the eigenvectors of m. And we're going to use these quantities, these elements, to build a new network that has the same properties as our recurrent network. So let me just show you how we do that.

So remember that what this eigenvalue-- this is an eigenvalue equation written in matrix notation. What this means is this is set of eigenvalues equations that have-- it's a set of n eigenvalue equations like this, where there's one of these for each neuron in the network. OK, so let me just go through that.

OK, so here's the eigenvalue equation. If M is a symmetric matrix, then the eigenvalues are real and phi is a rotation matrix. And the eigenvectors give us an orthogonal basis, all right? So everybody remember this from a few lectures ago?

If M is symmetric-- and this is why we're going to, at this point on, consider just the case where M is symmetric, then the eigenvectors, the columns of that matrix phi, give us an orthogonal set of vectors and their unit vectors. So it satisfies this orthonormal condition. And phi transpose phi is an identity matrix, which means phi is a rotation matrix.

OK, so now what we're going to do is rewrite. The first thing we're going to do to use this trick to rewrite our matrix, our network, is to rewrite the vector of firing rates v in this new basis. What are we going to do? Well take the vector and all we're going to do is to rewrite that vector in this new basis set. We're just going to do a change of basis of our firing rate vector into a new basis set that's given by the columns of phi.

Another way of saying it is that we're going to rotate this firing rate vector v using the phi rotation matrix. So we're going to project v onto each one of those new basis vectors. So there's v in the standard basis. There's our new basis, f1 and f2. We're going to project v onto f1 and f2 and write down that scalar projection, c1 and c2. So we're going to write down the scalar projection of v onto each one of those basis vectors.

So we can write that c sub alpha-- that's the alpha-th component-- is just v dot the alpha-th basis vector. So now we can express v as a linear combination in this new basis. So it's c1 times f1 plus c2 times f2 plus c3-- that's supposed to be a three-- times f3 and so on.

And of course, remember, we're doing all of this because we want to understand the dynamics. So these things are time dependent. So v is v changes in time. We're not going to be changing our basis vectors in time. So if we want to write down a time dependent v, it's really these coefficients that are changing in time, right? Does that make sense?

So we can now write our vector v, our firing rate vector, as a sum of contributions in all these different directions corresponding to the new basis. And each one of those coefficients, c is just the time dependent v projected onto one of those basis vectors. And questions? No? OK.

And remember, we can write that in matrix notation using this formalism that we developed in the lecture on basis sets. So v is just phi c, and c is just phi transpose v. So we're just taking this vector v, and we're rotating it into a new basis set, and we can rotate it back.

All right, so now what we're going to do is we're going to take this v expressed in this new basis set and were going to rewrite our equation in that new basis set. Watch this. This is so cool. All right, you ready? We're going to take this, and we're to plug it into here. So dv dt is phi dc dt. V is just phi c. v is phi c, and h doesn't change.

So now what is that? Do you remember?

AUDIENCE: Phi [INAUDIBLE].

MICHALE FEE: Right. We got phi as the solution to the eigenvalue equation. What was the eigenvalue equation? The eigenvalue equation was m phi equals phi lambda. So the phi here, this rotation matrix, is the solution to this equation, all right? So we're given m, and we're saying we're going to find a phi and a lambda such that we can write m phi is equal to phi lambda.

So when we take that matrix m and we run eig on it in Matlab, Matlab sends us back a phi and a lambda such that this equation is true. So literally, we can take the weight matrix m stick it into Matlab, and get a phi and a lambda such that m phi is equal to phi lambda. So m phi is equal to what? Phi lambda. That becomes this.

Now, all of a sudden, this thing is just going to simplify. So how would we simplify this equation? We can get rid of all of these things, all of these phi's, by doing what? How do you get rid of phi's?

AUDIENCE: Multiply [INAUDIBLE] phi transpose.

MICHALE FEE: You multiply by phi transpose, exactly. So we're going to multiply each term in this equation by phi transpose. So what do you have? Phi transpose phi, phi transpose phi, phi transpose phi. What is phi transpose phi equal to? The identity matrix. Because it's a rotation matrix, phi transpose is just the inverse of phi. So phi inverse phi is just equal to the identity matrix. And all those things disappear. And you're left with this equation-- tau dc dt equals minus c plus lambda c plus h, hf.

And what is hf? hf is just h rotated into the new basis set. So this is the equation for a recurrent network with just autapses, which we just understood. We just wrote down what the solution is, right? And we plotted it for different values of lambda.

So now let's just look at what some of these look like. So we've rewritten our weight matrix in a new basis set. We've rebuilt our network and a new basis set, in a rotated basis set where everything simplifies. So we've taken this complicated network with recurrent connections and we've rewritten it in a new network, where each of these neurons in our new network corresponds to what's called a mode of the fully recurrent network.

So the activities c alpha c1 and c2 of the network modes represent kind of an activity in a linear combination of these neurons. So we're going to go through what that means now. So the first thing I want to do is just calculate what the steady state response is in this neuron. And I'll just do it mathematically, and then I'll show you what it looks like graphically.

So there's our original network equation. We've rewritten it a set of differential equations for the modes of this network. I'm just rewriting this by putting an I here, minus I times c. That's the only change I made here. I just rewrote it like this.

Let's find a steady state. So we're going to set dc dt equal to zero. We're going to ask, what is c in steady state? So we're going to call that c infinity, all right? I minus lambda times c infinity equals phi transpose h. OK, don't panic. It's all going to be very simple in a second. c infinity is just I minus lambda inverse phi transpose h.

But I is diagonal. Lambda is diagonal. So I minus lambda inverse is just the-- it's a diagonal matrix with these elements with one over all those diagonal elements. Now let's calculate v infinity. v infinity is just phi times v infinity. So here, we're multiplying on the left by phi. That's just v infinity. So v infinity is just this.

So what is this? This just says v infinity is some matrix-- it's a rotated stretch matrix-- times the input. So v infinity is just this matrix times h. And now let's look at what that is. v infinity is a matrix times h. We're going to call that g. v infinity is a gain matrix. We're going to think of that as a gain times the input. So it's just a matrix operation on the input.

This matrix has exactly the same eigenvectors as m. And the eigenvalues are just 1 over 1 minus lambda. Hang in there. So what this means is that if an input is parallel to one of the eigenvectors of the weight matrix, that means the output is parallel to the input.

So if the input is in the direction of one of the eigenvectors, v infinity is g times f. But g times f-- f is an eigenvector v. And what that means is that v infinity is parallel to f with a scaling factor 1 over 1 minus lambda. All right? So hang in there. I'm going to show you what this looks like.

So in steady state, the output will be parallel to the input if the input is in the direction of one of the eigenvectors of the network. So if the input is in the direction of one of the eigenvectors of the network, that means you're activating only one mode of the network. And only that one mode responds, and none of the other modes respond.

The response of the network will be in the direction of that input, and it will be amplified or suppressed by this gain factor. And the time constant will also be increased or decreased by that factor. So now let's look at-- so I just kind of whizzed through a bunch of math. Let's look at what this looks like graphically for a few simple cases. And then I think it will become much more clear.

Let's just look at a simple network, where we have two neurons with an excitatory connection from neuron one to neuron two, an excitatory connection from neuron two to neuron one. And we're going to make that weight 0.8. OK, so what is the weight matrix M look like? Just tell me what the entries are for M.

AUDIENCE: Does it not have the autapse?

MICHALE FEE: No, so there's no connection of any of these neurons onto themselves.

AUDIENCE: So you have, like, zeros on the diagonal.

MICHALE FEE: Zeros on the diagonal. Good.

AUDIENCE: All the diagonals.

MICHALE FEE: Good. Like that? Good. Connection from neuron one to itself is zero. The connection from post, pre is row, column. So onto neuron one from neuron two is 0.8. Onto neuron two from neuron one is 0.8. And neuron two onto neuron two is zero.

So now we are just going to diagonalize this weight matrix. We're going to find the eigenvectors and eigenvalues. The eigenvectors are the columns of phi. And the eigenvalues are the diagonal elements of lambda. Let's take a look at what those eigenvectors are. So this vector here is f1. This vector here is another eigenvector, f2.

And how did I get this? How did I get this from this? How would you do that? If I gave you this matrix, how would you find phi?

AUDIENCE: Eig M.

MICHALE FEE: Good, eig of M. Now, remember in the last lecture when we were talking about some simple cases of matrices that are really easy to find the eigenvectors of? If you have a symmetric matrix, where the diagonal elements are equal to each other, the eigenvectors are always 45 degrees here and 45 degrees there. And the eigenvalues are just the diagonal elements plus or minus the off-diagonal elements.

So the eigenvalues here are 0.8 and minus 0.8. All right, so those are the two eigenvectors of this matrix, of this network. Those are the modes of the network. Notice that one of the modes corresponds to neuron one and neuron two firing together. The other mode corresponds to neuron one and neuron two firing with opposite sign-- minus one, one.

So the lambda-- the diagonal elements of the lambda matrix are the eigenvalues. They're 0.8 and minus 0.8, a plus or minus b. Now, this gain factor, what this says is that if I have an input in the direction of f1, the response is going to be amplified by a gain. And remember, we just derived, on the previous slide, that that gain factor is just 1 over 1 minus the eigenvalue for that eigenvector.

In this case, the eigenvalue for mode one is 0.8. So 1 over 1 minus 0.8 is 5. So the gain in this direction is 5. The gain for an input in this direction is 1 over 1 minus negative 0.8, which is 1 over 1.8. Does that makes sense? OK, let's keep going, because I think it will make even more sense once we see how the network responds to its inputs.

So zero input. Now we're going to put an input in the direction of this mode one. And you can see the mode responds a lot. Put a negative input in, it responds a lot. If we put a mode input in this direction or this direction, the response is suppressed by an amount of about 0.5. Because here, the gain is small. Here, the gain is big.

So you see what's happening? This network looks just like an autapse network, but where we've taken this input and output space and just rotated it into a new coordinate system, into this new basis. Yes?

AUDIENCE: Why did it kind of loop around on the one side [INAUDIBLE]?

MICHALE FEE: OK, it's because these things are relaxing exponentially back to zero. And we got a little bit impatient and started the next input before it had quite gone away. OK, good question. It's just that if you really wait for a long time for it to settle, then the movie just takes a long time. But maybe it would be better to do that.

So input this way and this way lead to a large response, because those inputs activate mode one, which has a big gain. Inputs in this direction and this direction have a small response, because they activate mode two, which has small gain. But notice that when you activate mode one-- when you put an input in this direction, it only activates mode one. And it doesn't activate mode two at all.

If you put an input in this direction, then it only activates mode two, and it doesn't activate mode one at all. So it's just like the autapse network, but rotated. So now let's do the case where we have an input that activates both modes. So let's say we put an input in this direction. What does that direction correspond to h up. What is that input mean here in terms of h1 and h2?

Let's say we just put an input-- remember, this is a plot on axes h1 versus h2. So this input vector h corresponds to just putting an input on h2, into this neuron. So you can see that when we put an input in this direction, we're activating-- that input has a projection onto mode one and mode two. So we're activating both modes. You can see that the input h has a projection onto f1 and projection onto f2.

So what you do is-- well, here, I'm just showing you what the steady state response is mathematically. Let me just show you what that looks like. What this says is that if we put an h in this direction, it's going to activate a little bit of mode one with a big gain and a little bit of mode two with a very small gain. And so the steady state response will be the sum of those two. It'll be up here.

So the steady state response to this input in this direction is going to be over here. Why? Because that input activates mode one and mode two both. But the response of mode one is big, and the response of mode two is really small. And so the steady state response is going to be way over here because of the big response, the amplified response of mode two, which is in this direction, OK?

So when we put an input straight up, the response of the network's going to be all the way over here. How is it going to get there? Let's take a look. We're going to put an input-- sorry, that was first in this direction. Now let's see what happens when we put an input in this direction. You can see the response is really big along the mode one direction, in this direction, and it's really small in this direction.

So input up in the upward direction onto just this neuron produces a large response in mode, which is this way, and a very small response in mode two, which is this way. The response in mode two is very fast, because the lambda, the 1 over 1 minus lambda, is small, which makes the time constant faster and the response smaller.

So, again, it's just like the response of the autapse network, but rotated into a new coordinate system. All right, any questions about that? So you can see we basically understood everything we needed to know about recurrent networks just by understanding simple networks with just autapses. And all these more complicated networks are just nothing but rotated versions of the response of a network with just autapses.

Any questions about that? OK, let's do another network now where we have inhibitory connections. That's called mutual inhibition. And let's make that inhibition minus 0.8. The weight matrix is just zeros on the diagonals, because there's no autapse here. And minus 0.8 on the off-diagonals. What are the eigenvectors for this matrix, for this network?

AUDIENCE: The same.

MICHALE FEE: Yeah, because the diagonal elements are equal to each other, and the off-diagonal elements are equal to each other. It's a symmetric network with equal diagonal elements. The eigenvectors are always at 45 degrees. And what are the eigenvalues?

AUDIENCE: [INAUDIBLE]

MICHALE FEE: Well, the two numbers are going to be the same. It's zero plus and minus 0.8, plus and minus negative 0.8, which is just 0.8 and minus 0.8, right? Good. So the eigenvalues are just 0.8 and minus 0.8. But the eigenvalues correspond to different eigenvectors. So now the eigenvalue mode in the 1, 1 direction is now minus 0.8, which means it's suppressing the response in this direction.

And the eigenvalue for the eigenvector in the minus 1, 1 direction is now close to 1, which means that mode has a lot of recurrent feedback. And so its response in this direction is going to be big. It's going to be amplified. So unlike the case where we had positive recurrent synapses, where we had amplification in this direction, now we're going to have amplification in this direction. Does that make sense?

Think of it this way-- if we go back to this network here, you can see that when these two neurons-- when this neuron is active, it tends to activate this neuron. And when this neuron is activate, it tends to activate that neuron. So this network, if you were to activate one of these neurons, it tends to drive the other neuron also. And so the activity of those two neurons likes to go together. When one is big, the other one wants to be big.

And that's why there's a lot of gain in this direction. Does that make sense? With these recurrent excitatory connections, it's hard to make this neuron fire and make that neuron not fire. And that's why the response is suppressed in this direction, OK? With this network, when this neuron is active, it's trying to suppress that neuron. When that neuron has positive firing rate, it's trying to make that neuron have a negative firing rate.

When that neuron is negative, it tries to make that one go positive. And so this network likes to have one firing positive and the other neuron going negative. And so that's what happens. What you find is that if you put an input into the first neuron, it tends to suppress the activity in the second neuron, in v2. If you put neuron into neuron two, it tends to suppress the activity, or make v1 go negative.

So it's, again, exactly like the autapse network, but just, in this case, rotated minus 45 degrees instead of plus 45 degrees, OK? Any questions about that? All right. So now let's talk about how-- yes, Linda?

AUDIENCE: So we just did, those were all symmetric matrices, right?

MICHALE FEE: Yes.

AUDIENCE: So [INAUDIBLE] can we not do this strategy if it's not symmetric?

MICHALE FEE: You can do it for non-symmetric matrices, but non-symmetric matrices start doing all kinds of other cool stuff that is a topic for another day. So symmetric matrices are special in that they have very simple dynamics. They just relax to a steady state solution.

Weight matrices that are not symmetric, or even anti-symmetric, tend to do really cool things like oscillating. And we'll get to that in another lecture, all right? OK, so now let's talk about using recurrent networks to store memories.

So, remember, all of the cases we've just described, all of the networks we've just described, had the properties that the lambdas were less than one. So what we've been looking at are networks for which lambda is less than one and they're symmetric weight matrices. So that was kind of a special case, but it's a good case for building intuition about what goes on.

But now we're going to start branching out into more interesting behavior. So let's take a look at what happens to our equation. This is now our equation different modes of a network. What happens to this equation when lambda is actually equal to one?

So when lambda is equal to one, this term goes to zero, right? So we can just cross this out and rewrite our equation as tau dc dt equals f1 f dot h. So what is this? What does that look like? What's the solution to c for this differential equation? Does this exponentially relax toward a v infinity?

What is v infinity here? It's not even defined. If you set dc dt equal to zero, there's not even a c to solve for, right? So what is this? The derivative of c is just equal to-- if we put in an input that's constant, what is c?

AUDIENCE: [INAUDIBLE]

MICHALE FEE: This is an integrator, right? This c, the solution to this equation, is that c is the integral of this input. c is some initial c plus the integral over time. So if we have an input-- and again, what we're plotting here is the activity of one of the modes of our network, c1, which is a function of the projection of the input along the eigenvector of mode one.

So we're going to plot h, which is just how much the input overlaps with mode one. And as a function of time, let's start at one equals zero. What will this look like? This will just increase linearly. And then what happens? What happens here? Raymundo?

AUDIENCE: R just stays constant.

MICHALE FEE: Good. We've been through that, like, 100 times in this class. Now, what's special about this network is that remember, when lambda was less than one, the network would respond to the input. And then what would it do when we took the input away? It would decay back to zero. But this network does something really special.

This network, you put an input in and then take the input away, this network stays active. It remembers what the input was. Whereas, if you have a network where lambda is less than one, the network very quickly forgets what the input was.

All right, what happens when lambda is greater than one? So when lambda is greater than one, this term is now-- this thing inside the parentheses is negative, multiplied by a negative number. This whole coefficient in front of the c1 becomes positive. So we're just going to write it as lambda minus one. And so this because positive. And what does that solution look like? Does anyone know what that looks like? dc dt equals a positive number times c.

Nobody? Are we all just sleepy? What happens? So if this is negative, if this coefficient were negative, dc-- if c is positive, then dc dt is negative, and it relaxes to zero, right? Lets think about this for a minute. What happens if this quantity is positive? So if c is positive-- cover that up.

If this is positive and c is positive, then dc dt is positive. So that means if c is positive, it just keeps getting bigger, right? And so what happens is you get exponential growth. So if we now take an input and we put it into this network, where lambda is greater than one, you get exponential growth.

And now what happens when you turn that input off? Does it go away? What happens? draw with their hand what happens here. So just look at the equation. Again, h dot f1 is zero here, so that's gone. This is positive. c is positive. So what is dc dt? Good. It's positive. And so what is--

AUDIENCE: [INAUDIBLE]

MICHALE FEE: It keeps growing. So you can see that this network also remembers that it had input. So this network also has a memory. So anytime you have lambda less than one the network just-- as soon as the input goes away, the network activity goes to zero, and it just completely forgets that it ever had input. Whereas, as long as lambda is equal to or greater than one, then this network remembers that it had input.

So if lambda is less than one, then the network relaxes exponentially back to zero after the input goes away. If you have lambda equal to one, you have an integrator, and the network activity persists after the input goes away. And if you have exponential growth, the network activity also persists after the input goes away.

And so that right there is one of the best models for short-term memory in the brain. The idea that you have neurons that get input, become activated, and then hold that memory by reactivating themselves and holding their own activity high through recurrent excitation. But that excitation has to be big enough to either just barely maintain the activity or continue increasing their activity.

OK, now, that's not necessarily such a great model for a memory, right? Because we can't have neurons whose activity is exploding exponentially, right? So that's not so great. But it is quite commonly thought that in neural networks involved in memory, the lambda is actually greater than one.

And how would we rescue this situation? How would we save our network from having neurons that blow up exponentially? Well, remember, this was the solution for a network with linear neurons. But neurons in the brain are not really linear, are they? They have firing rates that saturate. At higher inputs, firing rates tend [AUDIO OUT]. Why? Because sodium channels become inactivated, and the neurons can't respond that fast, right?

All right, this I've already said. So we use what are called saturating non-linearities. So it's very common to write down models in which we can still have neurons that are-- we can still have them approximately linear. So it's quite often to have neurons that are linear for small [INAUDIBLE]. They can go plus and minus, but they saturate on the plus side or the minus.

So now you can have an input to a neuron that activates the neuron. You can see what happens is you start activating this neuron. It keeps activating itself, even as the input goes away. But now, what happens is that activity starts getting up into the regime where the neuron can't fire any faster. And so the activity becomes stable at some high value of firing. Does that make sense?

And this kind of neuron, for example, can remember a plus input, or it can remember a minus input. Does that make sense? So that's how we can build a simple network with a neuron that can remember its previous inputs with a lambda that's greater than one. And this right here, that basic thing, is one of the models for how the hippocampus stores memories, that you have hippocampal neurons that connect to each other with a lot of recurrent connections [AUDIO OUT] in the hippocampus has a lot of recurrent connections.

And the idea is that those neurons activate each other, but then those neurons saturate so they can't fire anymore, and now you can have a stable memory of some prior input. And I think we should stop there. But there are other very interesting topics that we're going to get to on how these kind of networks can also make decisions and how they can store continuous memories-- not just discrete memories, plus or minus, on or off, but can store a value for a long period of time using this integrator. OK, so we'll stop there.