14. Rate Models and Perceptrons

• Download English-US transcript (PDF)

MICHALE FEE: So for the next few lectures, we're going to be looking at developing methods of studying the computational properties of networks of neurons. This is the outline for the next few lectures.

Today we are going to introduce a method of studying networks called a rate model where we basically replace spike trains with firing rates in order to develop simple mathematical descriptions of neural networks. And we're going to start by introducing that technique to the problem of studying feed-forward neural networks. And we'll introduce the idea of perceptrons trance as a method of developing networks that can classify their inputs.

Then in the next lecture, we're going to turn to largely describing mathematical tools based on matrix operations and the idea of basis sets. Matrix operations are very important for studying neural networks. But they're also a fundamental tool for analyzing data and doing things like reducing the dimensionality of high dimensional data sets, including methods such as principal components analysis. So it's a very powerful set of methods that apply both to studying the brain and to analyzing the data that we get when we study the brain.

And then finally we'll turn to a few lectures that focus on recurrent neural networks. These are networks where the neurons connect to each other densely in a recurrent way, meaning a neuron will connect to another neuron. And that neuron will connect back to the first neuron.

And networks that have that property have very interesting computational abilities. And we're going to study that in the context of line attractors and short-term memory and hopfield networks.

So for today, the plan is to develop the rate model. We're going to show how we can build receptive fields with feed forward networks that we've described with the rate model. We're going to take a little detour and describe vector notation and vector algebra, which is very important for these models, and also for building up to the matrix methods that we'll talk about in the next lecture. Again, we'll talk about neural networks for classification and introduce the idea of a perceptron. So that's for today.

So I've already talked about most of this. Why is it that we want to develop a simplified mathematical model of neurons that we can study analytically? Well, the reason is that we can really develop our intuition about how networks work.

And that intuition applies not just to the very simplified mathematical model that we're developing, but also applies more broadly to real networks with real neurons that actually generate spikes and interact with each other by the more complex biophysical mechanisms that are going on in the brain.

So a good example of this is how we simplified the detailed spiking neurons of the Hodgkin-Huxley model and approximate that as an integrate and fire model, which captures a lot of the properties of real neurons. Simplifies it enough to develop an intuition, but captures a lot of the important properties of real neural circuits.

All right, so let's start by developing the basic idea of a rate model. Let's start with two neurons. We have an input neuron and an output neuron. The input neuron has some firing rate given by u. And the output neuron has some firing rate given by v. So we're going to essentially ignore the times of the spikes and describe the inputs and outputs of this network just with firing rates. You can think of the rate as just having units have spikes per second, for example.

Those neurons, the input neuron and the output neuron, are connected to each other by a synapse. And we're going to replace all of the complex structure of synapses, vesicle release, neurotransmitter receptors, long-term depression and paired spike facilitation and depression, all that stuff we're just going to ignore. And we're going to replace that synapse with a synaptic weight w.

Just to give you the simplest intuition of how a rate model works, there are models where we can just treat the firing rate of the output neuron, for example, as linear in its input. And we can simplify this even to the point where we can describe the firing rate of the output neuron as the synaptic weight w times the firing rate of the input neuron. So that's just to give you a flavor of where we're heading.

And I'm going to justify how we can do this and/or why we can do this. And then we're going to build this up from the case of one input neuron and one output neuron to the case where we can have many input neurons and many output neurons.

So how do we justify going from spikes to firing rates? So remember that the response of a real output neuron, a real neuron, to a single spike at its input, is some change in the postsynaptic conductance that follows an input spike. And in our model of a synapse, we described that the input spike produces a transient increase in the synaptic conductance. And that synaptic conductance we modeled as a simple step increase in the conductance followed by an exponential decay as the neurotransmitter gradually unbinds from the neurotransmitter receptors.

So we have a transient change in the synaptic conductance. That's just a maximum conductance times an exponential decay. Now remember that we wrote down the postsynaptic-- we can write down the postsynaptic current that results from this synaptic input as the synaptic conductance times v minus e synapse, the synaptic reversal potential.

In moving forward in this model, we're not going to worry about synaptic saturation. So we're just going to imagine that the synaptic current is just proportional to the synaptic conductance.

So now we can write the conductance as just some weight times a kernel that is just some kernel of unit area. So what we've done here is we've just taken the synaptic current and we've written it as a constant, a synaptic weight, times an exponentially decaying kernel of area, area 1.

So now if we have a train of spikes at the input instead of a single spike, we can write down that train of spikes, the spike train, as a sum of delta functions where the spike times are t sub i. And if you want to plot the synaptic current as a function of time, you would just take that spike train input and do what with that linear kernel? We would convolve it, right?

So we would take that spike train, convolve it with that little exponential kernel. And that would give us the synaptic current that results from that spike train.

So let's think for a moment about what this quantity is right here. What is k, this k which is a little kernel that has an exponential step, and then an exponential decay? What do you get when you convolve that kind of smooth kernel with this spike train here? What does that look like?

We did that at one point when we were in class when we were talking about how you would estimate something from a spike train. What is that? What is that quantity right there? It's sort of a smoothed version of a spike train, which is how you would calculate what, Habiba?

AUDIENCE: Is it a window for the spike train?

MICHALE FEE: Yeah. It's windowed, but what is it that you are calculating when you take a spike train and you convolve it with some smooth window?

AUDIENCE: Low-pass window?

MICHALE FEE: It's like a low-pass version of the spike train. And remember in the lecture on firing rates, we talked about how that's a good way to get a time-dependent estimate of the firing rate of a neuron. We take the spike train and just convolve it with a smooth window.

And if the area of that smooth window is 1, then what we're doing is we're estimating the firing rate of the neuron as a function of time. Does that make sense? Yes?

AUDIENCE: So k is just a kernel?

MICHALE FEE: k is just is smooth kernel that happens to have this exponential shape.

AUDIENCE: Is it like [INAUDIBLE]

MICHALE FEE: Well, that's our model for how a synapse-- basically, what I'm saying is that when you take a spike train and put it through a synapse, what comes out the other end is a smoothed version of the spike train.

AUDIENCE: OK.

MICHALE FEE: That's all this is saying.

AUDIENCE: OK. [INAUDIBLE] they have this area or quantity?

MICHALE FEE: Yep. If k has-- you remember that if k has an area 1, then when you convolve evolve that kernel with the spike train, you get a number that has units of spikes per second. And that quantity is an estimate of the local firing rate of the neuron. Does that make sense?

So basically, we can take this spike train, and by convolve it with a smooth window, we can estimate the number of spikes per second in that window. So what do we have here? We have that the current is just a constant times an estimate of the firing rate at that time. If k is a kernel, a smooth kernel with an area normalized to 1, then this quantity is just an estimate of the firing rate.

So let's take a look at that. So here I have just made a sample spike train with a bunch of spikes that look like they're increasing in firing rate and decreasing in firing rate. If we take that spike train and convolve it with this kernel, you can see that you get this sort of broad bump that looks like it gets higher in the middle where the firing rate is higher. And it's lower at the edges where the firing rate is lower.

So the point is that you can take a spike train and put it into a neuron. The response of the neuron is a smooth low-pass version of the rate of this input spike train. And so you can think about writing down the input to this neuron as a weight times the firing rate of the input.

So that was a way of writing down the input to this output neuron from the input neuron, the current input. Now what is the firing rate of the output neuron in response to that current injection? So that's what we're going to ask next.

And you can remember that when we talked about the integrate and fire model, we saw that neurons in the approximation of large inputs have firing rate as a function of current that looks like this. It's zero for inputs below the threshold current. For input currents that aren't large enough to drive the neuron to threshold, the neuron doesn't spike at all.

And then above some threshold, the neuron fires approximately linearly at higher input currents. So the way that we think about this is that the input on is spiking at some rate. It goes through a synapse. That synapse smooths the input and produces some current in the postsynaptic neuron that's proportional approximately to the firing rate of the input neuron. And the output neuron has some output firing rate that's some function of the input current.

So we can write down the firing rate of our output neuron, v. It's just equal to some function of the input current, which is just some function of w times the firing rate of the input neuron. And that right there is the basic equation of the rate model. The output firing rate is some function of a weight times the firing rate of the input neuron.

And everything else about the rate model is just different rate models have different numbers of input neurons where we have more than one contribution to the input current. They can have many output neurons. They can have different FI curves for the output neurons.

Some of them are non-linear like this. Some of them are linear. And we're going to come back and talk about the function of different FI curves and why different FYI curves are useful. Any questions about this? That's the basic idea. All right, good.

So let's take one particularly simple version of the rate model called a linear rate model. And the linear rate model has a particular FI curve. That FI curve says that the firing rate of the neuron is linear in the input current.

Now why is this a really weird model of a neuron? What's fundamentally non-biological about this?

AUDIENCE: Negative firing rate.

MICHALE FEE: I'm hearing a bunch of right answers at the same time.

AUDIENCE: Negative firing rate.

MICHALE FEE: This neuron is allowed to fire at a negative firing rate if the input current is negative. That's a pretty crazy thing to do. Why do you think we would want to do that?

AUDIENCE: [INAUDIBLE]?

MICHALE FEE: Well, no actually we do. So you can have inhibitory inputs that produce outward currents that hyperpolarize the neuron. Any thoughts about that?

It turns out that as soon as you have your output neurons have this kind of FI curve, a linear FI curve, then the math becomes super simple. You can write down very complex networks of neurons with a bunch of linear differential equations. And it becomes very easy to write down what the solution is to how a network behaves as a function of its inputs.

And we're going to spend a lot of time working with network models that have linear FI curves because you can develop a lot of intuition about how networks behave by using models like this. As soon as you have models like this, you can't solve the behavior of the network analytically. You have to do everything on the computer. And it becomes very hard to derive general solutions for how things behave. So we're going to use this model a lot.

And in this case again, for the case of this two-neuron network where we have one output neuron that receives a synaptic input from an input neuron, the firing rate of the output neuron is just w, the synaptic weight times the firing rate of the input neuron. And we're going to come back to non-linear neurons because that non-linearity actually does really important things. And we're going to talk about what that does.

So now let's look at the case where our output neuron has not just one input but actually many inputs from a bunch of input neurons. So here we have what we call an input layer, a layer of neurons in the input layer. Each one of those neurons has a firing rate-- u1, u2, u3, u4, u5.

Each of those neurons sends a synapse onto our output neuron. Each one of those synapses has a synaptic weight. This weight is w1. And that's w2, w3, w4, and w5.

Now you can see that the total input, the total current, to this output neuron is just going to be a sum of the inputs from each of the input neurons. The total input is just a sum of the inputs from each of the input neuron. So the synaptic current-- total synaptic current into this neuron is w1 times u1, plus w2 times u2, plus w3 times u3, plus all the rest.

So the response of our linear neuron, the firing rate of our linear neuron, is just a sum over all of those inputs. So again, in this case, we're going to say that the total input current to this neuron is the sum over this. But then because this is a linear neuron, the firing rate is just equal to that current input. Does that make sense?

So you can see that this description of the firing rate of the output neuron is a sum over all of those contributions. It turns out that this actually can be written in a much more compact way in vector notation. What does that look like? Does anyone know in vector notation what that looks like?

AUDIENCE: Dot product.

MICHALE FEE: That's a dot product. That's right. So in general, it's much easier to write these responses in vector notation. And so I'm just going to walk you through some basics of vector notation for those of you who might need a few minutes of reminder.

Actually before we get to the vector notation, I just want to describe how we can use a simple network like this to build a receptive field. So you remember that when we were talking about receptive fields of neurons, we described how a neuron can have a maximal response to a particular pattern of input. So let's say we have a neuron that's sensitive to visual inputs. And as a function of one dimension, let's say along the retina, this neuron has a big response if light comes in central field, some inhibitory responsive light comes in outside of that central lobe.

Well, it turns out that a very simple way to build neurons that have receptive fields like this, for example, is to have an input layer that projects to this neuron that has this receptive field and has a pattern of synaptic inputs that corresponds to that pattern in the field. So you can see that if this neuron-- so let's say these are neurons in the retina, let's say retinal ganglion cells, and this neuron is in the thalamus, we can build a thalamic neuron that has a center-surround receptive field like this by having let's say this neuron has a strong positive excitatory synaptic weight onto our output neuron.

So you can see that if you have light here that corresponds to this neuron having a high firing rate, that neuron is very effective at driving the output neuron. And so the output neuron has a positive component of its receptor field right there in the middle.

Now if this neuron here, which is in this part of the retina, if that neuron has a negative weight onto the output neuron, then light coming in here driving this neuron will inhibit the output neuron. So if you have a pattern of weights that looks like this, 0 minus 1, 2 minus 1, 0, that this neuron will have a receptive field that looks like that as a function of its inputs.

So that's a on-dimensional example. And you can see that you write down the output here as a weighted sum of each one of those inputs.

This also works for two dimensional receptive fields. For example, if we have input from the retina that looks like this where we have-- I guess this was excitatory here in the center, inhibitory around, you can make a neuron that has a two-dimensional receptor field like this by having inputs to this neuron from all of those different regions of the visual field that have different weights corresponding to positive in the center. So neurons in the positive synaptic weights under the output neuron. And neurons around the edges have negative synaptic weights.

So we can build any receptive field we want into a neuron by just plugging in-- by putting in the right set of synaptic weights. Yes?

AUDIENCE: So would you rule out [INAUDIBLE]

MICHALE FEE: So in real life, I assume you mean in the brain?

AUDIENCE: Yeah.

MICHALE FEE: So in the brain, we don't really know how these weights are built. So one idea is that there are rules that control the development of these circuits, let's say, connections of bipolar cells in the retina to retinal ganglion cells that control how these weights are determined to be positive or negative.

Negative weights are implemented by bipolar cells connected to amacrine cells, which are inhibitory, and then connect to the retinal ganglion. So there's a whole circuit that gets built in the retina that controls whether these weights are positive or negative. And those can be programmed by genetic developmental programs. They can also be controlled by experience with visual stimuli.

So there's a lot we don't understand about how these weights are controlled or set up or programmed. But the way we think about how receptive fields of these neurons emerge is by controlling the weight of those synaptic input. That's the message here-- that receptive fields emerge from the pattern of weights from an input layer onto an output layer.

AUDIENCE: [INAUDIBLE] how many [INAUDIBLE]

MICHALE FEE: If you're going to build a model, let's say, of the retina. So it just depends on how realistic you want it to be. If you wanted to make a model of a retinal ganglion cell, you could try to build a model that has as many bipolar neurons as are actually in the receptive field of that retinal ganglion cell.

Or you could make a simplified model that only has 10 or 100 neurons. Depends on what you want to study. All right any other questions?

And again, even for these more complex models, you can still write down a simple rate model formulation of the firing rate of the output neuron. It's just a weighted sum of the input firing rate.

So each neuron in the input layer fires at some rate. It has a weight w. To get the contribution of this neuron to the firing rate of the output neuron, you just take that input firing rate times the synaptic weight, and add that up then for all the input layer neurons.

So as I said, we've been describing the response of our linear neuron as this weighted sum. And that's a little bit cumbersome to carry around. So we're going to start using vector notation and matrix notation to describe networks. It's just much more compact.

So we're going to take a little detour, talk about vectors. So a vector is just a collection of numbers. The number of numbers is called the dimensionality of the vector. If a vector has only two numbers, then we can just plot that vector in a plane.

So for a 2D vector, if that vector has two components, x1 and x2, then we can plot that vector in that space of x1 and x2, put the origin at zero. In this case, the vector has two vector components or elements, x1 and x2.

And in two dimensions we describe that as spaces, as R2, the space of two real numbers. We can write down that vector as a row in row vector notation. So x is x1, x2. We can write it as a column vector, x1, x2, organized on top of each other, like this.

Vector sums are very simple. So if you have two vectors, x and y, you can write down the sum of x and y is x plus y. That's called the resultant. X plus y it can be written like this in column vector notation.

You can see that the sum of x and y is just an element by element sum of the vector elements. It's called element by element addition. Let's look at vector product.

So there are multiple ways of taking the product of two vectors. There's an element by element product, an inner product, an outer product that we'll cover in later lectures. And also, something called the cross product that's very common in physics.

But I have not yet seen the application of a cross product to neuroscience. If anybody can find one of those, I'll give extra credit.

Element by element product is called a Hadamard product. So x times y is just the element-by-element product of the elements in the two vectors. In Matlab, that element-by-element product you compute by x dot star y.

Inner product or dot product looks like this. So if we have two column vectors, the dot product of x and y is the sum of the element-by-element products. So x dot y is just x1 times y1 plus x2 times y2, and so on, plus xn times yn. And that's that sum that we saw earlier in our feed forward network.

OK. So notice that the dot product is a scalar. It's a single number. It's no longer a vector. Products have some nice properties. They're commutative.

So x.y is equal to y.x. They're distributive so that vector w dotted into the sum of two vectors is just the sum of the two separate dot products. So w dot x plus y is just w.x, w.y. And it's also linear.

So if you have a x dot y that is equal to a times the quantity x.y. So if you have vector x and y dotted into each other, if you make one of those vectors twice as long, then the dot product is just twice as big.

A little bit more about inner products. So we can also write down the inner product in matrix notation. So x.y is a matrix product of a row vector.

Column vector, you remember how to multiply two matrices. You multiply the elements of each row times the elements of each column. So you can see that this in matrix notation is just the dot product of those two vectors.

In matrix notation, this is a 1 by n matrix. This is an n by 1. So 1 row by n columns, times n rows by 1 column. And that is equal to a 1 by 1 matrix, which is just a scalar.

All right, in Matlab, let me just show you how to write down these components. So in this case, x is a column vector, a 1 by 3 column vector. y is a 1 by 3 column vector. You can calculate those vectors like this. And z is x transpose times y. And so that's how you can write down the dot product of two vectors.

What is the dot product of a vector with itself? It's the square magnitude of the vector. So x is just the norm or magnitude of the vector. And you can see that the norm of the vector is just-- you can think about this as being analogous to the Pythagorean theorem. The length of one side of a triangle is just the sum of the squares of all the sides, the square root of that.

So a unit vector is a vector that has length 1. So a unit vector by definition has a magnitude of 1, which means its dot product with itself is 1.

We can turn any vector into a unit vector by just taking that vector, dividing by its norm. I'm going to always use this notation with this little caret symbol to represent a unit vector. So if you see a vector with that little hat on it, that means it's a unit vector.

You can express any vector as a product of a scalar, a length, times a unit vector in that direction. We can find the projection or component of any vector in the direction of this unit vector as follows. So if we have a unit vector x, we can find the projection of a vector y onto that unit vector x.

How do we do that? We just find the normal projection of that vector. That distance right there is called the scalar projection of y onto x. If you write down the length of the vector y, the norm of the vector y in the angle between y and x, then the dot product y.x is just equal to the magnitude of y times the cosine of the angle between the two vectors. Just simple trigonometry.

We can also define what's called the vector projection of y onto x as follows. So we just draw that same picture. So we can find the projection of y onto x and add that as a vector.

And that's just this scalar projection of y onto x times a unit vector in the x direction. So x actually is a unit vector in this example. So this vector projection of y to x is just defined as y dot x times x. Any questions about that?

I'm guessing most of you have seen all of this stuff already. But we're going to be using these things a lot. So I just want to make sure that we're all on the same page. And that's just a scalar times a unit vector.

Let me just give you a little bit of intuition about dot products here. So a dot product is related to the cosine of the angle between two vectors, as we talked about before. The dot product is just magnitude of x times the magnitude of y times the cosine of the angle between them.

So the cosine of the angle between two vectors is just the dot product divided by the product of the magnitude of each of the two vectors. So if x and y are unit vectors, the cosine of the angle between them is just the dot product of the unit vectors. So again, if x and y are unit vectors, then that dot product is just the cosine of the angle.

Orthogonality. So two vectors are orthogonal, are perpendicular, if and only if their dot product is 0. So if we have two vectors x and y, they are orthogonal if the angle between them is 90 degrees. x.y is just proportional to the cosine of the angle. Cosine of 90 degrees is zero.

So if two vectors are orthogonal, then their dot product will be zero. If their dot product is zero, then they're orthogonal with each other. And using the notation we just developed, the vector projection of y along x is the zero vector, if those two vectors are orthogonal.

There is an intuition that one can think about in terms of the relation between dot product and correlation. So the dot product is related to the statistical correlation between the elements of those two vectors. So if you have a vector x and y, you can write down the cosine of the angle between those two vectors, again, as x.y over the product of the norms.

And if you write that out as sums, you can see that this is just the sum of the element-by-element products-- that's the dot product-- divided by the norm of x and the norm of y. And if you have taken a statistics class, you will recognize that as just the Pearson correlation of a set of numbers x and a set of numbers y. The dot product is closely related to the correlation between two sets of numbers.

One other thing that I want to point out coming back to the idea of using this feed forward network as a way of receptive field, you can see that the response of a neuron in this model is just the dot product of the stimulus vector u. The vector of input firing rates represents the stimulus, the dot product of the stimulus vector u with the weight vector w.

So the firing rate of the output neuron is just w.u. So you can see that what this means is that the firing rate of the output neuron will be high if there is a high degree of overlap between the input, the pattern of the input, and the pattern of synaptic weights from the input layer to the output neuron.

We can see that w.u is big when w and u are parallel, are highly correlated, which means a neuron fires a lot when the stimulus matches the pattern of those synaptic weights.

Now, so you can see that for a given amount of power in the stimulus-- so the power is just the square magnitude of u-- the stimulus that has the best overlap with the receptive field, where cosine of that angle is 1, produces the largest response.

And so we now have actually a definition of the optimal stimulus of a neuron in terms of the pattern of synaptic weights. In other words, the optimal stimulus is one that's essentially proportional to the weight matrix. Any questions so far?

All right, so now let's turn to the question of how we use neural networks to do some interesting computation. So classification is a very important computation that neural networks do in the brain and actually in the application of neural networks for technology.

So what does classification mean? So how does the brain-- how does a neural circuit decide how a particular input-- let's say that it looks like you might eat it. How do we decide-- how do the neural circuits in our brain decide whether that thing that we're seeing is something edible or something that will make us sick based on past experience?

If we see something that looks like an animal or a dog, how do we know whether that's a friendly puppy or a or a wolf? So these are classification problems.

And feed forward circuits actually can be very good at classification. In fact, recent advances in training neural networks have actually resulted in feed forward neural networks that actually approach human performance in terms of their ability to make decisions like this.

All right. So basically, a feed forward circuit that does classification like this typically has an input layer. It has a bunch of inputs that represent sensory stimulus. And a bunch of output neurons that represent different categorizations of that input stimulus.

So you can have a retinal input here. Going to other layers of a network. And then at the end of that, you can have a network that starts firing when that input was a dog, or starts firing another neuron that starts firing when that input was a cat, or something else.

Now in general, classification networks that have one input layer and one output layer can't do this problem. You can't take a visual input and have connections to another layer of neurons that just light up when the picture that the network is seeing is a dog. Another neuron lights up when it's a cat.

Generally, there are many layers of neurons in between. But today, we're going to talk about a very simplified version of the classification problem and build up to the sorts of networks that can actually do those more complex problems.

So I just want to point out that the obviously our brains are very good at recognizing things. We do this all the time. There are hundreds of objects in every visual scene. And we're able to recognize every one of those objects.

But it turns out that there are individual neurons-- so in this case, I alluded to the idea that there are individual nones in this network that light up when the sensory input is a dog or light up when the input is an elephant. And it turns out that that's actually true in the brain.

So there have recently been studies where it's been possible to record in parts of the human brain in patients that are undergoing brain surgery for the treatment of epilepsy or tumors or things like that where you have to go in and find parts of the brain that are defective, find parts of the brain that are healthy. So when you do a surgery, you can be very careful to just do surgery on the damaged parts of the brain and not impact parts of the brain that are healthy.

So there are cases now, more and more commonly, where neuroscientists can work with neurosurgeons to actually record from neurons in the brain in these patients who are in preparation for surgery. And so it's been possible to record from neurons in the brain.

This was a study from Itzhak Frieds lab at UCLA. And this shows recording in the right anterior hippocampus. And what this lab did was to find neurons. So these were electrodes implanted in the brain. And then they basically take these patients and they show them thousands of pictures and look at how their brains respond to different visual inputs.

So let me just show you what you're looking at. These are just different pictures of celebrities. There's Luke Skywalker, Mother Teresa, and some others. This paper is getting old enough that you may not recognize most of these people.

But if you record from neurons in the brain, you can see that-- so what do you see here? I think that's Oprah. The image is flashed up on the screen for about a second.

You record this neuron spiking. Here you see a couple spikes. Here's when the image was actually presented. And here's where the image was turned off. You can see different trials.

So this neuron actually had a little bit of a response right there shortly after the stimulus was turned on. But you can see there's not that much response in these neurons.

But when they flashed a different stimulus-- anybody know who that is? That's Halle Berry. Look at this neuron. Every time you show this picture, that neuron fires off a couple spikes very precisely.

If you look at the histogram, these are histograms underneath showing as a function of time relative to the onset of the stimulus, you could see that this neuron very reliably spikes. There's a different picture of Halle Berry. Neuron spikes. Different picture, neuron spikes. Another picture, neuron spikes.

A line drawing of Halle Berry, the neuron spikes. Catwoman, the neuron spikes. The text, Halle Berry, the neuron spikes. It's amazing.

So this group got a lot of press for this because they also found Jennifer Aniston neurons. They found other celebrities. This is like some celebrity part of the brain.

No, it's actually a part of the brain where you have neurons that have very sparse responses to a wide range of things. But they're extremely specific to particular people or categories or objects.

And it actually is consistent with this old notion of what's called the grandmother cell. So back before people were able to record in the human brain like this, there was speculation that there might be neurons in the brain that are so specific for particular things, that there might be one neuron in your brain that responds when you see your grandmother.

And so it turns out it's actually true. There are neurons in your brain that respond very specifically to particular concepts or people or things. So the question of how these kinds of neurons acquire their responses is really cool and interesting.

So that leads us to the idea of perceptrons. Perceptron is the simplest notion of how you can have a neuron that responds to a particular thing that detects a particular thing and responds when it sees it and doesn't respond when it doesn't.

So let's start with the simplest notion of a perceptron. So how do we make a neuron that fires when it sees something-- let's say a dog-- and doesn't fire when there is no dog?

So in order to think about this a little bit more, so we can begin thinking about this in the case where we have a single neuron input and a single output neuron. So if we have a single input neuron, then what comes in has to be-- it can't be an image right? An image is a high dimensional thing that has many thousands of pixels. So you can't write that down as a simple model with a single input neuron and a single output neuron.

So you need to do this classification problem in one-dimension. So we can imagine that we have an input neuron that comes from, let's say, some set of numbers-- I'll make up a story here-- some set of neurons that measure the dogginess of an input.

So let's say that we have a single input that fires like crazy when it sees this cute little guy here. And fires at a negative rate when it sees that thing, which doesn't look much like a dog. So we have a single input that's a measure of dogginess.

And now let's say that we take this dogginess detector and we point it around the world. And we walk around outside with our dogginess detector and we make a bunch of measurements. So we're going to see something that looks like this.

We're going to see a lot of measurements, a lot of observations down here that are close to zero dogginess. And we're going to see a bump of things up here that correspond to dogs. Whenever we point our dogginess detector at a dog, it's going to give us a measurement up here. And we're going to get a bunch of those.

And those things correspond to dogs. So we need to build a network that fires when the input is up here and doesn't fire when the input is down there. So how do we do that?

So the central feature of classification is this notion of binariness, of decision-making. That it fires when you see a dog and doesn't fire when you don't see a dog.

So there exists a classification boundary in this stimulus space. You can imagine that there's some points along this dimension above which you'll say that that input is a dog, below which you say that it isn't.

And we can imagine that that classification boundary is right here. It's a particular number. It's a particular value of our dogginess detector, above which we're going to call it a dog, and below which we're going to call it something else.

How do we make this neuron respond by firing when there's a dog and not firing when there's no dog? Can we use a linear neuron? Can we use one of our linear neurons that we just talked about before?

We can't do that because a linear neuron will always fire more the bigger the input is. And it will fire less if the dogginess is 0. And it will even fire more negatively if the dogginess input is negative.

So a linear neuron is terrible for actually making any decisions. Linear neurons always go, ah, well, maybe that's a dog. Not really. There's no decisions.

So in order to have a decision, we need to have a particular kind of neuron. And that kind of neuron uses something very natural. In biophysics, it's the spike threshold of neurons. Neurons only fire when the input is above some threshold, generally. There are neurons that are tonically active. But let's not worry about those.

So many neurons only fire when the input is above some threshold. So for decision-making and classification, a commonly used kind of neuron takes this idea to an extreme. So for perceptrons, we're going to use a simplified model of a neuron that's particularly good at making decisions. There's no if, ands, or buts about it. It's either off or on. It's called a binary unit.

And a binary unit uses what's called a step function for its FI curve. That step function is 0-- the output is 0 if the input is zero or below. And the output is 1 if the input is above 0.

We can use that step function to create a neuron that responds when the input is above any threshold we want. So we can write down the output firing rate is this function, a step function-- that function of a quantity that's given by w times u, the synaptic weight times the input firing rate, minus that threshold.

So you can see if w times u, which is the input synaptic current, if that synaptic current is above theta, then this argument to this function is greater than 0, then the neuron spikes. If this argument is negative, then the neuron doesn't spike.

So by changing theta, we can put that decision boundary anywhere we want. Does that make sense?

Usually the way we do this is we pick a theta. We say our neuron has a theta of 1. And then we do everything else-- we do everything else we're going to do with this network with a theta.

So what I'm going to talk about today are just two cases. Where theta is a fixed number that's non-zero, or theta that's a fixed number that is equal to 0. So we're going to talk about those two cases.

So the neuron fires when the input w u is greater than theta. And it doesn't fire when it's less. So now the output neuron fires whenever the input neuron has a firing rate greater than this decision boundary.

So the decision boundary, the u threshold, is equal to theta divided by w. Does that make sense? U threshold is the neuron fires when u is greater than theta divided by w.

So the way we learn, the way this network learns to fire when that u is above this classification boundary is simply by changing the weight. Does that make sense?

So we're going to learn the weight such that this network fires whenever the input says there's a dog. And it doesn't fire whenever the input says there's no dog.

So let's see what happens when w is really small. If w is really small, then what happens is all of these-- remember, this is the input. That's that the dogginess detector.

If w is really small, then all these inputs get collapsed to a small input current into our output neuron. Does that make sense? So all those different inputs, dogs and non-dogs, gets multiplied by a small number. And all those inputs are close to 0.

And if all those inputs are close to 0, they're all below the threshold for making this neuron spike. So this network is not good for detecting dogs because it says it never fires, whether the input is a dog or a non-dog.

Now what happens if w is too big? If w is really big, then this range of dogginess values gets multiplied by a big number. And you can see that a bunch of non-dogs make the neuron fire. Does that make sense?

So now this one fires for dogs plus doggie-ish looking things, which, I don't know, maybe it'll fire when it sees a cat. That's terrible.

So you have to choose w to make this classification network function properly. Does that make sense? And if you choose w just right, then that classification boundary lands right on the threshold of the neuron. And now the neuron spikes whenever there is a dog. And it doesn't spike whenever there's not a dog.

So what's the message here? The message is we can have a neuron that has this binary threshold. And what we can do is simply by changing the weight, we can make that threshold land anywhere on this space of inputs.

And we can actually use the error to set the weight. So let's say that we made errors here. We classify dogs as non-dogs because the neuron didn't fire. You can see that this was the case when w was too small.

So if you classify dogs as non-dogs, then you need to make w bigger. And if you classify non-dogs as dogs, you need to make w smaller. And by measuring what kind of errors you make, you can actually fix the weights to get to the right answer.

So this is a method called supervised learning where you set w randomly. You take a guess. And then you look at the mistakes you make. And you use those mistakes to fix the w.

In other words, you just look at the world and you say, oh, that's a dog. And then your mom says, no, that's not a dog, that's something else. And you adjust your weights.

I think that was the example I just gave. You're going to make that w smaller. In another case, you'll make the other kind of mistake and you'll fix the weights.

So this is called a perceptron. And the way you learn the weights in a perceptron is you just classify things and you figure out what kind of mistake you made and you use that to adjust the weights. So that's the basic idea of a perceptron and perceptron learning. And there's a lot of mathematical formalism that goes into how that learning happens. And we're going to get to that in more detail in the next lecture.

But before we do that, I want to go from having a one-dimensional case. So here we had a one-dimensional network that was just operating on dogginess. And then we have a single neuron that says, was that a dog or not.

But in general, you're not classifying things based on one input. Like for example when you have to identify a dog, you have a whole image of something. And you have to classify that based on an image.

So let's go from the one-dimensional case to a two-dimensional case. So the classification isn't done on one-dimension, but it's based on many different features.

So let's say that we have two features, furriness and bad breath. That dog doesn't really look like it has bad breath. but mine does. So you can have two different features, furriness and bad breath. And dogs are generally, let's say, up here.

Now you can have other animals. This guy is definitely not furry. So he's down here somewhere. And you can have this guy up here. He's definitely furry.

So you have these two dimensions and a bunch of observations in those two dimensions, in those higher dimensions. And you can see that, in this case, you can't actually apply that one-dimensional decision-making circuit to discriminate dogs from these other animals.

Why is that? Because if I apply my one-dimensional perceptron to this problem, you can see that I could put a boundary here and it will misclassify some of these non-furry animals as dogs. Or I could put my classifier here and it will misclassify some of these cats as dogs.

So how would I separate dogs from these other animals if I had this two-dimensional space? What would I do? How would I put a classification bound? If this doesn't work and this doesn't work, what would I do?

You could put a boundary right there. So in this little toy problem, that would perfectly separate dogs from all these non-dogs. So how do we do that?

Well, what we want is some way of projecting these inputs onto some other direction so that we can put a classification boundary right there. And it turns out there's a very simple network that does that. It looks like this.

We take each one of those detectors, a furriness detector and a bad breath detector, and we have those two inputs. We have those inputs synapse onto our output neuron with some weight w1 and some weight w2, and we calculate the firing rate of this neuron.

Now we have this problem of how do we place this decision boundary correctly. What's the answer? Well, in the one-dimensional example, what is it that we learned? What was it that we were actually changing?

We were taking guesses. And if we were right or wrong, we did what? We changed the weight. And that's exactly what we do here.

We're going to learn to change these weights to put that boundary in the right place. If we just take a random guess for these weights, that line is just going to be some random position. But we can learn to place that line exactly in the right place to separate dogs from non-dogs.

So let's just think a little bit more about how that decision boundary looks as a function of the weight. So let's look at this case where we have two inputs. So now you can see that the input to this neuron is w.u.

So now if we use our binary neuron with a threshold, we can see that the firing rate of this output neuron is this step function operating on or acting on this input, w.u minus theta. So now what does that look like? The decision boundary happens when this quantity is pulled to 0. When this input is greater than 0, the neuron fires. When this input is less than 0, it doesn't fire.

So what does that look like? So you can see the decision boundary is when w.u minus theta equals 0. Does anyone know what that is?

Remember, u is our input space. That's what we're asking, where is this decision boundary in the input space. w is some weights that are fixed right now, but we're gradually going to change them later.

So what is that an equation for? It's a line. That's an equation for a line.

If u is our input, you can see w.u equals theta. That's an equation for a line, base of u. The slope and position of that line are controlled by the weights w and the threshold theta.

So you can see this is w1, u1, plus w2, u2 equals theta. In the space of u1 and u2, that's just a line.

So let's look at the case where theta equals 0. You can see that if you have this input space, u1 and u2, if you take a particular input u and dot it into w-- so let's just pick a w in some random direction-- the neuron fires when the projection of u along w is positive. So you can see here, the projection of u along w is positive.

So in this case for this u the neuron will fire. So any u that has a positive projection along w will make the neuron spike. So you can see that all of these inputs will make the neuron spike.

All of these inputs will make the neuron not spike. Does that make sense? So you can see that the decision boundary, this boundary between the inputs that make the neuron spike and the inputs that don't make the neuron spike, is a line that's orthogonal to w. Does that make sense?

Because you can see that any u, any input, along this line will have zero projection, will be orthogonal to w. Will have zero projection. And that's going to correspond to that decision boundary.

So let's just look at a couple of cases. So here a set of points that correspond to our non-dogs. Here are a set of points that correspond to our dog. You can see that if you have a w in this direction, that produces a decision boundary that nicely separates the dogs from the non-dogs.

So what is that w? that w is 1, comma, 0. And we're going to consider the case where theta is 0.

Let's look at this case here. So you can see that here are all the dogs. Here are all the non-dogs. You can see that if you drew a line in this direction, that would be a good decision boundary for that classification problem. You can see that a w corresponding to solving that problem is 1, comma, minus 1, and theta equals 0.

Let's look at the case where theta is not 0. So here we have w.u minus theta. When theta is not 0, then the decision boundary is w.u equals some non-zero theta.

That's also a line. It's a equation for a line. When theta is 0, that decision boundary goes through the origin. When theta is not 0, the decision boundary is offset from the origin.

So we could see that when we had theta is 0, the decision boundary-- that network only works if the decision boundary is going through the origin. In general, though, we can put the decision boundary anywhere we want by having this non-zero theta.

So here's an example. Here are a set of points that are the dogs. Here are a set of points that are the non-dogs. If we wanted to design a network that separates the dogs from the non-dogs, we could just draw a line that cleanly separates the green from the red dots.

And now we can calculate w that gives us that decision boundary. How do we do that? So the decision boundary is w minus u.theta.

Let's say that we want to calculate this weight vector w1 and w2. And let's just say that our neuron has a threshold of 1. So we can see that we have two points on the decision boundary.

We have one point here, a, comma, 0, right there. We have another point here, 0, comma, b. And we can calculate the decision boundary using ua.w equals theta, ub.w equals theta. That's two equations and two unknowns, w1 and w2.

So if I gave you a set of points and I said calculate a weight for this perceptron that will separate one set of points from another set of points, and I give you a theta for the output neuron, all you have to do is draw a line that separates them, and then solve those two equations to get w1 and w2 for that network.

It's very easy to do this in two dimensions. You can just draw a line and calculate the w that corresponds to that decision boundary. Any questions about that? Just that, if you have questions, you should ask because that's going to be a question you ought to solve.

So you can see in two dimensions you can just look at the data, decide where's the decision boundary, draw a line, and calculate the weights w.

But in higher dimensions, it's a really hard problem. In high dimensions, first of all, remember in high dimensions you've got images. Each pixel in that image is a different dimension in the classification problem.

So how do you write down a set of weights? So imagine that's an image, that's an image. And you want to find a set of weights so that this neuron fires when you have the dog, but doesn't fire when you have the cat.

That's a really hard problem. You can't look at those things and decide what that w should be. So there's a way of taking inputs and taking the answer, like a 1 for a dog and a 0 for non-dogs, and actually finding a set of weights that will properly classify those inputs.

And that's called the perceptron learning rule. And we're going to talk about that in the next lecture.

So that's what we did today. And we're going to continue working on developing methods for understanding neural networks next time.