4: Sensing: Gathering the Information

• Download English-US transcript (PDF)

The following content is provided by MIT OpenCourseWare under a Creative Commons license. Additional information about our license and MIT OpenCourseWare in general is available at ocw.mit.edu.

PROFESSOR: What I'm going to do for the next three lectures is to talk about how we see. What I'm going to do today is talk about how we take information in. Next Tuesday, I'll talk about the fact that we take in too much information -- more information than we can use and we have to select amongst that information. On Thursday of next week, what I'll talk about is the fact that perception is never just a direct, somehow registration of the outside world. It's always an active interpretation of the world, and I'll talk about how we go about interpreting the information that we've gotten from the outside.

I will be barely able to scratch the surface of the topic. I will abandon all discussion of the other senses, for instance, which is a great pity. Now, it's true that I can barely scratch the surface of almost any topic in this course, but here the danger is greater because this is actually what I do for a living. So, on any little bit of this, I can talk forever. So, exactly how much of the surface I scratch will be determined by how far afield I end up going.

This slide is up there to illustrate, in some sense, those three bits. Let's see, if I take the stage, chalkboard light off -- there we go, you want to go off? Then you can see a little bit better. You can sort of see stuff. That's this Bruegel's painting of the battle between the good angels and the bad angels. For me, the purpose is to say look, it's clear that you see something, it's clear that what you are seeing is an active interpretation. In fact, you may not have particularly known that it was a battle between good angels and bad angels before you look at it. It's also clear that even though you, in some sense, see the whole thing, you're not processing it fully, at least not all at once. So, for instance, you may not have particularly noticed -- I don't know what he is, he's a really cool beast -- I'm not sure if that's a bad angel falling or something. But you probably hadn't particularly noticed this fish guy. Actually, in this picture you're probably still not noticing the fish guy -- he just looks like a blob. It's a great picture, look at it sometime. And this guy's got very bad stomach problems, whatever he's doing.

OK. Rather than continuing to try to look at a murky picture, let's go to a less murky picture and let's start by talking about how you actually get information in from the visual system, which obviously involves going through the eye. On the handout, which -- do I have a copy of it here, yes, OK. A lot of terminology. Are you responsible for it? Yes, of course. If you miss some little bit of terminology, that's not going to sink you in the course, either on this topic or on any other topic. You want to put your effort into thinking about the broader topic, but it's useful in being able to talk intelligently about it to get a grip on some of this terminology, and hence, the tour of the visual system that I will now embark on.

From the outside-in, the purpose of the eye, or at least the front half of the eye is to form an image on the back half of the eye. So all of this, sometimes called the anterior segment of the eye, is there to form an image, like a camera would, on the back surface of the eye. Most of the optical power of the eye is at the very front surface of it. That front surface is called the cornea. It has, apart from its virtues as a lens focusing light, it is also one of the two surfaces in the body most sensitive to pain. You may try this out if you want at home. I don't recommend it, but if you scratch your cornea you will know it and you will be happy. What's the other one? What's the other surface, do you think? Anybody care to venture a guess? Thumbs? No, that's a novel thought. No, look -- he's thinking like the cartoons, right, you hit the thumb with the hammer and ow, it hurts. Let me tell you, if you hit your cornea with the hammer it hurts a lot more. Yes?

AUDIENCE: Some place inside the mouth?

PROFESSOR: Some place inside the mouth. No. Again, you think about that inside the mouth is a place where pain happens, but that's because you do stupid things with your mouth, like hey, that pizza looks good. Ow. You would never stick the pizza on your eyeball. So, again, nicely innervated for pain but not particularly. Not the other great location. I'll take one more here.

AUDIENCE: [INAUDIBLE] at the bottom of your foot?

PROFESSOR: Yeah, yeah, same sort of thing, right. You step on nails or something. The bottom of your foot's a good place. There's two people here, I've just gotta take their answers because they're so eager.

AUDIENCE: Yeah, that's what I was going to say, somewhere inside your ear.

PROFESSOR: Somewhere inside your ear is the point, not the chunk that so many people are thrilled with. Oh, let's put a hole in this -- oh, that was fun, let's do it again. That obviously hurts, I guess. I've never gone in for this myself. One of the reasons your mother said don't put anything inside your ear that's bigger than your elbow is because if you put something small and pointy in there, it eventually reaches your eardrum and that's the other surface. When you had an ear infection as a kid and the pressure from your middle ear, the fluid in your ear, stretched that, it hurt, and that hurts a lot. So, don't put the hot pizza in your ear either.

OK. The cornea is just a relatively thin layer. It's held in shape by fluid, a watery-like fluid, which is known technically as the aqueous humor -- humor in this case has nothing to do with being funny. It's an old word for a fluid, It forms a space that separates the cornea from the iris. The iris is the bit that gives your eye its color, but it's basically a sphincter of muscle with a hole in it, and that hold is the pupil of the eye, which is the aperture of the camera. That's where light's getting in from the outside into the inner portions of the eye. OK. The pupil changes size, because you've got this sphincter of muscle that either relaxes or contracts. What makes it change size?

AUDIENCE: Light.

PROFESSOR: Light. We know that. More light makes it do what?

AUDIENCE: Get smaller.

PROFESSOR: Get smaller. Why?

AUDIENCE: [INAUDIBLE].

PROFESSOR: Yeah, so there's less light. Sure, it's going to be less light, but why do you have this? What did they teach you in 3rd grade?

AUDIENCE: [INAUDIBLE].

PROFESSOR: Never stick your elbow in your eye. Yes, I know that. No, I need a hand here. There's too many mutterings. There's a hand.

AUDIENCE: Because you don't want to go blind.

PROFESSOR: Because you don't want to go blind. Why would you go blind?

AUDIENCE: There's too much light.

PROFESSOR: There's too much light. See, that's what they teach you in 3rd grade. It's bogus. It sounds good and it seems to make sense, but ask yourself what level of protection that pupil can give you. Right, so how much light is it going to cut out? The answer is it cuts out a bit more than a factor of 10. From the largest it can be to the smallest it'll be if you stare at -- you know, if you're out in really bright light. So it can take it can change light levels by about a factor of 10. If you go from in here right now -- actually, it's fairly dim in here -- but if you go from here to just go outside into the courtyard, the change in light level is going to be on the order of a factor of 1,000 right there. The change in light level that your visual system can deal with from, say, a moonlit night to bright sun on a snow field is on the order of 10 to the twelfth or 10 to the thirteenth. So there's no way that the need to protect yourself is going to be taken care of by the pupil of the eye. There's just not enough range there. The pupillary constriction, what it's probably doing for you, those of you who are photographers, is it's increasing the depth of field on your camera. A pinhole camera will focus many different distances at once. A larger aperature will only focus one plane at one time. And so if you've got lots of light, you crank down the pupil in order to be able to focus -- in order to get better focus, basically. But the protection thing just doesn't -- it's not a stupid answer because it's what everybody believes, but it doesn't turn out to be a correct answer.

Oh, now there's several other things that will change the size of your pupil, but one other of interest, what else changes the size of your pupil? Anybody know anything else that they can change the size--? Yup?

AUDIENCE: Looking all the way down there at you.

PROFESSOR: Looking all the way down there at me. Well, yes, but for reasons that are--. Why? Well, no, well you're changing -- sorry, that's the next bullet here. What you're changing when you change focus of the eye is you're changing the shape of the lens -- at least you are. I'm not because at my age the lens has basically become a rigid lump, and so when I want to look from there to here, I use an external lens.

Yes, back to the pupil.

AUDIENCE: When you lie.

PROFESSOR: When you lie. Yes, that will work a little bit maybe, not reliably enough to -- you know, the CIA or something every now and then gets the idea that that really works. It's on the right general track. Yeah?

AUDIENCE: Some drugs do that?

PROFESSOR: Yup. Some drugs will certainly modulate it. All right, [UNINTELLIGIBLE].

AUDIENCE: When you look at the object of your affection.

PROFESSOR: There you go. That's particularly the one I was fishing for. Your level of arousal will change the size of your pupil. Notably, if you look at something that you like or someone that you like, your pupils get bigger. Obviously, not everybody here knew that explicitly, but you sort of knew that implicitly, because the result of that is that you think that things with big pupils are cuter than things a little pinhole pupils, right. That's why sappy cartoons or cards have big pupils. It is also why in years gone by, women would put a drug known as atropine or known as a cosmetic as belladonna for pretty woman, pretty girl. In their eyes what it did was it paralyzed and relaxed the muscles of the irish and made a great big pupil. It didn't do anything good for their vision because it made things kind of blurry. It also paralyzing the lens. But it gave them great big pupils, and so they're looking at the guy who may or may not be the object of their desire -- they're looking at some guy, they can't see him so well, which who knows, that may have helped. But he's looking at her thinking implicitly her pupils are really big, I know what that means. Well, it turns out, following up on whoever it was over there, it actually means she's on drugs. In any case, pupil size is modulated by emotional state and it's one of these things that we sort of knew implicitly, people know implicitly, and think big pupils are attractive for that reason. It's only when you show up here you find out these things explicitly.

All right, I've already said something about the lens. The most notable thing about the lens is it gets harder with age and -- you could try this out. In fact, it's a good time in your life to start this experiment. Take something like the handout and ask yourself how close you can get and have the letters still be in sharp focus. This is so sad it like makes me weep. But I know you're getting there, too. Now what you can do -- you could actually, if you're compulsive about these sort of things, write it down somewhere and just check every year. This is one of those things where the signs of aging start showing up now. You will be able to see, even during the course of your twenties that that so-called near point is moving further out. For me without my glasses we're out here somewhere. And eventually you're locked at one plane of focus, and depending on where that is, determines which particular set of glasses you need -- do you need glasses to read, do you need bifocals, whatever. It's because you've lost the ability to fine-tune that focus.

Now, you need something to hold the rest of the eye in shape. What you've got in there is it's really a jelly called the vitreous humor for the glass-like substance. For our purposes it's not desperately interesting, it's there to hold your eye out so that the front end of the eye can make an image on this back surface. The back surface of the eye where the transduction from photons, the light energy, into signals in the nervous system where that takes place, that tissue is known as the retina -- the word comes from the word for a net. The fovea is the point -- fovea comes the word for pit -- it's the point on the retina where you have the highest visual aquity, the greatest ability to resolve small details. You don't particularly realize in the course of your normal day-to-day existence just how bad vision gets away from the fovea. You have this notion of a highly detailed world that's in focus all over the place. But that, to jump ahead to the third lecture in this series, is a construction. It's not reality. And that you can prove to yourself. Take the handout again, hold it at some comfortable reading distance, fixate on the l on that second line there where it says Lecture 4. If you're fixating on the l, the l is forming an image on your fovea. Ask yourself while you're fixated on the l, how many letters off to the right you can actually read. And the answer is going to be not many. If you think you can read to the end of line it's because you're not fixating -- you're moving your eyes around.

Again, like the pupil thing, this is something you knew implicitly. Nobody sits there and says I'm going to read this handout. I can see it, it's there, but I can't read it because the outside of the fovea, I simply don't have the resolving ability to do it. Why not? Well, I can give you at least one reason. You have about a hundred million photoreceptors in your retina. Photoreceptors are the cells that convert light energy into a signal in the nervous system, they're the transduction elements. You have only about a quarter of a million to a million axons, nerve fibers running from the eye to the brain. So that hundred million has got to get boiled down to the much smaller number that you've got available to you. So that means you can't treat each photoreceptor as a pixel with a private line to the brain. You do have something approximating this in the fovea where you have fine detailed resolution. But in the periphery, away from the fovea, what you have is many, many, many, many photoreceptors that are all ganged together to send their signal off to the central nervous system. So you lose resolution out there in the periphery.

Let's say a little more about the details of that retina. Oh, the optic nerve is the bundle of fibers going off from the eye to the brain. I'll say a bit more about that in a minute or so. A quick tour of the retina itself. Up at the top there you've got the photoreceptors, these transduction elements. There are two flavors of them called rods and cones -- named rods and cones because under the microscope rods are rod-shaped and cones are cone-shaped. They are different in their functions and capabilities. Cones operate in bright light at daylight levels. They do mediate color vision and they are concentrated at the fovea. They are most densely represented at the fovea. Rods work in dim light levels. They are not able to mediate color vision, which is why on a moonlit night you don't see colors -- it's not because color's somehow drained out of the world. It's because the visual system that's looking at that world can no longer analyze the input for color information, but only for grey scaled information. Rod photoreceptors are concentrated away from the retina -- I'm sorry, away from the fovea. They are, in fact, absent from the central fovea. The best place to see this actually is if your room gets dim enough at night that you can't see color, you're working in the rod realm, and then you will discover that if you fixate on a dim spot of light you can't see it. That there might be something out here -- you notice it and then you move your eye to look at it and it'll disappear, because the image, if it's small enough, will now fall into the fovea and there are no rods there, you can't see it.

This is why if you're a star gazer and you're looking at a dim star, you want to look at that star out of the so-called corner of your eye. The corner of your eye is about 20 degrees away from straight ahead because that's where the density of rod photoreceptors turns out to be greatest. I was going to say something else about rod photoreceptors, but that has disappeared from my mind. So I will say that the rest of this is a collection of nerve cells, of neurons. It's really a little chunk of brain. Developmentally the retina is a chunk of brain that's pushed out into the eye, and this is the start of the visual nervous system. For present purposes it's worth thinking about two aspects of this organization. One is a through path that runs from the photoreceptors through so-called bipolar cells to ganglion cells. There's sort of a vertical path in this picture heading from the photoreceptors out to the brain. Then there are cells whose processes are sitting here and here making horizontal connections so that photoreceptors don't behave like isolated pixels so that they can talk to each other, and so you can start doing some sort calculations, if you like. Those are cells like horizontal cells that have their processes up here, and amacrine cells that have their processes down here. So those are things making horizontal connections across the retina. Photoreceptors, bipolar cells, ganglion cells are forming this through pathway going from eye to brain.

Now, if you were to look at this and I was to say where is light coming from in this picture, the reasonable answer would be?

AUDIENCE: [INAUDIBLE].

PROFESSOR: It would seem like a reasonable answer would be like it's coming from the top hit in the photoreceptors and heading off to the brain and out the bottom. But in fact, light's coming from there. The retina is put on backwards in an interesting kind of a way, or intuitively backwards at least, so that light shines through all of this stuff before actually getting to the photoreceptors. Why is that? Well, there are a variety of possible answers. One of the answers is -- let's see, the remaining term on the list that I haven't mentioned in that retina list, which is actually at the top ofo the list, it says pigment epithelium. Up here there would be a layer of what boils down to black gunk coating the back surface of the eye, and the photoreceptors are stuck into that. That's the pigment epithelium.

What it's there for is -- well, let's suppose that all of you guys are photoreceptors and here I am, I'm a photon and I'm part of this image of the world on the outside and I'm coming flying in, and with luck I'll be absorbed by Mr. Photoreceptor here. But suppose I don't? Suppose I hit over here. Well, photons don't have much conscious life as far as we know. You, the owner of this visual system, don't want that photon bouncing around anywhere else. You want it either to be absorbed by a photographers or to be gone. Because if it bounces and hits up there somewhere it's going to degrade the image, right. It's going to be like haze across the image if it's not landing where it should to make a nice starp image. So this black gunk is there to soak up extra photoreceptors. A very good idea if you are a diurnal animal, an animal who works out in bright light. Not such a good idea if you're an animal who is nocturnal or running around in dim light. There what you want to do is catch every photon that you can possibly catch. So if you're a cat, for example, you do this differently. Instead of a pigment epithelium at the back of your eye, you have a structure known as the tapetum -- it's actually very beautiful, it looks like Mother of Pearl, but you only get to see that if you take it out of the cat, which makes the cat less beautiful. But it's a reflective surface. This is why if you get a cat in your flashlight beam, its eyes seem to glow at you, because the light comes in -- if it doesn't get absorbed by, say, the photoreceptors, it can bounce and an amount of it will bounce back out. Some amount of light will bounce back out of your eyes. That's what you see as red eye in an unlucky photograph, for example.

Is that a hand?

AUDIENCE: Wouldn't that distort the image?

PROFESSOR: To the cat.

AUDIENCE: Yeah.

PROFESSOR: Yeah, absolutely. That's why you almost never see cats reading The New York Times. No, cat visual aquity is pretty lousy. A cat has different visual desires than you and I do. The cat's not big on sweating the visual aquity thing. The cat wants to know, did it move? Can I jump on it? Can I bite it? Is it another cat? Can we do stuff? All this sort of stuff. This sort of thing doesn't require a lot of visual aquity and the cat wants to be able to do it in dim light. So, we have this notion that cats can see in the dark, which, of course, is not really true -- no light, no vision. People think cats can see in the dark because cats can see stuff in dim light that you and I can't, and part of the reason for that is that they are willing to let stray photons go where they will. Because, in fact, I gave a sort of a radical example. If my hypothetical photon missed my hypothetical photoreceptor here, odds are it's not going to go out to outer Mongolia, it's going to get absorbed someplace nearby, and so the effect will be a blurring, not a complete degradation of the image. I think the cat can cope. The cat doesn't mind too much.

By the way, if you're interested in design, taking a look at the design of eyes across the animal kingdom is a beautiful--. The number of different ways that evolution or the hand of God, you can take your pick, has chosen to solve the problem of collecting light, an amazing array of possibilities. Once you start -- cats are one thing, but you go off into insects and other beasties and it's great. Wonderful stuff. If you want to be an engineer, probably lots of good industrial design there.

All right, let me take a quick trip up into the rest of the visual system here. This picture that's up on the screen now is mimicking the picture on the upper left of the second page of the handout. It's showing the connections -- oh, I just remembered. I want to do a quick demo here before I do that. Let's go back and do that.

If the retina is on backwards, you got this interesting problem. So, there are those nerve fibers to the optic nerve. How are they going to get out of the eye if light is coming from there? What they do is they run across the surface of the retina until they reach the optic nerve and then they go out up to the brain. Where are the photoreceptors at the optic nerve? The answer is there aren't any photo receptors at the optic nerve. If there aren't any photoreceptors at the optic nerve, what do you see there? The answer isn't nothing because you're not aware of the fact that there's some big hole in your visual field. But there is. You will automatically fill it in because if it's green here and green here and there's no information there, it's a good bet that it's probably green all the way across and your visual system can figure this out. But it's worth it to prove it to yourself that there is a blind spot there. Let's see, on the handout -- well, flip to the back of the handout where you got a little extra white space, and put an x on handout here like this, and a little ways away like about like that, put a little black dot or a little dot -- I don't care if it's black. So the dot is to the right of the x. Close the left eye, cover your left eye. Look at the x, hold it so that the dot is straight out to the right and then move the paper back and forth and you will find that there's a spot where the dot completely disappears. What you've done at that point is move the dot into the optic nerve head -- the place where the optic nerve is leaving the eye. If you're saying I can't do this, I don't see a blind spot, it's not because you're special and don't have one, it's because you're doing it wrong. Well, there is an alternative. I'm given to understand that the octopus retina is not put on backwards in this way and so perhaps you're an octopus.

OK, let's go back to this wiring diagram here. You can play with this later. It's cool. Do things like -- well, if you flip it around you can find that you got one in the other eye, too. You can also do things like draw a line with a hole in it. If you draw a line with a gap in it and put the gap in the hole, the line will appear to complete. If you get really adept at this you can do what King Charles the Second of England was reputed to have liked to do, which is you can look at your friends and put their heads in blind spots and watch their heads disappear. Charles the Second didn't get to chop off as many heads as his predecessors had, so this was the best he could do I guess.

All right. How do the axons from the optic nerve feed the visual centers of the brain? You will recall, from the first lecture perhaps, that everything from the right side of my body on the skin senses gets represented in the sensory homunculus on the left, and everything from the left in the sensory homunculus on the right. The way to not do this individual system would be to have let's have everything from the right, that's my right eye, my right eye go to my left hemisphere, and everything from the left eye go to the right hemisphere. That wouldn't be any good. Isaac Newton figured out without bothering to do any anatomy, Isaac Newton figured out that that wouldn't be any good. Because I want to look out at the world with two eyes and I want to see one world. So I'm going to have to do something that brings the information from the two eyes together. What you actually do is the following. Everything from the left side of each retina ends up in the left hemisphere, and everything from the right side of each retina ends up in the right hemisphere. The image on eye is flipped -- simple lenses do that, right. So I've got a simple lens in that anterior portion of my eye. That means that the image of you guys landing on my retain has your feet up, your head down and left and right reversed. So what that means here is if I stare at the person wearing yellow here -- if I stare at the person wearing yellow she'll look embarrassed. It always works but that's a different phenomenon. All right, but I'm staring at you but I'm not actually paying any attention to you. What I'm actually doing is looking at -- my aquity's so bad, it's a black blob right about there. I assume she's a person -- yeah, she's a person, OK. Anyway, acquity's very bad. If you're on my fovea, the woman to the right is in the right half of my visual field landing on the left side of my retina. She lands on the left side in both eyes, and as a result, she's landing in my left hemisphere. If I'm looking at you, some arbitrary blob -- there's a blob that just moved over there. Oh, she's a person, too. This person in the left visual field ends up on the right retain in each eye and in the right hemisphere. So there's only one of her, in a sense, up in my brain, and only one of her up in my brain and they're getting together in that--. And the result is that half the fibers from each retina have to cross to the other side.

That's what this cute little map on page 2 is telling you. But the important thing not to get confused about is don't tell us on some exam that your left eye goes to your right hemisphere or something. It's left half of each retina goes to left hemisphere, right half of each retina goes to the right hemisphere. What it does once it gets there is -- this is a monkey brain, and what you're looking at is the cortical surface here. That must be the outside surface and this is the inner surface of the hemisphere, if there are two hemispheres. The one on the bottom is looking at the inside surface. The colored areas are ones that are important in vision. I talked at the beginning about the fact that primary visual cortex is right here at the back of the brain, but it's amazing how much of the brain is devoted to vision. It's very hard to look at in a picture -- well, it's very hard look at, period. What vision researchers like to do is to make maps like this where you take the cortical surface and flatten it -- it's all this wrinkley involuted structure -- use a variety of techniques to flatten it out like a map. Then you see that this is the piece that's at the back of the head, the one primary visual cortex. Then you've got all these gazillion other regions that are important in visual processing, some of them for rather precise sort of things. Where did mt go? I think right there is mt, for example, which is important in motion processing. Lots of work being done trying to figure out what different bits of it do.

Look, let me give you a broad organizing scheme for how to think about this. It's not perfect but it ain't bad. From primary visual cortex where very basic information about spots and lines and primitive bits of motion gets pulled out, more advanced processing can be thought of as in two large streams. One large stream going up towards the parietal lobe is telling you things about where stuff is in the world. How is the world laid out? So, my notion that this space is laid out as a big slanted plane in front of me is probably a product of this sort of a pathway. A second pathway going down into the temporal lobe is very concerned with what stimuli it might be. So there, if you're a monkey and I spear individual cells, it might be where I would find a cell that was particularly interested in a hand, a monkey hand, or a hairy leg or something like that. It turns out that the hairy part can be important. Some of the work on things like monkey hand detectors was being done by one of my professors at Princeton, and actually my TA in intro psych, when I was an undergrad, was a grad student who was working on this, and he reported to us that he was recording -- you go spear a cell. It's one thing if you're just trying to figure out if it likes spots of light, but how do you figure out if the cell likes something like a hand. Well, they had in those days drawers full of stuff and you waved all this stuff in front of the monkey and see what makes the cell go. These experiments in the good old days tended to run for very long time, because this was an anesthetized monkey and the monkey wasn't going to wake up from this particular operation. You ran the experiment for days and days and days and you stayed up the whole time. So, Bob Desimone, now a famous visual neuroscientist is up late at night with this stupid monkey with his stupid cell and the cell isn't doing anything. He's waving everything at it. So finally, he took off his pants, wrapped a towel around his waist and did a little dance in front of the monkey. The cell went nuts. Being a good scientist, he then narrowed this down to the notion that this was a cell that actually seem to be interested in hairy legs, and the hairy part turned out to be important. I feel free to tell this story about Bob Desimone because Bob Desimone is running around the country with a talk at the moment that has a picture of me in it kissing a wombat. Don't ask. It all seems fair to me.

All right, so if you go up into, way down here in the temporal lobe, you might find cells that respond only to something like a hairy leg or a monkey hand or something like that. If you're hanging out back here in primary visual cortex -- well, actually let's go back and hang out in the retina even. Look at that, we'll hang out right there. If you're hanging out in the retina, if you like much simpler needs than this hairy leg detector somewhere up in the temporal lobe, I mentioned before that you might be bringing together a whole bunch of photoreceptors through, say, a bipolar cell to a ganglion cell, to bring together a whole bunch of them to send a signal off to the brain. But what I didn't mention is that not all the photoreceptors that are hooked up to stimulated ganglion cell are going to excite that cell. Let's draw some pictures over here. What actually happens, and I'm just going to mimic this over here, is that you'd have some photoreceptors who are set up to excite a ganglion cell. Then some other guys around them who would be set up to inhibit it. The result is that what this cell -- that's a one-dimensional slice and this would be a two-dimensional picture -- the retina's after all a two-dimensional surface. So what you'd really get is some region of photoreceptors that excite the cell, and some surrounding region where light inhibits the subsequent ganglion cell. Then light out here does nothing. So this is the receptive field of this cell -- this is the only part of the retina that this cell over here cares about. It's going to send its signal off to the brain. What does this cell do? Well, among the things that this cell can do is it can tell you I am most excited by a spot that's exacty this size. If the spot gets any bigger it's going to encroach on this inhibitory stuff and reduce my excitation. If the spot's any smaller, I'm not going to get all of my excitatory guys excited, and the response will get smaller. So I'm most excited by a spot of exactly this size. If I build receptors with different size centers, this guy's only going to be excited by a little spot and I could make one that would be excited by a bigger spot. So, even at the level of the retina, you can start to get cells that are telling you more than just light on, light off, they're starting to tell you something about the size of things in the visual field. When you get up to the cortex, you find that now cells come to be interested in things like lines -- not just a spot but a line. And this cell will way, I like lines in this orientation. I don't like lines in this orientation. In fact, I like lines in this orientation only if they move this direction. So you can imagine that what you start to do early in the visual system is to pull out a whole bunch of little features, if you like, that you can then use subsequently to try to figure out is there a monkey hand here. Is there a laptop computer or something like that. Early in the system you've got basic features being pulled out of the image.

Now, this presents a problem. Suppose that what you want to do -- Let's start with orientation. Suppose you want to decide that a line is a particular orientation. How good are you at that? The answer turns out to be that you are good enough to be able to tell roughly 1 degree increments. How are you going to do that? Well you could have a cell specifically designed to look for zero degrees vertical, 1 degree, 2 degree, 3 degree, 4 degree, and you need one of those everywhere in the visual field, and at all different sizes -- a little line that's vertical, a big line that's vertical. You're going to need an awful lot of cells then. That's not what we do. What we seem to do all over the place in the visual system, and in sensory systems in general, is to make comparisons. That comparisons are very powerful in this game. So how do you decide that something's vertical? To a first approximation what you do is you look at whether or not the lift-tilted guys and the right-tilted guys, the cells that like stuff -- the cells that look for something that's tilted a little to the right, cells that are interested in stuff that are tilted to the left. Are they roughly equally excited? If they're roughly equally excited in balance one with the other, then odds are that you're looking at a vertical stimulus. OK? If the right-tilted stuff is stronger than the left-tilted stuff, so that balance gets tiled, you're going to say that the stimulus is tilted. It turns out that if you do something to make -- let's say you go off and you make the left-tilted stuff weak by tiring it out -- how do you tire it out, you show it a bunch of left, show it a bunch of left lines, that makes this weaker. Then something that's vertical's going to turn out to look tilted to the -- you're not going to believe that. Let me show this to you. I'll show it to you in -- I'm going to jump to here. Well, we'll jump here first. OK. Here are a very boring picture. It will be even more boring if I turn out a little more of the lights here. OK. How you know that something's white or grey or achromatic? Again, in this case you've got color mechanisms that say how red or green is this? If the answer is well, it's balanced between red and green, and you also ask how blue or yellow is this? If it's not blue and it's not yellow and it's not red and it's not green, well, you know what it's probably white. OK, well, so here let's take this red/green thing and let's make the green side weak by wearing it out. We'll show you some green.

Now I'm going to show you something white -- the green is weak, the red is strong, the balance goes this way, and you think that something that was once white now looks kind of green. There, see, isn't that amazing? They're not convinced. The ones who are convinced are confused. Stare at the black dot at the center here. You just want to stare at the black dot and think about, say, that red dot up at 12 o'clock. What you're doing is you're tiring out the red side of this red/green equation in this particular case, and just keep staring there, and then I'm going to go boink. So, the people who went "oh," presumably saw colors. Let's do that again, here. What you want to notice is that the color is this opposite color, it's going the other direction. So the red one will turn green, the yellow one will turn bluish, the green one will turn reddish, right. So stare at the center, boink. It works. That's what's known as a negative after image. You can just toggle back and forth. If you look at the black dot on the right and then move your eyes to the left, you'll see the colors show up in the left hand circles, because that after image is essentially cooked onto your retina -- it's there in retinal coordinates and moves with your eyes. Alright, now let me illustrate to you that this is a ubiquitous phenomenon that shows up for basic phenomena all over the place by turning this back on. Is that going to be enough light for this? So, I won't show you the orientation one, even though I managed to get a PhD studying it, but I'll show you the motion equivalent of it. Here's what you want to do. You'll still see it over there, except for the guy who's asleep behind the pillar, that's his tough luck. So, look at this, and we're doing the same game. We're just taking the game and transferring from the color realm, or the orientation realm, down to the motion realm. So this is contracting, right? Stare at the center, that's important. So stare at the center. Again, think about 12 o'clock. All the contours are moving down. You know that something's stationary because it's not moving up, down, left or right. But at 12 o'clock I'm wearing out your down detectors, and at 6 o'clock I'm wearing out your up detectors and so on. The consequences of this evil act of mine will be seen vividly if, when I take it away, what you do is look at my nose. Of course, what really worries me here is several people's pupils got much larger, which means they think I look better that way.

Now, a word of warning. Two words of warning. This is the best demo in the whole class. It's all downhill from here. The other word of warning is it's a very salient demo, but it's there for a purpose. It's really lame on the mid-term or the final when some motion -- this is called a motion after effect -- when a motion after effect question shows up to say oh, that was cool, man, his head was shrinking and stuff. Because you don't get much points for that. What you want to know is why this is actually interesting, which I will reiterate by going the other direction since it seems a pity not to, right? OK, so again, the way you figure out that something is, in this case, stationary is by making a set of comparisons, and Iam using this clever little tool to systematically distort the comparisons that you're making. By the way, I'm also causing you to make one of these inferences that we'll talk about next week. By seeing all these various bits of odd motion, you are making the conclusion that my head is shrinking or growing or something like that. So this is expanding, right, so this is the amazing pinhead version. Ready?

The original version of this, by the way, is known in the literature as sometimes it's the waterfall illusion, because it gets described multiple times in the literature, but one of them is by a Scotsman named Adams who was staring at a waterfall and then noticed that the rocks on the side of the waterfall seem to be drifting up. This is a known waterfall in a known spot in Scotland and there's a group of vision researchers who meet there every year and the entry requirement for joining this particular elite group is a bottle of scotch of a variety that they have not tried before, which I think helps make the rocks move in all sorts of interesting ways. They've never invited me so how would I know. OK, this is what I'm going to do. Let me follow up. So you've seen a negative color after image. You've seen a negative motion after effect. I've asserted that these exist in orientation. You can show this often -- you show somebody big stuff and then medium stuff looks small. It's ubiquitous. It shows up all over the place. I want to show you one more version of this kind of a effect, and that's going to take a certain amount of build-up here. So the first thing to do is say the vertical and horizontal stripes, do they look different to you? If they look different somebody raise their hand and tell me what the difference is. What's the difference?

AUDIENCE: The horizontal ones are horizontal.

PROFESSOR: Yeah, alright. In their color or shading in some fashion. Yeah?

AUDIENCE: The vertical ones look blurry.

PROFESSOR: The vertical ones look a little -- they do look a little blurry. Yeah?

AUDIENCE: The vertical ones -- the black lines look like they're thicker than the ones--.

PROFESSOR: Well, we're not getting a lot of vivid color reports here.

AUDIENCE: The horizontal lines look brighter.

PROFESSOR: Brighter? Nobody knows what the word color means. Last try.

AUDIENCE: The horizontal lines are greyer and the vertical lines are black.

PROFESSOR: OK, look, this makes the point just fine. Never mind guys. There is no in general consensus here. I'm about to produce a general consensus. That's the motivational part of today's lecture. No, actually what I have to do now is for the next few minutes what you want to do is keep your head upright. You don't have to stare at any one spot. You don't even have to your head upright, just keep it in the same orientation. If you flop all over the place and fall asleep nothing good happens. So you're going to look at these green vertical lines and these red horizontal lines. All I need to do here is toggle back and forth between them. Now you can see the negative after image here. If you stare at this green -- oh, I should have done that quickly before. So stare at this green thing for a while. You get an nice purply red thing? OK. So that's just the negative after image. But we're not interested in the negative after image anymore. Actually, we are. There's an aspect of the negative after image that's really cool here, which you may start to notice shortly. During the blank period, shout out the orientation that you see.

AUDIENCE: [INAUDIBLE PHRASE]

PROFESSOR: That's a little weird. Wouldn't you think? Now that's a little weird because if you stare at these horizontal stripes -- in fact, I should be able to get you back to seeing horizontal. So, stare at these -- fixate, don't move your eyes around and we'll just stare at that for a minute or so. Has it been long enough? Yeah. Does it look horizontal? Still looks vertical, oh boy. Well, here's the get rich quick part of this lecture, assuming that you're unscrupulous, you can use this to make your fortune claiming that you can teach people to be psychic, right. What's the next orientation I'm going to show? What's the next orientation I'm going to show? If you can make your fortune out of that, just send me 10%. But what's really going on here, what this is actually illustrating, this aspect of it is illustrating, is that you're seeing with your whole visual system, not with your retina all by itself or something like that. So, each time you look, say, the red horizontal, you're building up a green horizontal negative after image on your retina, right? And this is building up a red vertical kind of after image on your retina. If you go back and forth, you build up sort of a plaid. So think about your retina as having a plaid on it. Now you don't see it here, but the plaid is -- when you're looking at a blank screen, it's just that plaid that's getting shipped up to your brain, basically.

OK, well what else is happening? Well, when you're staring at this green vertical thing, up in your brain the cells that are interested in vertical are saying we're getting tired. Go to the blank part, tired vertical guys and more awake horizontal guys are looking at this weak plaid and they're only seeing the horizontal part. Now you look at the horizontal, the horizontal guys get tired out -- oh, we can only see the vertical, and so on. You may ask isn't it about time we saw something interesting here? The answer is undoubtedly so, but this particular effect takes a few minutes to build up, and so you gotta keep looking at this. Oh, you've got to ask yourself what the effects ought to be. What I'm building up here is an effect called the McCullough Effect, named after Celeste McCullough, who unlike most people who discover visual effects, was actually looking for this. She knew what she was looking for and she knew what she could find. Now I already showed you that the pattern you're going to look at again, right, it was that black and white pattern of vertical and horizontal lines. What do you think the vertical lines should look like when we're done? They should look kind of reddish maybe. The horizontal should look?

AUDIENCE: Green.

PROFESSOR: Green. Let's see, well we might have done this long enough. This is one of these things where if I do it long enough I can hit absolutely everybody, and if I quit too soon not everybody will see the effect. All right, here we go. So, close your eyes, because that will let me get to the right slide, but it will also let you perhaps see that checkerboard floating there in the dark? Murmur if you can see a checkerboard floating in the dark. OK, good. That's that's the negative after image thing. OK, now take a look at this test pattern again. How many people think that the vertical and horizontal look different now in terms of their color? How many people think it might look a little sort of maybe kind of different, but it's not exactly biting me on the posterier? Alright, that's a better test. Now, the way to find out whether you've actually got the right effect is without knocking out your neighbor, tilt your head 90 degrees. So that's cooler. I'll say a word more about this, but I won't say a word more about this until you've had a chance to get up and stretch for a second.

OK. How many people when they look at these dots still see reasonably compelling colors? Not many. How many people when they look at my head still see it shrinking? Yeah, except for the weird people. That's because your visual system is continuously re-generating itself. You go stare at the red thing, you wear out, in a sense, the red detecting chunk of your visual system, and then if you flip your eyes over you find out that you see a white thing is green. But then you recover from it, right? Good. How many people still see this effect? Tilt your head just to make sure if you think it's there. A little weak today. We probably should have gone a little bit longer. Those of you who are seeing it, and perhaps even those of you who are not -- did I put a test pattern on the handout, I forget? No, OK. I'll just bring the Powerpoint back next week. Not only are you still seeing this some minutes afterwards, but particularly if I don't show you the test pattern a lot you may see it for days and weeks. The longest report in the literature for the McCullough effect is three months. The only reason it's only three months is because at that point the undergraduates, who, of course, were the subjects, went away on vacation and the experiment ended. The evidence suggests that if we locked you in the dark that the McCullough effect would last forever. We won't lock you in the dark. I teach this up at Harvard where there are many more pre-law students, and the immediate reaction to this news is half of them consider filing suit. This will not, in fact, ruin your life in any fashion. In fact, it'll only be noticeable if you happen to end up looking at a test pattern like this or perhaps serving time in the near future. But this was something that McCullough was not expecting. McCullough thought that what she had done here was, OK, you can fatigue color, you can fatigue orientation, well I'll fatigue the cells in the brain and there are cells in the brain that are interested in red vertical lines. They'll get tired, I'll see white vertical as green, they'll recover, I'll see white vertical as white. But what happened was you looked at red vertical lines for a while, white vertical looked green and white vertical looked green for like a really, really long time thereafter. What seems to be going on here is something like this. Let's suppose that the way you decide that something is white vertical is by looking at the balance of -- well here, let's draw it this way. You've got red vertical input and you've got green vertical input, and they're being like let's put a little dial here. If they're roughly equal you decide something looks white. So I show you red vertical for a while and this thing gets weak and cruddy, and now something that should be producing equal input gets more from there and it ends up looking a little green. Alright, that's fine. What happens if this cell gets sick -- it's not going to be one cell, but suppose that this process gets weak in some fashion, not because you were looking at some stimulus, but because you went out and ate the mushroom that you shouldn't have eaten or whatever and this cell just took a hit. And so now this is kind of a cruddy cell. Well, there are a couple of possibilities. One of them is that for the rest of time you're going to end up seeing white things as looking green. That's not going to be good right. You don't want that happen. So if you decided that this was the situation, what you would want to do is change the gain on here. So if this was previously just equal gain, let's crank this up a little bit so that we can drag that back up to being equal. That sounds OK. That seems like a clever enough thing to do.

But how are you gonna do that? How are you going to figure out that you're broken. How would you know that your visual system was not reporting the truth to you. Well, I mean one way would be you're saying hey, that looks white to me, and somebody else says that looks green and they laugh at you or something. Nah, it doesn't work. No evidence that that works. What you do is that you seem to have some sort of notion of the statistics of the world, the way the world ought to be. If the world is systematically different from that, you draw the conclusion that you've got trouble here of some variety or other. Well not necessary, you don't necessarily draw the -- let me give you a non-troubled version of this. How many people have been to one of these omni theater kind of movies? The main reason for going to an omni theater movie is what? You want to feel like you're moving, right, when you're not moving. Because all the movies that they ever show you at the science museum are things where you go driving around in race cars, stuff like that. Why does this work? You're sitting at this huge screen, and everything in your visual field is moving in the same way. You know something about the world. The world on average is stationary. If the whole world is moving there's a good explanation for that typically. The answer is not the world, it's you. What is it that causes the whole world to slide across your retina? This is what causes the whole world to slide across the retina. Sit in the omni and the world is sliding across your retina. Your brain says uh-huh, the world isn't moving, I'm moving, and I think I'll throw up now for a while or something like that. That's a separate interesting question. What is it here that could tell you that you're out of whack? Well, what is the normal relationship between color and orientation in the world? Typically in the world if you see something that's vertical, what do you know about its color? That's right, nothing.

So what's going on here? It's vertical. It's green. If it's horizontal it's red. If it's vertical, it's green. If it's horizontal, it's red. You do this for long enough and a little chunk of your brain, rather like the little chunk of your brain that's saying I ate the cupcakes, I threw up, I'd better do something about this. This little chunk of your brain is saying, oh, there's a correlation between color and orientation. The only way that happens is if I'm broken, and the only thing to do about that is to adjust the gain on chunks of my visual system. So, everything looks green. All the vertical things look green. Oh my goodness, what am I going to do? Well I better crank down the gain on the green side, it's obviously too high, and I'm want to crank up the gain on the red side. Unfortunately, that was a goof. And it was wrong, so then when you look at a white thing it now looks red because you cranked the gain up too much over here. How do you recover from this? Well, if I now show you the opposite one -- what were we doing before? What was it, green vertical? If I show you red verticals and green horizontals for a while, that'll push it back the other way. But that doesn't typically happen in the world. Simply being exposed to a world where there's no correlation will slowly push you back. But the reason the effect lasts so long is that the evidence from the real world is kind a weak compared to this artificial situation that I cooked up. The way I'm thinking about this is like -- here's another way of thinking about this. Suppose you're measuring average weight of people. I'm just going to keep weighing people and what's the average weight of people. It'll bounce around a little bit. Alright. This is how much people on average weigh. Now what happens is the amazing 1,000 pound guy comes into my experiment. What happens to the average? Now I start weighing normal people again. Does it just go -- no. Weighing regular people it'll take a comparatively long time for me to get back to normal. So that's what's going on here. The McCullough effect adapting stimulus is the 1,000 pound guy that's causing you to think that your brain is broken. You go and re-adjust your brain and now slowly it will get better. How's it look there? Not better yet. How many people have a nice after effect? Still, nice McCullough effect? Celeste McCullough would be so proud.

You will notice that I have not said a word about signal detection theory. I noticed that, too. There is a chunk in [? Gleichman ?] about it. I might decide to talk about it on Thursday. Oh, you may also notice that you have lost your cell phone. If that is true come and talk to me about it because we found it. One last thing. How many people know what that is? I didn't ask you what it was, I just wanted to know if you knew what it was. How many people recognize this? Only a few. OK, now you can shout out.

AUDIENCE: It's a dog.

PROFESSOR: It's a dog. How many people does that help? It's head is -- well, wait a second. There's its head, foot, back paw, it's back, it's sort of sniffing at a ground plane that's like here. You can store that one away till next week and we'll talk about it.