Lecture 4: Designing Homework and Exam Problems

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK, welcome everyone. It's 8:05, well, 9:05. So today's lecture and discussion is about how to make good problems, whether it's on homework problems, exams, even problems in class. Its general principles that you can use constructing questions that you ask, so problems broadly conceived. Now why problems?

So problems are, I think, one of the central ways of thinking about teaching. And I'll show you how problems fit into the grand scheme of this course, of rethinking all the aspects of teaching. So one way that I like to think about teaching is called backward design. So this is a way of designing a whole course, could even be for a curriculum. And doing problems is one aspect of that.

So why is it called backward design? Well, it's backward from the usual way. So I'll show you what backward is and then show you what the usual way is.

So in backward design the highest-level thing you figure out-- not necessarily the first thing you figure out, but the highest level thing-- is your course goals. And that then feeds to OK, if those are your course goals how do you know whether students have reached them? And that's problems.

Problems, tasks, projects, whatever it may be, things that students are going to do. And so generally speaking, things to do. And then given that you've operationalized the goals in these problems what are you going to do in class, so that students actually are able to do these kind of things?

OK, so that is backward design because the usual way is the following. You say, OK, well, you have a bunch of lecture notes. Where do you have those?

Well, either the last person who taught the course gives you their lecture notes. Or you just take the standard book in the subject and you just do the lectures one per chapter or one week per chapter, something like that. And you just go through in the order.

So in the usual way you start with this. And then you think oh, whoops, oh yeah, I've got to make some problem sets. And oh, some exams. Well, what have we done this last couple weeks? OK, that's what we're going to do.

So the usual way is you start with here. And then you get here. And you somehow never get here.

The course goals are, basically, to get through the book. Right? That's the implicit course goal. But it's never explicitly planned.

So that's a terrible way, but the normal way of designing a course. So this is backward design. So this is due to Wiggins and McTighe.

OK, so here we're going to be discussing this today. So we're doing this first. And then the next session is this one. And I think the session after that is, what do you do in class, interactive teaching and things like that.

So why am I doing it in this order? Well, in the levels of abstraction these are the most abstract. And this is the most concrete, OK, what question I'm going to ask students in class now.

And this is things like goals. I want them to really understand conservation. They're really high level.

So I think if you start here, by trying to figure out course goals, I haven't found that works very well for me because it's too high level. I can't think about course goals in absence of actual problems and things I want students to be able to do.

So this has the right level between concrete and abstract. So it's actually where I start my thinking. Now, you may actually find you want to start somewhere else on the continuum. But this is what I found works quite well.

You start here, you operationalize what your goals are in the problems that you think are interesting. And then you look at the problems. And that sort of gives you an idea of the course goals.

And then you think about the course goals. And that helps you think of problems. But I start here.

And then, once I have things like that, then class time is actually much easier to plan. Because class often is going to be problems like this, related to this, maybe shorter. But it's easier to plan here after that.

Whereas we start here, you don't know, you're back in the old way. So I don't want to start here. This is too abstract, so I start here. So that's why we're doing this now. And then next week we're going to do this one. Question?

AUDIENCE: So when you say that you're setting goals for your class--

PROFESSOR: Pardon?

AUDIENCE: So if you say that, basically, setting the goals for your class, or for your own course. So when you design your course, basically you write out some kind of goals you want to reach? So how do you then do the problem? Do you go in detail before you've even started the whole class? Or do you do it that on a week-to-week basis?

PROFESSOR: Oh, what do I tell the students about the course goals?

AUDIENCE: Yeah, I mean, if you say, OK, basically, you have your goals and then you design problems so you can reach the goals? But then do you also design these problems before you even start the whole lecture? The whole semester?

PROFESSOR: Oh, well, it's pretty rough. I mean, usually the first time you teach the thing it's too abstract to design-- I find-- everything in advance. Maybe that's my extrovert nature, which is that I find it hard to design things for people I don't know. If I don't know the audience I don't even know what to say.

So I want to actually teach the course one time at least. So then the audience becomes concrete for me. So really, the first time through the course nothing is ever ideal. And you're doing everything on the fly. But the second time through the course, you can actually, basically, make all the problems ahead of time. So the second time through the course you actually refine your goals and your problems together.

So that shows you the structure of where we are and where we're going to go and then later. So this is two weeks from now. Today, here.

So how do you make problems? Well, the fundamental principle is, again, deduced by induction from what happens with regular problems. So if you remember, I gave you an example earlier of the results of standard problem solving. And I'll just remind you that we're putting up the percentages again.

OK, so after doing lots and lots of standard multiplication problems, students were-- if you remember-- unable to do the following problem. So these 13-year-Olds and 17-year-Olds were asked to estimate 3.04 times 5.3. So here were the answers.

So they were given four possible answers. And they could also not answer. So this is no answer.

So that's the age 13. So of the four possible answers-- or let's say no answer is possible as well-- so five, they're 1% over 1/5 will get the correct answer. So that's a bit depressing. So you think, oh, OK, the 17-year-olds are going to be better.

And they are, but they only have 37% correct. OK, so here is a serious problem, which is rote learning. And that's the fundamental thing to avoid when doing problems.

So these students-- especially the 17-year-olds-- were actually perfectly capable of doing exactly multiplication. So on the same test they were asked questions like multiply 2.7 by 8.32 and given just empty space on the page to multiply it out. About 80% did it right.

So it's not that they don't know how to multiply. They just don't know what multiplication means, which is a far more serious problem. I'd rather they knew what multiplication meant and didn't really understand, didn't really know how to multiply.

Because the algorithm you can teach them later if they understand what it means. But if they don't even understand what it means that basically it's 3 times 5, so it's got to be somewhere around 16, they don't understand what multiplication means or the number system means. That's a far more serious problem.

And that's produced by the traditional kind of problems. The traditional kinds of problems are, generally speaking, too low level and can be solved, to easily, by rote. Even the complicated traditional problems can eventually be turned into rote learning, and that's the danger.

So the main goal in making good problems is-- I would say-- how to make problems that fight rote learning. OK? That produce long-lasting actual conceptual understanding along with whatever else you're trying to produce and teach. But really, fight the rote learning because that is the fundamental problem.

So a way of thinking about this is to categorize the levels of thinking that you're expecting of students when you're asking them to do a problem. And for that there is work from 50 years ago now-- which is still useful-- which is Bloom's Taxonomy.

So Bloom's Taxonomy-- let's see, when was it? I think it was 1956. I think it was '56. I'll put a handout on the website for you which summarizes in one page. But I'll summarize it even shorter here.

So what is it a taxonomy of? Well, when you read the title the book, it's Bloom's Taxonomy of Educational Objectives. And it's not really clear if that's going to be useful for anything at all. Because you think, what the hell is that?

So the shortest way of understanding it is that it's six levels. And each level is higher than the one previous to it in terms of the level of thought required of a student. So it's not really a strict order. You can have really hard questions that seem to be low level just because they address misconceptions. But generally speaking, it's a rough order.

So the order is knowledge questions. Those are things like define, name, state, list, You know, it's basically almost rote. I mean, these are the kind of problems that everyone would agree are rote, state the definition of Newton's Second Law.

The second is comprehension. So that's sort of like knowledge but it's a higher level. So that's being able to grasp the meaning of the material, not just recite the facts. But generally, that doesn't include the idea of being able to transfer it to new situations.

That's where application comes. So can you apply the material to something? For example, often compute, demonstrate, show that, those are typical verbs that are used when you have application questions.

The next level is analysis. So that's really trying to understand the structure of the material. So things like diagram, differentiate, infer, outline, point out, identify, distinguish, discriminate, all those kinds of verbs com into analysis kinds of questions.

Synthesis-- so this is breaking things apart. This is the ability to put things back together in new ways. OK? So design, explain, categorize, create, revise, rewrite, reorganize, devise.

And the highest level is evaluation. So judging the value of material, for what purposes is this good versus that good? Explain, contrast, summarize, support, recommend, evaluate, criticize, appraise, so generally quite high-level skills.

So this is the taxonomy and this is a higher level. OK, so a useful exercise-- whenever you're thinking of problems-- is to try to place them on this. Now it's not that you want no-knowledge kind of questions and you want only to ask these kind of questions. Because if you just ask these kind of questions people will not be ready for them.

And they're not prepared. It doesn't actually teach anything. You want a mix of them. But generally speaking, what happens to often, is that most of the questions that happen are usually in this zone, maybe a few of those.

And then only later in people's careers do they get over here. For example, in engineering curricula you learn a ton and ton of application, maybe comprehension, Maxwell's equations, differentiation, integration. Then we start to synthesize stuff.

You know, in your senior year you do design and project classes. But you don't have to wait all the way until then. You can actually start putting stuff in much earlier.

And it doesn't have to be an entire class. You can have tasks that are like that in a short time, on a problem set in the middle of lecture time. OK? So questions about Bloom's Taxonomy?

OK, so what I'll do, is I'll give you an example of questions that in freshman physics mechanics-- on the topic of Newton's laws-- that fit into each of those levels. OK, so just to give you an example, to make it concrete, to see what those kinds of questions are. And some of the categorizations I'm going to give you are debatable.

But the rough idea is that they go from high to low. So this is on the subject of Newton's second and third law. So a knowledge question, state Newton's second and third law.

Newton's second law, state Newton's third law, so you can see, just because people can state it doesn't mean they can actually do anything with it. That's why this is way up just at the knowledge end. Computers are actually really good at this. Right? You can look it up on Google. But your computer can tell you what Newton's second and third law are but you can't actually do anything with them.

So comprehension-- so give an example of Newton's Second Law, using Newton's Second Law. OK, this next one is an application. It turns out to be, actually, quite a hard application. But it's still an application.

So it's to apply Newton's second and third law to show the following, which the following is suppose I have a piece of chalk in my hand and I stand on a weighing scale. I want the students to show that the weight that the scale reads is equal to the sum of the two individual weights. So I weigh myself.

And then I put the chalk on the scale. And I add up those two weights. And that should be the same as when I hold the chalk in my hand.

OK? So I'll diagram that as-- OK, so that's me plus chalk equals me plus chalk. So this is a scale. OK?

So there is the piece of chalk. So now this is an example of where that's quite a hard application question, even though it's kind of high up on the Bloom's list, so towards the lower end. Because this is a serious misconception that students have.

Newton's second and Newton's third law, people are completely confused about, you'll find. They don't know when to use which. And they think any time two forces are equal it's sort of 50-50, whether it's because of Newton's third law or Newton's second law. So this-- if you're addressing misconceptions-- makes the problem, actually, all the much harder.

So at analysis-- so the analysis comes out-- oh, question sorry.

AUDIENCE: What are Newton's second and third laws?

PROFESSOR: Ah, fair question. So Newton's Second Law says force equals mass times acceleration. Third law is-- the way it's usually stated, which it's terrible-- action equals reaction. Or for every action there's an equal and opposite reaction.

Oh, which reminds me, one of the questions from before, which is what name would I give? Because I said I slagged off the reaction force as the name and the normal force as a name. People ask me, what should you call it?

And I actually thought about that. And I think the answer is contact force. Because that tells you what the thing is. It's a force of contact.

So Newton's third law says action equals reaction. So now, the problem is that students have, is that whenever they see two forces are equal they assume it's because the action equals reaction. Or they think Newton's third law is the reason for it.

So for example, the bouncing-ball question that we talked about last time. So in the bouncing ball the question is, when it's stationary on the ground what are the forces on it? And they'll often say, well, there's a weight downwards. And there's the force upwards. And they'll often say, well, the force upwards equals the weight downwards. These are both forces on the ball because action equals reaction.

And they won't realize that action and reaction have to be on different objects. OK? So they're fundamentally confused about the meaning of action and reaction, which you can bring out with the following analysis question-- When two forces are equal in magnitude how do you tell if it's Newton's two or Newton three?

So how do you know if it's Newton two or Newton three? So just ask them that. Yes, question?

AUDIENCE: This might be a little bit off subject, but should we use Newton's third law ever? It seems like it's a natural consequence from saying a equals 0.

PROFESSOR: Ah, oh, OK, interesting. So OK, let me actually explain that one. I'll put it over there because I didn't do the boards right over here. So that is exactly the issue brought out by this question. And let me show you the difference.

And then there was another question. That reminds me of another whole set of questions from the sheets, which was how do you develop your physics intuition? So let me answer that in a moment. After this and then we'll have a break.

OK, so the bouncing-ball question-- actually let's not do the bouncing ball. Let's just do an object sitting on the ground. Oh, great, can you pass them around?

Oh, Leeann's just coming around with a handout on Bloom's Taxonomy. And let's see, I'll do these after the break. OK, suppose you have this object sitting on the ground and it has Mass m. OK, let's look at what the forces on it are. Well, there's mg down-- and actually, in this case, the contact force, our normal force is equal to mg.

OK, so now the question is, are these two equal because of Newton's Second Law or Newton's third law? That's what this question asks. So it's actually worth understanding the difference here.

So these guys are equal because a equals 0. So this thing's just sitting on the ground, a equals 0. So the net force is 0, so these two forces have to cancel. OK, so that's Newton's Second Law.

So then the question is, where does Newton's Third Law show up in this? Ah, Newton's Third Law shows up here. Here is the Earth. And here is your object. Let's just separate them just so you can see them separate, although this is sitting on there. So this is mg. Well, this is the mg because of gravity, right? The earth gravity is attracted to mass, m. Well, the mass is also attracted to the earth. So what's the force on the earth from the mass?

AUDIENCE: mg.

PROFESSOR: mg, it has to be. Now this mg equals that mg. Call this f2 and this is f1. So f1 equals f2 by Newton's third law. So that's a fundamentally different set of forces. So are equal and opposite. They're an action/reaction pair.

So what Newton's third law really says-- when you understand it-- is that all forces in the world come in pairs. So if you ever see one force you have to hunt around for his other half. And the other half here is this.

So actually, there's another question you could use for Newton's third law. So I don't have it on the list here. But I'll ask it right now, which is suppose someone comes to you and proposes the following gravitational law-- f gravity equals g m1 squared m2 over r squared when the g is the proper units. So m1 squared times m2, is that a legal force law for gravity?

No, because when you switch m1 and m2 as you're doing here-- you're switching the mass and the earth-- you actually change the force. So then this would violate Newton's third law. OK?

So Newton's third law, it actually is a constraint on the laws of physics. It's actually conservation of momentum. And for momentum to be conserved you can't have any arbitrary law of physics that you want. Does that help clarify that?

AUDIENCE: Come up with a better set of taxonomy for describing it that doesn't lead people-- it leads people into thinking about that.

PROFESSOR: I know. I agree. So I think the terms are very important. So that's why don't like this form, action equals reaction. I actually would prefer it were stated all forces come in pairs that are equal and opposite. Right, and language does have a powerful effect. Yes?

AUDIENCE: The technical [? route's ?] obvious, just in calling the novel force the contact force, but that's not refined enough. Because both friction and novel are contact force?

PROFESSOR: Yeah.

AUDIENCE: [INAUDIBLE] constraint force because it could constrain into the [INAUDIBLE].

PROFESSOR: Yeah, that's probably a better name. Yeah, I agree with you. So the comment was that contact force isn't quite right because there's friction force as well. And that's also a contact force. And maybe a better name is constraint force. I think that is a better name.

OK, so let me finish this list of two more examples of Newton's Second and third law. And then we'll have a break. OK, synthesis.

So creating a problem using Newton's second and third law. So for example, this is a problem like that. But here I'm asking the students to make one up. And here is a very hard evaluate question, which is what, if any, are the limits of validity of Newton's Second and third law?

So that's, actually, a very interesting historical question. Interesting historically because Einstein thought about that question. He wanted to know whether Newtonian gravitation and special relativity were compatible. And basically, they can't be because of Newton's third law.

Because gravity should propagate with some speed, not the speed of light. So you can't actually conserve momentum right away. Suppose the sun does something strange. Well, it takes a while for the earth to actually respond to that.

And it's in that intervening time Newton's third law seems to be violated. So Einstein thought very carefully about that question and came up with his theory of gravitation. So this is quite a high-level question. But it's to show the whole range of things you can ask about the second and Newton's third law as you climb Bloom's Taxonomy.

OK, so what we're going to do in the second part is we're going to actually practice rewriting a couple of questions. And I'll show you a couple more examples. And seeing how we can make questions worse and better, because I find that's actually quite a good way to get practice with designing questions is taking good questions, making them bad, taking bad questions and making them good. Both ways are useful.

OK, so we'll do that. Let's say it's 9:04, 9:10 let's say. So it's 904 on that clock. So see everyone, sorry it's 10:04 on that adjusted clock. So see everyone at 10:10 or 10:11.

And meanwhile, here are the feedback sheets. So I'll just put a bunch in the two back things. So when you come back just grab those.

OK, so I'll just answer a couple questions which I promised to answer. So the comment made by several people about last week's examples, say, of the rock and the rolling was that oh no, that showed me that my physics intuition is quite off, and bummer.

And so then if you're going to teach it, how do you make sure, well, A, how do you learn to think intuitively about these things? And B, if you're going to teach it how do you make sure-- related to that-- you actually have the intuition before you teach it.

There's two answers to that. One is that it's very rare that anyone ever achieved a goal that they didn't set out to do. And then just happened upon it. So I would say that's true, also, with developing intuition in these areas.

You have to make it your goal. So it's not going to just happen by chance. So then questions, OK, if it is your goal how do you go about it?

One way is anytime you do a calculation, at the end of the calculation ask yourself this question. So John Wheeler-- who was Feynman's adviser-- he recommended the following question, which I think is a fantastic question. So the question is if you could time travel back and talk to your earlier self what one or two sentences would you tell your earlier self before they started solving the problem that would make the problem just sort of flow smoothly? OK?

So what sentence would you tell your earlier, pre-problem solved self? So this is after you solved the problem. So that forces you to give a insightful summary of what you learned from the problem. So it forces you towards the intuitive way of reasoning.

And you'll find it's quite hard at first. Because you don't have the intuitive library. But at least it points you in the direction to be going.

OK, now what else? Well, in chemistry there's various books that help develop intuition. They have all kinds of intuitive reasoning questions in them.

And some are quite good. Some of the books are quite good. So the one in chemistry that I like is called Voyages in Conceptual Chemistry. And in physics one of the ones I like is called Thinking Physics by Epstein. I think Epstein's like that.

So what that is, is that's basically, one question per page. And it's multiple-choice intuitive reasoning question. And they're about all kinds of interesting physics, like why do tea kettles whistle?

If you shake bottle of soda up, why does it explode? And so it's filled with really, really fascinating puzzles and ways of thinking. And then with explanations.

So just by thinking about them, reading the explanation, trying to understand, OK, then asking yourself Wheeler's question at the end of reading the explanation, even if you didn't solve it, you can develop a lot of intuition. I don't know a good one for math or biology.

And if anyone does, I definitely appreciate any suggestions. OK? But there are resources in some fields.

And the other is just whenever you find paradoxes go after them and share them with everyone. And ask everyone you know,do you know any paradoxes I can work on? That'll also developed intuition.

OK, and then related to that, one comment was I've noticed that teaching is the best way of learning. How can you incorporate that into a course?

So yeah, that's true. So sometimes you just have to accept that the first time you teach the thing you're going to be learning a lot. So that's fine. And the second time you teach it, it'll be much better. So you'll actually develop a lot of intuition the first time you teach a course.

So for example, I found when I was a graduate student, I was a teaching assistant for Physics 1A at Caltech, which was like 801. And I found I had a huge number of misconceptions about tension. I just had no idea what tension was.

For example, I thought tension was a force. And now that seems ridiculous to me. And it doesn't seem ridiculous to you that's because you're in exactly the same state I was in before I was taught physics 1A. And I've got myself all in a twist when I was trying to explain it. And I finally tracked it down to actually, that particular misconception. So then I developed a bunch of questions for myself, which I also asked students about tension.

So teaching is learning. And then how do you incorporate that into a class for the students? Well, you can ask them to teach each other and argue with each other. So we'll talk about that in the session on interactive teaching.

But yeah, it is the best way of learning. And so if you can get the students to do that as well you'll be increasing their learning. And then the last question for the moment was what's my response to the following article-- so it was in the globe on February 15, 2009. And it said, don't open with a joke.

So it was summarizing a paper called "Increased Interestingness of Extraneous Details in a Multimedia Science Presentation Leads to Decreased Learning." Journal of Experimental Psychology, December, 2008. So they summarized that.

So basically, what they found is that if you start with a joke that's extraneous to the material it actually decreases the learning. And that, I think, is quite plausible. So I recommend that you start with something interesting in the beginning.

But it should be related to the material. So the problem wasn't the joke. The problem was that the joke wasn't related to the material. So the thing should be interesting. Because you want to draw people in. You don't want to conclude from that article that you shouldn't start with some interesting. But you should start with something interesting that's related to the material.

OK, so redesigning questions. So here is an actual question that we will redesign. And I've left up the Bloom's Taxonomy there. So actually, I have a bunch of questions. We'll do a couple of them. And we'll review, probably, one of the questions we did earlier, which is about the cones.

OK, so the question to start with-- and we're going to go that way and that way-- is find the eigenvalues of this matrix. So for those whose linear algebra is rusty, the eigenvalues of a matrix so a matrix, m, times some eigenvector e is equal to lambda times e. So if there's some vector that the matrix just brings back to itself with some constant, that's the eigenvalue.

And this is the eigenvector. So eigen meaning own in German. So somehow this vector belongs to the matrix.

It's said to be an eigenvector of the matrix. It's preserved when you hit it with the matrix. You get back the thing itself.

And in that way it belongs to the matrix. OK, so there are recipes for computing it. And so here the question is, I'm asking students to use the recipe for computing the eigenvalue.

OK, so now I'm going to ask you the following question, which is to design-- well, first of all, where would you say that guy goes? It's sort of comprehension. You know, it's basically computing.

Maybe it's a bit of application. But it's basically comprehension. OK, so can we go lower on the taxonomy towards knowledge? Can we go higher towards the higher-level reasoning? Let's try to construct questions on this end.

OK, so everyone understand the thing to work out? I'm going to ask you to try to think of stuff that could be put here and here. So this is lower level and this is higher. Does anyone need me to explain-- or want me to explain-- more about eigenvalues or eigenvectors?

OK, so find a neighbor or two. And try to think of stuff that can go in this box or this box. And generally, the higher-level questions requires more knowledge about the material, which is one aspect of their being higher level.

You need a bigger margin of safety when you're trying to design them. But don't let that bother you. See what you can construct. And you can use the levels on Bloom's Taxonomy to try to create some kind of questions around it.

By the way, the answer to this question, by the way, is i and minus i, just in case you are curious. OK, so let's see what examples you've come up with, so related to this question. Obviously, you'll change it around some. Let's say either side, this side or this side. Someone I haven't heard from? Yes, can you tell me your name?

AUDIENCE: Mike.

PROFESSOR: Mike, yeah?

AUDIENCE: What is an eigenvalue?

PROFESSOR: OK, so what is an eigenvalue? Where would you put that?

AUDIENCE: Lower. [INAUDIBLE]?

PROFESSOR: Oh, yeah, good question. It depends where you put this one. This one is computing, which I roughly put that somewhere between comprehension and application.

So right now I would say, we're somewhere around there. So your question is to define eigenvalue? OK.

I'm just short-handing it as defined lambda. OK, so that's one question. OK, another question. Can you tell me your name?

AUDIENCE: Wendy.

PROFESSOR: Wendy, yes?

AUDIENCE: So related to that, is that you could put that question higher. If you asked them, what does an eigenvalue mean? If you word it differently instead of [INAUDIBLE].

PROFESSOR: OK, which may have been what you meant as well and I may have defined lambda. OK, so you're saying, well, you can actually ask them to say, what does eigenvalue mean? OK, so how would you if they know what it means? So what would you look for in something like that?

AUDIENCE: If they could describe it to you, or maybe tell you a problem in which it'd be useful, in using it?

PROFESSOR: OK, so yeah, let me write that too. Create a problem that would benefit from lambda, that would need lambda. OK, so for example here, create. That's gone pretty far down towards synthesis.

I really should have flipped the list around. But synthesis is quite high on Bloom's taxonomy. So you're actually going to create a problem. It's like the Newton's law example. Yes?

AUDIENCE: [INAUDIBLE] instead of saying, again, what does lambda mean? You could say what do the results from finding lambda? What are the results of that solution?

PROFESSOR: OK, so how does it help you to know the eigenvalues? Is that the question?

AUDIENCE: No, I'm just saying instead of the general asking for the definition, what does lambda mean? What is an eigenvalue? What does it mean? But after having solved the problem saying, what does your answer mean?

PROFESSOR: Ah, OK, so even if you ask them this first you can say, OK, interpret your answer. Yeah, and that's a-- generally-- very good policy. So interpret. and where would I put that? Eh, interpret is somewhere like analysis, maybe a bit of synthesis. But yeah, it's quite high on the list.

So interpret, that's another really key word. In fact, it's often neglected, right? We just ask students to solve the problem. And wander, wander, wander.

They get the thing that looks like an answer. They put a box around it and they're done. But actually, you can massively increase the level of a problem just by adding to the problem, interpret your answer. What does it mean, that that's an eigenvalue?

And maybe they would say something like this. But even that would be better than just having done the computational procedure and got an answer out of it. Yes, can you tell me your name?

AUDIENCE: Ben.

PROFESSOR: Ben, yes?

AUDIENCE: I might ask, does that matrix have any eigenvalues?

PROFESSOR: OK, so yeah, so let's put it over here. Right, so there what you're doing is you're not presuming. You're not key-wording what they should do.

When you do this-- especially if it's a linear algebra course-- they've probably seen a bunch of things like this. Like in a calculus course, integrate these things by parts. They just know I have to integrate by parts, whereas here you're at least one step higher.

Because they don't know whether there's an answer to the problem or not. And they have to think, well, how do I know if they're eigenvalues? So there's actually one intervening level of sophistication. Other suggestions? Yes, Adrian?

AUDIENCE: Just give them two numbers and say write a matrix that has these as the eigenvalues?

PROFESSOR: OK, so let me continue the list over here. OK, so create a matrix which has the following eigenvalues. So say lambda equals 2 comma 3. So create a matrix, create a meaning.

We're already high up in the taxonomy toward synthesis. And actually, I would even use a related one, which is-- so create a real matrix with eigenvalues i comma minus i. So that one, is quite related to this. And this is the answer.

So it's quite a hard question. Even if people can do this and find i comma minus i, this one is very hard because you're asking to apply a constraint that the thing be real. So it's not just any matrix. But it's that the thing be real.

And then you have to really understand quite a lot about matrices to do that. Yes, can you tell me your name?

AUDIENCE: Scott.

PROFESSOR: Scott.

AUDIENCE: So, I was thinking of a synthesis question which would be is after multiplying the item the eigenvector by the matrix how is it rotated?

PROFESSOR: OK, so what is the geometric effect on the eigenvector of the matrix on the eigenvector?

AUDIENCE: Right, but the question is if they understand what an eigenvalue is.

PROFESSOR: They should be able to answer that right away. Right, but it really tests for understanding of what the meaning is. So it's another way of approaching the question of do they understand eigenvalue means?

What is the geometric effect of the matrix on the eigenvector? And then hopefully, they'll be able to say well, actually, it doesn't rotate it. It just scales it. And the eigenvalue is the amount of scaling. OK, and that demonstrates quite a high level of understanding too.

And This is one reason that oral exams, generally, are quite good ways of evaluating. Because you can ask these kind of questions. And as soon as you see that they understand you move on to something else.

Which of the flip side is that's why oral exams are very disconcerting for the students. Because you spend no time on the things they understand. Right?

So as soon as you know they understand it, you move on to something else. So the student just feels like, what the hell happened to me? I was doing fine. And they asked me about something else.

Yeah, that's exactly right. That's why they asked you about something else. They knew there was nothing more here to find out. You understood it.

But that's why oral exams are such a good way-- so efficient-- for evaluating. So in Cambridge we used to do oral exam interviews for admission for undergraduate. So people would be admitted to the major right away.

So we had two 20-minute interviews. And we found those were much more reliable than all the exams that they did in high school, the equivalent of AP exams. Because you could really ask questions that really probed understanding and right away see what people could do and couldn't do.

So geometric effective of m on e creating a matrix, so you're asking, so what's the geometric effect? It's not just comprehension. You're asking them to apply it and maybe, in a bit in a new area.

They hadn't really thought of eigenvalues and eigenvectors as geometric at all. And now, all of sudden, you're asking them that. You're asking them to transfer their knowledge, which is one of the fundamental goals. Yes?

AUDIENCE: Once you give them two matrices that has the same eigenvalues you can ask them to see if they are related in some ways.

PROFESSOR: Right, so for example, let's do this one. So how are those two matrices related? So they both have eigenvalues i and minus i.

This one obviously does because those are the diagonal values and there's nothing else. And this one, you have to do it by the magic procedure. You don't have to.

And so how are they related? So that is some kind of analysis question. And it's quite a difficult analysis question.

It tests for really, do they understand what matrices mean? So that's why I've underlined this word. The questions all in this column are all somehow testing that they understand what things mean.

So that ties back to the point I made earlier, which is that the whole fundamental point of asking good questions is to avoid rote learning. Is that by asking good questions you're fighting rote learning. And rote learning is the bane of most education. Right?

Rote learning is too often the result of education. And it's the-- in my view-- fundamental problem with education. So by asking questions that are different you're actually starting to fight that and break the habit.

So for example, how does rote learning show up? Well, in the bouncing-ball question, so the force is-- in the instant that it's stationary, while it's bouncing-- people say mg. While this mg is right. The gravity, but the contact-- let's say the constraint force-- in this case it is a very full-contact force is mg.

One of the reasons they say that is the misconception that they think a equals 0 because v equals 0. So that's one reason. But the other reason is that they've done so many problems with books resting on tables, balls resting on tables, where the force is mg. Right?

So they've now induced a new rule of physics, let's call it Newton's fourth law. That anytime you see a contact force it is mg. So contact force equals mg maybe times cosine of theta.

[LAUGHTER]

PROFESSOR: Right? or sine theta. I guess it's mg-- well, it depends how you measure theta. But theta is the tilt of the plane. So if the plane is tilted 0 degrees just to ground, well it's then mg.

So that kind of learning is the result of rote learning because they've solved problems without understanding what they mean. So instead of figuring out that oh, actually, you could understand it from Newton's Second Law and understanding acceleration, they think oh, there's a new pattern here. Right? Oh, a new pattern is whenever I see contact forces it's mg times maybe cos theta or sine theta. Yes?

AUDIENCE: What's the exact question with that system and what's the normal [INAUDIBLE]?

PROFESSOR: Yeah, so the question was so you drop the ball, say from here. And when it bounces off the steel table there's one instant when it's stationary on steel table. And what are the forces on it at that instant?

AUDIENCE: I remember from last week.

PROFESSOR: Yeah, no, fair enough. I should have stated the question out again. So this is stationary for an instant. All right, so actually the ball is quite compressed here. Yes?

AUDIENCE: Isn't it impossible to give an exact answer for the force with the [INAUDIBLE] if you don't have the duration?

PROFESSOR: It is, right. So it is impossible to give the exact answer. But what you ask them is, well, how big is it roughly?

So I always protect myself by saying roughly. And then you can ask anything you want. And because, really, all I'm interested in is do they think it's 1 times mg or do they realize it's 10 to the 4 times mg?

And those are so far apart that I don't care about factors of two or three or even 10. Now, how can you know that it's 10 to the 4 times mg? Well, that is quite a hard question. That's way down there.

Even though you'll see some lists of Bloom's Taxonomy say estimate is a low-level skill, not that high in Bloom's Taxonomy. It's almost like computing somewhere in comprehension. I don't agree with that.

Because as you saw from the example where students couldn't estimate 3.04 times 5.3, they're terrible at estimation. So estimation actually hits one of the big misconceptions and reveals a lot of the misconceptions. So I put estimation almost at the level of synthesis because it forces you to really synthesize your understanding. You can't hide behind formulas.

So the next thing I'll do is say, OK, let's estimate this and see if you can do that. And if they really understand the system they can estimate it. But it's quite hard. And I find almost no students can do it on their own.

But the method of estimation is, as you say, you do need to know the contact time. So the acceleration is not 0 as you say. But it's delta v over delta t, where delta t is the contact time. Well, it's going v and then it's pretty much going v up, so that's about 2v. But I don't care about factors of 2.

And then, what's the contact time? Well, there's a couple methods for doing the contact time. One is you listen for the pitch. And whatever the highest frequency in the pitch is-- suppose it's 10 to the 4 hertz-- well, then 1 over that time, 1 over the frequency is roughly a time. So it's probably the contact time.

So that's one approach. The other approach is to say well, actually, physically what's happening? The ball is hitting the bottom. And this end compressing.

But this end keeps going. Because it has no knowledge that the bottom of the ball hit the ground. How's it going to know about that? Well, the bottom of the ball has to tell it. How's it going to tell it?

Well, it has to send some kind of signal. How does it send signal? Sound.

So it sends a sound wave upward. And the sound wave travels in steel about 5 or 10 kilometers per second. So you suppose it's a 5-centimeter ball, then it's about 10 to the minus 5 seconds is your contact time.

So this is really short. So suppose v is something like a meter per second, and divide it by 10 to the 5 seconds. You're talking about something like 10 to the 5 meters per second squared, which is 10 to the 4 times the acceleration due to gravity. OK?

So I went through because I think it's interesting but also to show that estimating actually requires generally, very high-level skills. And it's quite hard for the students. Therefore you should do it. So there are some people who say, well, it's very hard for the students. Don't do it.

But again, there's another rule of thumb in teaching, which is nobody ever learned anything that you didn't actually try to help them with. Well, that's not quite true. But if you never teach something they're not very likely to learn it.

So if it's something hard but valuable you have to start teaching at some point, which is why I put up that graph earlier. Oh, it's gone now. Where in the earlier stages of teaching you do things more approximately. And then in the later stages you do it more exactly.

So what I've told you now is a slight lie. So this brings up the question from last time, what about lying? Is it useful?

Yes, it's essential because this argument is simple enough that once you understand it you can just keep it in your head. And it's a bit of a lie, the reason being that the ball doesn't have a uniform cross section. So as the ball compresses the contact area changes, whereas if the ball were cube or a rod the contact area would stay the same as it compressed.

So there's a correction factor because the contact area changes. Now, that correction factor is not very big. It turns out to be the fifth root of some dimensionless thing.

And so this sound-wave argument you put in to the 4/5 power and then you put in the other thing to the 1/5 power. So it's almost correct for this spherical ball. So in a later of course it actually adjusts for that factor.

And then in a much later course, probably, what I would do is calculate the exact 2 pi over whatever factor that shows up as well, by way of increasing the symbolic complexity. But even this estimate, doing that is conceptually hard. So I would say start that early.

AUDIENCE: I think the point I was trying to make was that this-- I suspect that the students answered that wrong because of real problem solving, and that they expect there to be a solution given the data that you [INAUDIBLE].

PROFESSOR: Well, if you've worked with them a bit on estimating they know that they're going to have to do some thinking. If you say compute the force I agree with you for sure. They're going to think they have to be able to do it from the data there. And they'll just say it's got to be mg because that's the only other force I know.

But if you say estimate then there is a bit of a queue that they have to include some other information which might not put in the problem. And so that mitigates the rote learning. But what I find, even then, is they usually still can't do it because their misconceptions about acceleration are so deep and about equilibrium and force. They think f equals mv, basically.

OK, so let's just look at what we've done here. So I would say you can define lambda. You can define eigenvector. You can even say state the computational procedure.

I'm not sure where I would put that. Depending on how people learn it, this could be purely a knowledge question. Because people could state the procedure but not really understand it, not be able to do anything. Or actually, you could say program the procedure.

Well, that would be something over here probably. Because to program it you probably have to understand what you're doing pretty well. So stating the procedure, it depends how you ask them to state it, whether it's a low-level or a high-level question. So I'll put program over here.

OK, so we've gone down and up. And again, I want to remind you, I'm not saying all your questions need to be here. But generally, the questions are too much over here and not enough over here. So this gives you a recipe for thinking of new questions. Yes?

AUDIENCE: So throughout the course do you basically, maybe start on the first three in the beginning and then you move it more through until you reach the end of the course?

PROFESSOR: More fractal. The question was as the course moves on do you move more towards synthesis? I actually try to make it fractal. So within each unit and each day even, I try to maybe have something, knowledge or comprehension just to make sure everyone's tracking.

But try to have something interesting as well, all the time. So something synthesis, analysis, maybe some evaluate. And so each problem set-- for example-- I'll put, generally, warm-up questions. So basically they're marked warm up so that people know OK, those are going to be knowledge and comprehension. And you need that.

Because otherwise there's no point trying to do synthesis if you don't have the base knowledge. But just make sure you understand those, just in your sleep. And then regular problems, which tend to be somewhere in the middle, say application or analysis. And then bonus problems which are just optional.

But they're for people who either find all the other problems easy or are really curious. For them I put either synthesis or evaluation. For example, in the approximation class the last problem set had design an email indexing system.

So that was the bonus problem. And there was a bunch of warm-up and regular problems which were involving Unix text processing. And then for the people who are really curious actually, put it all together and make a really fast email-indexing system so you can just find any email that you want just by doing keyword search. Sort of like a web search but just on your email inbox.

Because I actually use that myself. Because I find folders are just hopeless. It just takes all my time. I have to figure out what folder things are in. And that could be many.

So I just keep it all in one giant inbox with, I think, 150,000 messages in it. And I just search through that. And I have an index that's rebuilt every week or so. And so I can search through it really fast. So it's actually very practical problem and it gives you an example of the levels of things you can ask.

So try to do it on a fractal scale. Each class should have something, mix the levels each homework, the exam. So the general rule is to mix them up all the way. OK?

So hopefully, from that, you feel confident that you have tools that you can make interesting questions. And this does have lots of payoffs, which one of them is that questions that are enjoyable and educational, students love them. So when I practiced what I preached and made these kind of problems, mix of problems for my classes, often students have said, oh, can you please have more problems? Which is very rare.

They said, oh, these are some of the most interesting problems I'm likely to get at MIT. And I used to read them out in my living of my independent-living group just so that other people could hear the interesting problems and we'd try to solve them together. And then I've had several emails after class finished, saying, oh, you promised to post the optional problem set. Where is it?

So it's because the problems, actually, somehow connect to things that people find real and interesting. So that's a mix of the perceptual. Don't make it just purely symbol manipulation. Try to connect to ways people look at the world. But also, try to connect to higher-level reasoning skills. Question?

AUDIENCE: Do you always use bonus problems?

PROFESSOR: I do. I find that works really well. Because it's like a pressure-escape valve. It caters for the diversity of backgrounds in the class. The people who are feeling very worried by the material, they can just skip them, no problem. And they don't feel like they have to do them. And they don't feel like, oh my god, I can't do them. I'm going to get points off.

And the people who find the other stuff easy, they find something for them too. So yeah, I try to always use bonus problems. And I learned that from Knuth's book, called Concrete Mathematics.

So actually, I think they have five levels of problems at the end of every chapter. There's warm ups. There's homework problems.

There's exam problems, bonus problems, and research problems. So the research problems were the ones that are not yet solved. So at least you know they're not yet solved. So you're going to be spending a while on them. And then in the second edition I think some of them were actually solved.

OK, so if everyone could just take a one minute and fill out the feedback sheet? Oh yeah, and let me just say, why I'm doing this now at 10:50, of the questions on the sheets before was the feedback sheet, the person said, I've been using them in my class and they've been really useful.

But I find that the response rate has been dropping. What do you do about that? And I have to say, that I found the same thing. This year I found the same thing.

Not in this class but in my other class, whereas I didn't find that last year. And I was trying to figure out why. And theory I have is that I was trying to put in too much material in the class and not leaving the students time before the five-minute mark to fill out the sheet.

So one thing, I've been trying to mend my ways and make sure I end class at 9:50 or 10 minutes to the hour so the students have at least one or two minutes to fill out the sheet without feeling pressured to get to their new class. So I'll let you know how that goes over the next week. So take a minute now and then that'll maybe make you think of some questions which we can answer in the next two minutes.

And then while you're doing that, just a couple announcements. So sorry I didn't say anything before about what to do with the equation treatments that you shared with each other. So just discuss them with each other. You have a collaborator.

And then revise your treatment based on what that person told you. You don't have to turn in your revised treatment. You guys are all graduate students, or almost all of you. You're all taking the class because you're interested. I'm not trying to police everyone.

But the goal here, and what you should do, is take the benefit of the collaboration, revise your equation treatment, and share it with each other. And then just bring in your original one next week. And then I'll also have a couple of readings for you to do for next week. And also a short problem to work on, things to think about to bring in next week as well.

OK, so sheets and then questions. Question?

AUDIENCE: So I tried doing these sheets for a class that I was teaching yesterday. In fact, I found that I got a response rate of 15%. If your response rate from the beginning's very low and [INAUDIBLE] to increase that rate?

PROFESSOR: Right, so 15%, yeah, I find I'm getting that around now too. And I think what's happened is that I've got people in the habit of not filling them out because I wasn't giving them enough time. So I may be hosed because now they're in the habit. So I'm trying to mend it. And I'll let you know if that fixes it by giving people actual time before they feel rushed.

But it may be too late. And I worry that it is too late. Because last year I was actually more diligent about making sure they had time. And I found the response rate stayed about 60%, 70%. But even I find the 15% very useful.

And I found that-- people even in the past, when the response rate was 30% in another class people said, I didn't have a question all the time. But I knew when I did I could ask it. So even the 15% isn't bad. It's still useful. But yeah, I would like it to be higher. And I'll tell you how that goes. Question?

AUDIENCE: [INAUDIBLE]? Wiggins and--

PROFESSOR: McTighe, yeah.

AUDIENCE: Oh, McTighe. Where are they based?

PROFESSOR: So where are they? They're in New Jersey, and they actually consult on educational course design and run workshops and courses. And we actually invited Wiggins last year to come and speak. So I'll put a short reading selection from their book called Understanding By Design for optional reading for people so then you can learn more about that.

I find the basic form that they say very useful. I find applying it the way they describe, sometimes I just can't square it. But it may resonate with you quite well. And the overall approach resonates very strongly with me. So hopefully you will like it too. So I'll put that on the website and some readings for next time about course design. Yes?

AUDIENCE: Just a comment, actually on your web page, where the response sheet [INAUDIBLE] and situation where only a few people were very dislike the current [INAUDIBLE]. They don't like it this way and so they want to change it. But the 80% who don't feel [INAUDIBLE] is actually OK with it. [INAUDIBLE] the change. It makes most people unhappy.

PROFESSOR: Unhappy, right, so that can happen. And so that's the silent majority problem. So what I do when I get comments like people saying, well, I wasn't happy about this. Sometimes what I do is-- especially if it doesn't resonate with my intuition about how things were going, and I thought they were going well-- I'll say, well, there were a few comments about this issue. What do people think? And just take an informal straw pole before I make big changes.

AUDIENCE: I think it helps to have a positive question on [INAUDIBLE], like, what did you take away from this lecture? Or what did you really understand?

PROFESSOR: Right, I think that's right. It helps to have a positive question. So that's why I say, either what you take away, or what helped or hurt? So there's an opportunity to say something helpful. And any other comments is also interpreted in a positive way.

Answers from Lecture 5 to questions generated in Lecture 4.

PROFESSOR: OK, and questions from last time. So they grouped into several ones. One is how should you choose your levels of your problems-- say, in class or in homeworks-- among the different levels of Bloom's Taxonomy?

Should it be, for example all towards the evaluation? So your side would be here, sort of the high level. A Mix? A flat mix? Should it be peaked towards the middle with some tails?

So that depends, partly, on your course design. So to answer that question, you can't really answer that question until you figure out what your course goals are. So in some ways there's an argument for doing course design first and then working on what problems to do and how to design problems. But actually, I like the other way around. I like doing course design second.

Because actually, doing problems is concrete enough to do something. And then it leads to questions about course design, which we'll answer how to mix the levels. OK, another one was several people wanted to know, why is tension not a force? Or in fact, maybe is a force.

So let me explain that. Because that actually gives you another example of sorting out misconceptions. So it's always useful. So I'll show you the sequence of questions I used with myself when I finally convinced that tension wasn't a force. And here's how I did it.

And then I used the same sequence of questions with students to fight the misconception that's been induced by many pictures of-- for example-- a pulley. And here's a rope and there's tension. Sorry, this is actually done the other way. So here's the rope. And there's a t labeled there. And then on the mass there's a force labeled as t upward.

So when you see that you think, oh, tension's a force. So you see that a whole bunch of times and you're pretty sure of it. So the sequence of questions is as follows-- say OK, here's a tree. And I'm going to tie a rope to the tree, rope or a string, it's a mass-less thing. And I put a force on it.

OK, so I'm going to pull that tree. The tree's not going to go anywhere. We're going to pull on the rope with 100 newtons of force. OK, so you can try this yourself too. What's the tension in the string?

OK, so take 30 seconds and I'll take a tension. OK, a tension, anyone?

AUDIENCE: 100 newtons.

PROFESSOR: OK, so in fact that is 100 newtons. And most people will say 100 newtons. OK, so now the next question is this-- here is a river and here's a rope. And me and a friend each pull on the rope with 100 newtons. OK?

OK, so discuss with your neighbor-- a, b, or c, what's the tension in the rope? And we'll take a vote. OK, so take 10 seconds, get your vote ready. We'll take a quick straw pole.

Who votes for 0 newtons? Who votes for 100 newtons? Who votes for 200 newtons?

OK, so now here, how do you know what it's going to be? Well, you make a force diagram. And there's a way to convince yourself that there's no doubt about what it should be.

And we just draw the forces on the rope here. Oh, we've already done that. Let's draw the forces on the rope up there. Well, what are the forces on the rope here? Well, it's 100 newtons from you. But there must also be 100 newtons from the tree.

All right? So in fact, these two situations are identical. So yeah, 100 newtons seems pretty clear here. So it has to be 100 newtons up there too.

OK, now that-- so far-- isn't the hard part. But now is when you actually create the contradiction with the idea of it being a force. It can't be a force because if it were a force you would just add up vectors. Right?

So there's a couple answers you get if it was a force. If you're a bit sloppy you'd just add up the magnitudes and get 200 newtons. If it's really a vector-- you're putting two vectors on it, two forces-- you should get 0 because they're two opposite forces. But actually you get 100.

The only way to get 100 is that the tension has to be something completely different from a force. So that's how you know it's not a force. But how do what it is?

Well, it turns out to be a tensor. So it's not a vector. Forces are vectors.

It adds differently than vectors do. It adds like a tensor. So tension is more like a pressure.

It doesn't have a particular direction. Pressure is in all directions. So things like that, they can't be described by vectors. So you need something else. And that's why we introduced tensors.

So now, in freshman physics I wouldn't give them a whole story about tensors. But I do actually teach this much of it so that they know that is a different object. And then what do I do about this?

Well, I won't write t unless it's for myself. But for the students I think it's sloppy and it increases misconception. It's much better to write the force sub t, which is the force due to tension. OK?

So even a slight change like that can mitigate the rote learning, where people think oh, tension's a force. Because you always see it drawn like every other force. There's mg down, There's t up, must be a force too.

AUDIENCE: [INAUDIBLE]?

PROFESSOR: What are the two objects you hit it with to make a scalar? It's the two directions that you're interested in, basically. So it would be the x direction or the direction along the string and the direction along the string.

AUDIENCE: So it's the two direction then?

PROFESSOR: You can use the same direction twice. So it's the direction along the string and along the string. There's no cross direction. So it has no shear. So there's no cross direction.

So it's some component, tension is one component of the stress tensor. So now, again, freshman don't need to know all the details. This is back to the subject of lying they do need to not be told fundamental lies. Like for example, tension is a force.

OK, so now, how do you deal with misconceptions that show up on problem sets? This is a good question. So the point was made well, in class you can sort it out right then. Suppose you ask a question, say, about the Ideal Gas Law.

And you find that people have confused the Ideal Gas Law and the Adiabatic Gas Law. Well, you can talk about it right then, which is a good reason to do those things in class. What happens on the problem set if you find there's a bunch of misconceptions?

Well, one thing is, the first time you do the problem set, yeah it's a bit of a problem. Because you don't know what the misconceptions are unless you've researched ahead of time. But the second time you do the class you know what they are.

So you can address the misconceptions in the solution set. So when I'm not totally sleep deprived I always try to get the solution set out the exact same day that the problem set is turned in. So the students can right away see. They can turn in their problem set, pick up a solution set or look at it online.

So they can right away get quick feedback. And they can sort out the misconceptions that way. So that's a good question. So it is harder. Yes, question?

AUDIENCE: What do you do about the fact that a lot of people will just ignore the solution sets? Or do you just consider that their problem?

PROFESSOR: So the question is, what do I do about people who just ignore the solution sets? Because a lot of students do that. Yeah, partly I consider it their problem. Because look, that's there for them to learn.

But partly, to the extent that they're ignoring the solution set because they only cared about the grade. So one cause of that is they really only care about the grade. So they just want to wait for the grader to give them back their solutions, their problem set with a mark on it. And they don't care what's right and wrong.

That, I think, partly, is our responsibility that we've chosen boring problems. So to fix that, I think, one of the key areas is to use more of the Bloom's Taxonomy and make interesting problems. And the flip side is to deemphasize the grading.

So actually, I like the PDF scale on homework. I think we are too micromanaging, generally, about homework, you know, grading it on a percentage scale, on A, B, C, D. P, you did a reasonable. And D, you blew it off. You did something but it was a joke. And F, you didn't turn it in.

I think that scale puts the right emphasis on the grading. Look, we just want you to do the thing and try it. And if the problems are interesting that allows people to actually get involved in it. So there's lots of studies that show that if you reward people for things they already enjoy-- so you start paying them more for doing things they really like-- they actually start liking them less.

So I'll talk about that as one of the political barriers to educational change. Because that's completely against the conventional psychology of this society, where everything has to be rewarded, merit-based pay and this and that. So how do you go against that? It's quite difficult.

But one way, in general, in those problem sets is to deemphasize the grading. Related method-- which I haven't tried yet but I'm planning to-- is reading memos on the solution set. So I talked about reading memos before. And maybe one of you will try it before me and can tell me how it works.

A reading memo is-- I talked about-- where you have the students take a reading and mark-- either online or on a piece of paper-- things that were confusing or things that were interesting and things they noticed. Well, you can do that with the solution set too. You say, OK, your assignment, homework 3A is to look at the solution set for homework two and mark anything that's confusing.

So that forces people to read the solution set. And often what you'll find is that they'll find mistakes in the solution set, which is good to know because you can correct them right away. But you'll see what parts are confusing to students or not.

Did that help answer your question? Great. Do I ever asked Wheeler's question explicitly? Yes, sometimes. I should do it more in class in lecture.

But I do it a lot to myself. Whenever I solve a problem I think oh, what was the key idea? And then once I see the key idea happen many times I think oh, that's a very key idea. That should be taught explicitly in class.

So I do that. And I'm going to talk about that principle of course design today, basically organizing courses around large themes and principles, which is one of the main [? cures. ?] OK, so what's wrong with tension as a force?

How far should I go down the taxonomy? Basically, where in the course should I be? Really low in the taxonomy early and higher later? Well, I talked about that a bit last time.

The fractal picture is not bad. So yeah, you have a few low-level examples and higher-level examples. But you do that throughout the course.

So you don't wait until the end of the course to get the reward of the really interesting problems. And then related to that, what about exams and homework? Should they be the same or different?

My view is they should be the same given the constraint that maybe the exam is done in a shorter time. So if anything, the exam should be easier than the homework. Generally, homework people will spend six or seven hours on.

When I was an undergraduate we had take-home exams. And we would spend 24 hours on them. You pick them up, you signed it out. You'd turned it in in 24 hours.

So that was misery. And it was interesting. You learnt a lot.

But it was kind of miserable because it just wiped out a whole day from your quarter. And you had that for four classes. So four days from your quarter were just gone.

So generally, exams are three hours, maybe two hours. And so that's much less time than a homework set. So if anything, exams should be easier, definitely not harder. Student loath when the exam does different things than the problem sets.

And that is a general principle of course design, that if you want to prepare them for being able to do stuff-- for example on the exam-- you have to prepare them with things. You have to "teach to the test." That's actually good.

You want your homeworks to be like the exam. And that's fine if the test is good. If the test tests real-world, interesting skills sure, then it's fine to teach the test because they'll come out being able to do that, no problem. So the exam and homework-- as a rough rule of function-- will be pretty much the same. Or if anything, the exam should be easier.

Oh yeah, the first time you teach a class. So all this Bloom's Taxonomy and designing questions, it takes a long time. And it's really hard to make good questions. It's easy to make define XYZ questions. But what about the higher-level questions? What do you do to mitigate that problem of a huge amount of time?

Well, one of the cures is to cooperate with other teachers. So you steal their good problems. So I gave you the T.S. Eliot saying before, that talent invents and genius steals. So be a genius.

Whenever you see a good problem just use it. And whenever someone wants a good problem, if you have one, share it. And now, should you guard your problems?

I think those days of guarding problems are gone. They've gone by because of things like OpenCourseWare, putting problem sets online. It's just not possible anymore to guard problems. I'm not sure it was ever a good idea. But now it's not even practical.

So everything is online, all from the previous year. So the problems can't be guarded. And it's not practical to make up a whole set of new good problems for the next year. You'll never sleep. So I just trust the students not to cheat by looking at the old solution sets.

And if you're grading, that's yet another reason to minimize how strict the grading is. So if the grading is really, just that they make an effort and try to learn something, then there's no benefit to cheating and looking at the old solution sets. So you actually make the incentive structure so that they don't need to do all that. Question?

AUDIENCE: Do you have to change your problem to a certain extent in terms of plagiarism?

PROFESSOR: So the question is, do I have to change the problems to a certain extent to avoid plagiarism? I don't even care. I don't even change them.

I think, again, it's sort of what Adrian has said, is it their problem if they do it? And to some extent it is their problem. If they're going to cheat and look at the old solution set and not acknowledge it. There's only so much you can do.

And you then get in an arms race with them. You change the problem to some extent. But then someone else makes it.

So they can always subvert that. One person can solve the new problem and give it to everybody else. So you have no hope of actually winning that war, even if it were a moral war to try to win. So I don't even change them.

And I say look, just acknowledge, just like you would in science. I think it's Larry Lessig, the copyright scholar, professor of law. He used to be at Harvard. Now it's Stanford.

He founded Creative Commons and various fantastic copyright projects. So he said, well, there's a couple ways to get-- not compliance-- people to do things. One is you can do it through rules. For example, this is illegal. You just can't do this. We'll punish you if you do this.

The other one is through norm. So this is just not done. In our community we just don't do things like that.

And generally, this kind of-- not exhortation-- but push works much better than the punishment kind. So I just tell them, look, in science when you grow up and become a scientist of course you acknowledge your peers and people you learn from. Because that's just polite.

It's just how we are. We say, please. We don't say gimme, gimme, gimme. I'm working on teaching my daughter that right now.

She says, daddy read. Daddy read. Mommy read. Elsa read. I'm like OK, I'm fine with the Elsa read.

But daddy read, mommy read, that's nice. And we love reading. But how do you ask? And now she says mommy read please, daddy read please. And so it's the start.

So people already learnt that from young. So you can tap into that. Look, how do you act as a member of the community?

We're social animals. So you don't have to put too much pressure in that direction. Whereas if you try to do it through a punishment you have to work much harder. The punishment one-- underneath-- has the idea that we don't trust the students. Right?

And in that way it's common with changing the numbers on the problems set. It's that we don't trust you so we're changing the numbers. So I try to avoid those things as much as possible and try to go more towards this. Yes?

AUDIENCE: Has anyone tried to have problem sets without solutions? So then there's no grading. And then the problem sets are just for practicing.

PROFESSOR: Yeah, have problem sets for practice with no solutions or no grading on the problem sets? Have people tried that? So there's actually whole educational systems based on that.

Not so much in America, but in England, the Oxford and Cambridge tutorial system. You're given problems to work on. So I was on both sides of that system.

You're given problems to work with your tutor. And they're not graded. I mean, your tutor helps you make sure that you can do the problems.

But everything that's graded happens at the end. Well, when I was an undergraduate it happened at the end of your entire degree. There was eight three-hour exams that counted for everything. And nothing else counted.

So it had some craziness. That time at the end was pretty stressful. But before that your tutor was actually your friend. And your tutor wasn't the person doing the exams. So you would do problems because you wanted to learn and your tutor would help you so that you would do well on the exam that other people were setting. So you had a nice alliance with your tutor.

AUDIENCE: Does that work well?

PROFESSOR: I would say it works well. At MIT it would work great. Given [INAUDIBLE] that people are too overloaded. And they would say, oh well, if there's not going to be any grading at all, I don't have to do anything, while I'd really like to do this problem set. But I have four other classes that are making me work 20 hours a week. And I don't have any time.

So then because of the other pressures they might fall away. But they would like to do it. So that's why I compromise by just saying, you have to do the thing and make an effort. And then I trust that once they start trying it they'll enjoy doing it.

So I tried to make it as light as I think you can get away with at MIT, given the pressures from all the other teachers. But in the ideal world I think that's what we would do here. There wouldn't be that pressure.

AUDIENCE: [INAUDIBLE]?

PROFESSOR: Yeah, in Oxford and Cambridge it works very well, provided you have a good tutor. And there's many students who are there just because it's a thing to do, to go to Oxford and Cambridge. But if you factor out those students who are just there for the stamp of the place, for the other students it works fantastically well. I mean, the tutorials are a great discussion time. You can really kindle people's curiosity because it's really not based on grading, it's based on helping.

AUDIENCE: [INAUDIBLE]?

PROFESSOR: It's so hard to do. I mean, I don't know of any comparative studies within one culture. And I don't think you could really do them well between, say, the English universities and the American just because there's so many other social variables that are so different like the population that goes to university, which universities are talking about the exam system itself, so I don't know of any studies that have been done across cultures and that I would trust.

And within I don't know of any. But it's a good question. I know studies in general about rewards, which is that the more you reward people for things that they like the less they like doing them. So I might give you some readings for that for very last penultimate session.

And for the reading that I'm going to give you for the next session is going to be about tutoring, actually. So I'll put that on the website today. OK, questions? Yes?

AUDIENCE: [INAUDIBLE] of a question. So I was wondering, if you do exams or homework problems throughout the semester, you can, maybe, also better monitor how the students develop compared to having one big exam on the end. And then maybe using [INAUDIBLE] lectures.

PROFESSOR: Right, so having continuous assessment is one phrase for it. It allows you to monitor, make sure things are going OK. So all of a sudden you don't find, oh my god, the person didn't know anything.

They got a third, which is like straight D's in your whole undergraduate. So in Oxford and Cambridge things are graded on first, second, third. And all of a sudden you had had no idea the student was going to get a third, or they were sick or something like that.

So I think, yeah. So homework problems and exams have a really good purpose in assessment and helping students get feedback on how they're doing as well. So for that purpose, grading is useful. And one solution to that dilemma-- because I don't like to grade exactly because it stresses them out with the grade-- is to give them the solution set right away.

So they can get feedback from the solution set whether they understood things, not whether they had a 90 or and 80, but whether it made sense to them. And that's, I think, a much more important thing. But then the other part of grading, which is that it's used publicly-- as part of your record and written to other people-- that part of grading I think is not right.

I don't think it's morally right to, basically, reveal information about students to other people without their permission. I don't think that should be the purpose of the university. But the other purpose of grading, helping students know where they are. And helping us know where they are so we can help them more, I think, is a completely valid purpose.

OK, so there was another suggestion which was to use as many non-physics and math examples as possible. So I'll try to reform my ways for the non physicists and mathematicians. It's just those come most naturally for me. OK, so I think that was the main-- yes, question?

AUDIENCE: [INAUDIBLE] for evaluating the students or for teaching the students.

PROFESSOR: Pardon?

AUDIENCE: Homeworks?

PROFESSOR: Yes, for evaluating versus for teaching?

AUDIENCE: Yes, because that's one thing to evaluate, right? [INAUDIBLE]?

PROFESSOR: It's both. So there's two, yeah. So the other purpose is that they learn and they do things. That's for sure.

So you want them to do that. And you can do that in class and homework. And the take-home exams that we had when I was an undergraduate had that purpose too. You spent 24 hours really learning stuff.

So that purpose is very valid too. But it's also very valid to know where the students are. So for example, that basically closes the feedback loop. You're doing all this stuff. And now you're seeing the result of it.

And you want to know, well, did any of it take? Did any of it stick? And if none of it did, well, maybe you want to change what you're doing.

AUDIENCE: Oh, but you cannot evaluate a person every week, right? And that's what drives students to copy solutions because they care about the grade. [INAUDIBLE].

PROFESSOR: That's why distinguish two senses of evaluate. There's evaluate for the purpose of helping the students. And there's evaluate for the purpose of publishing their grades elsewhere.

And it's the publishing their grades elsewhere that worries them. So you want to minimize that part. And you want to do as much evaluation towards helping them as possible.

So a solution set-- for example, one thing you can do is say OK, here's the solution set. Ask me about anything that's confusing. And then you can put some of the responsibility for this evaluation onto them so it's self evaluation.

But you're still using the homework for evaluation. It's just you're not trying to say look, that's the main thing. I'm going to use that to tell everyone else about it. That's what worries them and produces copying.

So you want to minimize that as much as possible. And it's hard. And that's the subject of the political obstacles to educational change session that'll be in the second-to-last lecture.

OK, so yeah, I don't pretend any of these have easy answers, unfortunately. Oh, there was another suggestion, which is that I make a web forum where people can ask questions afterwards. Because it's hard sometimes think of a question on the fly. And that's true.

And the problem is it's not easy to make web forms on the MIT.edu site. So I'll try to figure out a way to do that. But for security reasons you can't really easily make web forms that just append to a file or do something like that, although I'm going to look into it and see what workarounds I can find. Yes?

AUDIENCE: [INAUDIBLE] scripts?

PROFESSOR: Yeah, you can use scripts at MIT.edu. So I have to figure out how to use that. But yeah, I had that in the back of my mind. But thanks for pointing that out. Yeah?

AUDIENCE: What about forms?

PROFESSOR: Pardon?

AUDIENCE: Forms?

PROFESSOR: Forms?

AUDIENCE: [INAUDIBLE]?

PROFESSOR: Yeah, so then you get email. So the only problem with that is that you get a ton of spam. So there are all these spam bots that float around and just fill out forms and send junk emails all the time. So maybe that can be filtered out and it's not so bad.

So but yeah, that's one of the few things you can do on MIT.edu, directly on the web.MIT.edu. So there's a very solutions to it. I'll try to think of something around that because it is a good suggestion. OK.