Lecture 5: Course Design

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: Welcome to Course Design today, how to plan a course and avoid the pitfalls that many courses fall into. In the past few sessions we've seen some of the results, for example the misconceptions, the rote learning, the lack of transfer. And how do you design a course so that your course doesn't commit more of that and contribute to that problem, but instead perhaps mitigates it? And it's a difficult problem.

So related to that, sort of flowing almost directly from course design is that making large changes to courses is politically very difficult. And several people have asked me questions about that. So actually the penultimate session will be political barriers to educational change.

So for example, the Benezet experiment, what happened to Benezet's experiment? Why isn't it done more? So actually, I went to the Manchester School Board, Manchester, New Hampshire, and looked at all the school board minutes from the 1930s to see what happened.

So I'll talk a bit about that, talk about what are the structural obstacles to change so that you know what you're up against so that you can make strategic, sly choices and not get yourself shot at too much. But that's sort of a consequence of good course design. And that we'll do in the penultimate session.

This time we'll talk about if you had your ideal world, what would you do? Sort of a process of successive approximation. In the ideal world what would you do? And then in the real world, how do you get as close to that as you can?

So today, course design, which is how do you design a course so that you avoid the main problems with the traditional courses? So now before I say how do you design a course like that, let me review a bit. We've seen some of it in the previous sessions. But what are the problems with the traditional courses?

So the problems in the traditional courses in broad outlines are, one, rote learning. I think that's the most fundamental problem. So we've seen many examples of that. And I'll just remind you of a few. And I'll show you a couple. So I'll remind you of a few just to summarize it in one second.

And related to rote learning is that there's no transfer. So what are the causes? Well, we'll come to that in a second.

But for example, what do I mean by rote learning? So rote learning is the fundamental problem. And it actually causes this one. If you learn by pattern matching, you're not going to be able to transfer to a new situation unless you happen to have by luck seen that pattern before. But any really new situation will be completely unfamiliar and you won't be able to do it.

So the examples-- so the canonical example, because I find this actually very helpful to have a canonical example of what is rote learning, I think this is the canonical example. It's that. So I have that picture in my mind and the whole story behind that picture is this one. So this is from Wertheimer again.

So students are learning to find the area of this parallelogram. And they move that triangle over there, make a rectangle. They know how to find the area of the rectangle.

So rote learning, you can test for rote learning by giving them this picture. And the students who didn't understand but it only seemed like they understand, they take the triangle and they don't know what to do with it. So they can't find the area anymore, because what this reveals is that they didn't actually understand what they were doing.

They didn't understand that the purpose of this procedure was to make a rectangle because you know how to find the area of a rectangle. They thought what you do is you chop off the left triangle, move it over there, and then you do some w times h. So operationally, it looks almost exactly the same. But cognitively, underneath, what's going on under the hood, is something completely different, you find out, when students have done rote learning, because they can't do this problem.

So this one actually has all aspects of rote learning. So if they can't even do this problem, they haven't even done any learning. But if they can do this one but not this one, they've done rote learning. OK.

So now if they haven't done any learning, that's a completely separate problem, generally not the problem we have. We have problems with students who have done way too much learning, but it's all in the wrong direction. So that's one example.

Have I shown you folks-- probably some of you, the Army bus problem? The Army bus problem is very amusing. So there are 1,128 soldiers. And they have to be bused from one base to another.

So this actually, again, was another National Assessment of Educational Progress problem. So it was given to a sample of about 50,000 high school students. So it was a huge study.

And there's 36 per bus that you can take. So the question is, how many buses? So it's not a rocket science question. They're just supposed to figure out how many buses to use and maybe one is the right answer, maybe. But you just keep shuttling back and forth. Let's assume that's not what was looked for.

So I'll just give you this. So 1,128 over 36 is 31 and 1/3. So now I'm giving you that basically to simulate the state of mind of the students, because it turns out about 70% of the students did this division correctly. So now there's another 30% who didn't even learn division correctly. Let's leave those people aside.

Of the 70%, in other words most of them, who could do the division correctly, what did they say? OK. So take a minute. Find a neighbor and see if you can predict what the students say and what the distribution of answers is. So this is practice understanding where students are coming from. This is American high school students.

OK. So can someone suggest one answer that people gave? Yes.

AUDIENCE: Actually 31 [INAUDIBLE] buses.

PROFESSOR: Right. So that was one of the answers. So actually, the way that in America at least when I was in school you were taught to write it as 31 R 12. So that means 31 remainder 12, which I think is a completely broken way of doing things. But that's what we learned in school. And that's what some fraction of students wrote.

OK. Another answer.

AUDIENCE: 31.

PROFESSOR: 31.

AUDIENCE: 31 because they round off.

PROFESSOR: 31 to round off. And another one?

AUDIENCE: 32.

PROFESSOR: 32 would be nice. Right. So these in fact were the three answers. And they were all equally chosen, except for this was sort of slightly preferred to the others. So they were all roughly 25% of-- this was, say, 75%. These were all about 25%. I think this was 28%.

So this one was the winning one. Now just stare at that for a minute. What does that tell you? It tells you that-- what it tells me is that the problem had no meaning for the students. Again, they'd done learning. So of the rite learning, they'd done learning. There's no question about that.

They'd learn how to divide. They all could divide. But they didn't know what the hell they'd done. So if you wrote, oh, yeah, if you filled out on the requisition form for the Army I need 31 R 12 buses, where would you be? You'd probably be peeling potatoes for the next week, on KP duty.

So this shows no meaning. And what it shows is that for most students, there was actually no meaning. Here, again this also shows no meaning, but in a more sophisticated way than this. At least they thought about what to do with the R 12. And they came up with the wrong answer, but they thought about it. And here, we'll hope they actually understood.

So of the 100%, 30% couldn't do the division. Roughly another 20%, it had no meaning. Another 30%, no meaning. So now we're at 80%. And only about 20% really understood what was going on in a really fundamentally simple problem.

So that is a striking example to me of how deep the rote learning is. It's the same as that problem I showed you last time, which was 3.04 times 5.3, I think it was. You had to multiply. And students had to choose-- they had to estimate it. They didn't actually have to do the calculation. They had to choose between 1.6, 16, 160, one of these four.

And it was just chance, basically. So again, the number system didn't mean anything to people. So it's really fundamental. It's really terrible. And it's been studied a lot. So the problem is not just in American teaching. It's across the world.

It shows up in physics. So the physics example I showed you was this. So this is the bouncing ball. So it's stationary just for an instant while it's bouncing.

And what everyone drew was that there was-- normal force equals mg and there's mg. Why? Well, v equals zero. So there's several reasons. Here, partly there was rote learning involved, because n was always equal to mg in all these other problems. So they just triggered off of that.

The other one is that there is indeed misconception. They actually think f equals mv. So that's the third.

So here, there's deep rote learning. There's no transfer. What are the causes? So there you see one of the causes.

AUDIENCE: Sorry. [INAUDIBLE]. Couldn't it also be a problem of multiple choice thing, because [INAUDIBLE]?

PROFESSOR: It wasn't multiple choice. They were just given the problem just like this. No. I don't think anyone who made a multiple choice would ever have dreamt to put that one on there.

AUDIENCE: Well, I mean this is [INAUDIBLE].

PROFESSOR: Yeah. But I think it was phrased like how many buses should you order, something like that. I mean, they should come up with some integer.

AUDIENCE: They would write in an answer that they would get.

PROFESSOR: Yeah. No, it was a blank, not a multiple choice. It was a short answer. So that's how they could see that they'd done the division correctly or not. And so all of the ones who had done the division correctly, this is what you got. Another question? No. OK. Question.

AUDIENCE: So I have another question. You've shown us many studies that prove that there's more rote learning going on. And all those studies come from the '30s and the '60s. So why is it still a problem today?

PROFESSOR: Oh. So this one was '84, I believe, '83 or '84. And let's see, when were some of the others? So Wertheimer was the '50s or the '40s. Those examples are the '40s, so yeah, you're right.

AUDIENCE: Why is it still a problem?

PROFESSOR: Well, every time I ask this one, I get the same results. So this I've asked between 1998 and 2009, and always got the same results. What are some of the others? So another one is quel age est le capitaine? So that's a French study. So they asked students-- the question was, how do we know it's still a problem today?

So they've asked students, there are 26 sheep and 10 goats on a ship. How old is the captain? It's a good question. The problem is the answer. The answer is that people answer 36, because they have to stir the numbers around somehow and get something. So that was done in the '80s, I believe. Yes?

AUDIENCE: That's not quite what I'm asking. I'm asking, if there is so much evidence that supports the existence of the problem, why isn't it solved?

PROFESSOR: Oh. OK. Good question. OK. Sorry. I thought your question was, wasn't it all done long ago so it's not a problem now. Oh, yeah. If there's so much evidence to support the existence of a problem, why isn't it solved?

So I would say there's two parts to that answer. One part is that, a, lots of people don't know about this. So normally when you teach-- so there's sort of an implicit contract, which is that if I don't ask you questions like that, you won't reveal to me that I'm not teaching you any of these important things.

So for example, if you just take a regular course, it's actually possible to do all the problems by pattern-matching, mostly. So as an example in physics, you'll have-- say you're doing circular motion. And someone will say, well, there's a car going around a turn and the radius of the turn is this. And it's going at 30 miles an hour. What's the force? Or the car has a mass, m, what's the force that the road has to exert to keep it in the turn?

Well, you don't have to know anything about circular motion. All you have to know is that there must be some formula which has-- so let's look at the variables that I just said. There is F is the unknown. m is given. v is given. And r is given.

So you just flip through the book until you find a formula with all of those guys, which happens to be F equals mv squared over r. And then you just plug everything in. So you can do all those problems without understanding anything about centrifugal and centripetal.

For example, 95% of students who can find this formula with no problem, if you ask them, well, can you draw me a force diagram, 50/50 on whether the force is outwards or inwards. You can ask them things like, OK, a car is accelerating. Where does the force come from? Or a car is decelerating. Where does the force come from?

Oh, you know, the road. That seems really strange. But it's true. It's in fact from the road.

So they have huge misconceptions even about linear forces, let alone circular forces. So the regular way of teaching actually doesn't expose those. Yes?

AUDIENCE: Do you think if the standardized exams had higher-level questions on them. like Bloom's Taxonomy, would that help [INAUDIBLE]?

PROFESSOR: Yeah. That would help, actually. So I can say something about how-- maybe I should've said this last time, about how-- so the question was, if standardized exams were better and had higher-level questions, would that help?

I think it would help. And you can still even do it multiple choice and still have higher-level questions. So here's actually a pretty striking example. Let's see if I can remember exactly. Oh, yeah.

So you ask students-- these are young kids, but it applies generally. And it's a question about how to design tests. So you ask them, I have a piece of red cellophane and blue cellophane. And I put them together. What color do I get?

OK. So now the question isn't so much what is the right answer? But what do you think students say when you ask them that? And these are young kids.

So takes a minute and see if you come up with what they say as their answer, their most common answer. So you ask this as an open-ended question. You don't give them multiple choice. So talk to your neighbor about that.

OK. So I think it's actually very difficult to predict what students actually say. And I'm willing to eat a chalk if anyone-- well, I don't know if I should say that. That's a bit risky. But let's just-- chalk is probably safe.

But let's say I'm willing to eat a chalk, maybe with salt and pepper, if anyone can actually figure out what the students actually said, the main thing they say, because it's really, really counterintuitive. Yes? Oh, go ahead.

AUDIENCE: Reddish-blue.

PROFESSOR: OK. So reddish-blue is one thing. Yeah. So that's one answer they give. Yeah. Yes?

AUDIENCE: Brown.

PROFESSOR: Brown. Yes, they probably do say that.

AUDIENCE: Reddish-blue is like violet?

PROFESSOR: Reddish-blue is like violet. Yeah. But if they are young enough, they might not know of the color violet. They might, or purple. Yeah.

AUDIENCE: I'll go with saying violet-purple.

PROFESSOR: OK. So violet-purple. Yes?

AUDIENCE: Yellow.

PROFESSOR: Yellow. I'm still safe. So they do say all these. I don't know about all of them in particular, but they say most of these things. I'm so far safe from my chalk wager. Yes?

AUDIENCE: Blue.

PROFESSOR: Blue. OK. Yes?

AUDIENCE: Black.

PROFESSOR: Black. That would be nice. Yeah. Yes?

AUDIENCE: Would they say it depends which one is on top?

PROFESSOR: Uh-oh. I've got to eat my chalk. Yes. So does it matter which one's-- yeah. In particular, they say, which one's on top? Can I have salt with that?

So that is in fact the main question. So now why do I bring this question up? Well, the normal way you'd make a multiple choice question is you'd ask the question yourself.

You'd think, well, what are the plausible answers? You'd come up with many of the choices here. And you'd put them down in the multiple choice. And you'd never think of putting this one there.

So one way to actually design the questions is that you first do it with a pilot group with a short answer. And then you make your multiple choices based on the main misconceptions that they have. OK. So now what's the obstacle when you do this?

The obstacle when you do this is you get this very funny item response curve. So the item response curve is the following. So the idea response curve you get, so it-- OK.

So on this axis is the student's overall grade. And this is the-- so you have a test say, of 50 questions or whatever. And this is their overall grade on this axis. On this axis is their probability of getting one particular question correct.

So now, a usual, a normal question is something like that. If they don't know anything, they have no chance of getting it right. Maybe they have some monkey line level over here. So let's call it something like that.

OK. But these kind of questions actually have a very different shape. Their shape is shaped like a J. So what happens is that-- so this is the good questions. I think they are much better questions. But they get this shape.

So what happens is when the students are just guessing, they're somewhere down here. But as they start to think about it and learn more, then they fall into the misconceptions. So they actually go down. So they actually can even be below chance. And then only when they really understand things do they get up here.

So now what happens, for example, if the SAT ever contains a question like this, they just throw it out. They say, oh, this question does not measure how good a student we have. It's actually skewed. It's all messed up. So you throw it out.

So these normal, standard ways of making standardized tests actually throw out all the good questions. And all you're left with is these kind of questions. So I actually saw an example of this in Cambridge, which was-- so you remember the rolling down the plain question that I showed you last week or the week before?

So the rolling down the plain question is an intuitive question. It's qualitative. It's hard. If you try to calculate it, you'll get yourself in a knot unless you know exactly what you're doing. But if you reason about it intuitively, you can do it quite well.

OK. So that question was then put on the end-of-year exam for the sophomores. And the examiner said, well, actually this question was not a good question, because it didn't measure who the good students were. So what they meant was that the good students actually sometimes did worse on that question than the not-so-good students.

So they said, OK. We're just going to throw that question out. I mean, they didn't throw it that time. But we're not going to use questions like that next time.

So again, here you had a good qualitative question. And it didn't measure the quote, "good" students. So it didn't give them what they wanted. So they threw it out.

So if you design a test like that, you're going to fill it with questions that are just sort of low on Bloom's Taxonomy. But it doesn't mean you have to do that it way. So if you want to make a test that is higher on Bloom's Taxonomy and tests misconception and qualitative reasoning and intuitive reasoning, the way you do it is you do a short answer first, put the main misconceptions down. And you just have to be willing to live with this kind of curve.

AUDIENCE: But when you say you don't evaluate the good student, but the question is how do you in the first place evaluate the good students if you have a normal kind of system? How do you basically evaluate them?

PROFESSOR: How do you tell who the good students are?

AUDIENCE: Yeah.

PROFESSOR: Well, it's a reification that's going on. I mean, there's this idea that there's such a thing as a good student. And just because you call it that doesn't mean there is such a thing. So you need some definitions.

So like for example, what is IQ? The definition is it's what IQ tests measure. I mean, that's the only, I would say, valid definition of IQ tests. So here, good student meant doing well on the standard problems.

Now, I don't think that's a good definition of a good student. But that is the implicit definition in most courses. So to answer your question, it's the standard way things are done, the implicit definition that it makes it so hard to change the rote learning, but also so hard to even see that it's going on.

And then the other half of the answer is that-- and when you try to change it, you find all these political obstacles. And that's what we're going to talk about in the penultimate session. But for example, people downstream. And one question was raised about this in the questions. The next courses will say, well, you didn't teach them X.

Suppose you say, oh my god. I'm teaching them sophomore mechanics and they don't know Newton's three laws. Well, I could just do oiler's top and the herpolhodes and the polhodes, and all of that stuff and gyroscopes and all this complicated stuff, which is sort of a nonsense if they don't understand Newton's laws. Or I could actually sort out their misconceptions on Newton's laws.

Well, suppose you take time to do that. Well, then they haven't learned what the herpolhode is. So who knows what the herpolhode is? So I sort of remember a while ago. But even I taught the course and I hardly remember it.

But the next downstream course will say, they didn't know the herpolhode because you spent time on Newton's laws. So that's the political problem. So you have to then somehow get the people downstream to see that this is a serious problem, that we can only build sand castles in the air for so high before they collapse because the air can't support that much. And that's what I'm going to talk about in the penultimate session. Question?

AUDIENCE: [INAUDIBLE] problem, was the purpose of the test, because if the purpose of the test is to evaluate the students, you don't want to throw in tricky questions.

PROFESSOR: Well, so the question is which question you use, does it depend on the purpose of the test? It does. But I think you absolutely need these kind of questions, or you're not actually evaluating the right thing. So if you only put these kind of questions in, you're not exposing any misconceptions. So you're actually not evaluating some of the important questions about what the students understand.

Because, for example, if you just give them colors on this question, you'll never realize that that's what they're thinking about. So actually you're just evaluating can they give you one of the pre-prescribed color answers. But that actually doesn't necessarily mean that they understand. So you're not actually evaluating their understanding.

So to evaluate understanding you do need questions like this. And should you punish them for being down here? No. That's the other part of it.

If they're down here, that means they're thinking about it. So if you then grade them and they came down here and you say, oh, you did really badly, no. Actually you're down here, that's because you're thinking about it and you got caught by the misconception. But that's the only way to sort out the misconception. You have to go beyond it.

OK. Other questions? And then we'll take a break. OK. So it is 10:05, actually, according to that clock, plus an hour.

So 9:15, according to that clock. So 10:15, see everyone back here. Take a short, 10-minute break, just as if it were two 1-hour classes with the 10 minute break. And then I'll put the feedback sheets in the corners. So when you come back, just make sure you grab a feedback sheet.

OK. So before, we talked about the problems with what most courses do and how rote learning is a primary product. And as a consequence of that, there's no transfer to new situations. So the knowledge is very brittle.

So what are the causes? And what can you do about it? So just briefly, the causes partly are no account is taken of the misconceptions students have. So new rote learning is piled on old misconceptions, because by definition learning is going to turn into rote learning if there's no understanding and meaning given to it, just like the 1,128 divided by 36 is 31 R 12. So no account is taken of what students have rote learned and what they've really understood, because the way the course is normally taught, those ideas, those common misconceptions aren't exposed in the normal run of the way the course happens.

And another is-- so that's directly related to the problem. Another one is that the applications are very narrow. So because the applications in the class are so confined to the same area, it's very easy to produce rote learning.

And so I have a-- I keep saying how pictures are the way to understand everything. So I have a picture for that. So I show everyone this picture. And if you keep this picture in your mind, it'll give you a guide for how you should make problems and what examples to choose.

So suppose there is some idea you'd like to teach. We'll talk in a bit about why you'd like teach an idea at all, but some idea you want to teach. And you have a clear conception of this idea. For example, maybe the idea is something like DNA is the fundamental unit of information in living systems.

OK, well that means something to you. But to the students, it doesn't actually mean anything. You need to give them examples. So you're giving them an example. A related one is you have some idea, which is dimensional analysis. And you want to give them an example of doing it, because just stating the Buckingham pi theorem for them doesn't really help them. So you do an example.

But now picture yourself back in the student mind. What do the students see? Well, they don't have this idea as a separate unit yet, because they're still students. And the purpose of the course is to teach that. So they haven't got yet.

So they have this merger of idea and example together. So they see this big package and have trouble making that division between idea and example, in other words, between transferable idea and thing that was particular to the situation. So of course what do you do? You give them another example.

So the normal way it's done is here's your second example. So there's the second example. But now, again from the student's point of view, there's still quite a big overlap between idea plus example one and idea plus example two. So that overlap is-- if they intersect things mentally in their head and see what's common, they still get a lot of stuff that's beyond just the idea.

For example, if you do a bunch of dimensional analysis examples related to entropy, then they'll think, oh, dimensional analysis is a way of solving entropy problems. So here, the overlap is too big. So what do you have to do?

AUDIENCE: More examples.

PROFESSOR: Pardon?

AUDIENCE: More examples.

PROFESSOR: More examples. And where should they be in this space? Yeah, way around, right? So you should make an example over here. So this example has very little intersection with the other example. So here, this one and this one and this one, they all have very little overlap. So now the overlap among all three is pretty much just the idea.

But think how different that is from the way most courses are designed. Most courses, the examples, you'll do 18 problems with the ideal gas law. You'll do 18 integration by parts. You'll do 18 solutions of first order differential equations. You'll solve 18 ion channel problems with a Nernst equation.

All of those actually don't teach you the fundamental idea to be able to transfer it. So that's another cause, see, that the applications are too narrow. So you have to choose your applications as widely as possible. And this makes teaching very hard, because it means you can't really-- yes, question?

AUDIENCE: So it can't just be like also the structure involved with it. it's defined as having a class and having problems. So obviously, do something in written. But you never really communicate that. And if you would have like let's say in physics, you would have [INAUDIBLE]. And maybe you have an exam, an oral exam where you basically can evaluate the students and really can find out if the students understood the main idea. And then he has to be transferred, transfer the skills. So I mean in terms of like broadening the way of how you question the students will help basically get the idea across.

PROFESSOR: I think that's right. So the suggestion was that oral exams are actually-- in general, it's true, are a much better way of evaluating whether the student understands or not. It's very disconcerting for the students, because, I think I said this last time, basically in an oral exam, as soon as the student understands something, you ask something else. So you're like, OK. They know that. Let me ask them about something else.

So the student feels like they're never talk about something they know. But that's the purpose of it. So yeah, an oral exam, like a PhD qualifying exam, forces transfer. That's what you're checking for in the PhD qualifying exam. So if you do things like that in the undergraduate course, that's setting a goal of transfer, that this is one of the goals of the course, that you be able to use these skills in new situations I'm going to ask about.

But of course, to be fair, you have to give them practice in doing that. You can't just give them regular problems and all of a sudden give them an oral exam where you say now transfer it to a new thing. They'll just hit a brick wall. But it helps you in designing the course. So I'll put the backward design diagram up again, to show that process.

So here, you want to choose from as wide a set as possible. And generally, that's not done. So I wanted to read a couple of selections from the Eric experiment about what happens in class. So I know you had a choice of reading. So who chose the Eric experiment?

I thought they were all fascinating. So if you remember, there was a couple quotes from Eric himself. So for those who didn't read it, Eric was a non-physics student who took a summer session physics course freshman, physics course, and recorded-- I think he was a philosophy major. And he recorded his impressions of what it was like to be in a physics class.

So that's actually-- you get quite a different view of what a physics class should be when someone from outside comes, sort of like this diagram all over again. He said the class consisted basically of problem solving and not of any interesting or inspiring exchange of ideas. The professor spent the first 15 minutes defining terms. And apparently that was all the new information we were going to get on kinematics.

Then he spent 50 minutes doing problems from Chapter 1. He was not particularly good at explaining why he did what he did to solve the problems, nor did he have any real patience for people who wanted explanations. You know, the why, the what were the ideas, why would you do it this way rather than that way, those weren't generally part of the course. So by not making them part of the course, you're actually pushing towards their rote learning.

And that doesn't mean that there are not all these interesting questions. So later on in this selection, he says, well, actually, you know, I wonder about all the names that physicists give. But there was no space to ask those questions in class.

He says action/reaction, that's Newton's third law, that presupposes a cause and effect relationship, which implies duration. But in physics, actually action-reaction happen simultaneously. So those are the kind of the questions that he's asking himself, Eric, the student, that get away from rote learning. But the structure of the class doesn't encourage anything like that.

And the only reason he has time to think about that is he's not actually trying to just survive the class. He's sort of going in as an experimental student, whereas the students who are just trying to survive the class, if any thoughts like that occur to them, they just say, forget it. No time to think about that. i have to solve the 10 problems on the end of Chapter 3 right now.

So those are the causes. What do you do about it? Well, there are several principles of design. So last time I talked about two organizational principles, the signals and systems model of how to teach a course. So this was the teaching equation.

So this is the output, what students can do. So you choose that. You also have to understand students', students' thinking. And then you figure out your teaching given what Given you want them to be able to do and how students think, the misconceptions, how people learn, visual, by examples, diagrams like that, how should I teach? So that gives you an overall structure for course design.

The other structure, which is quite similar, is backward design. So backward design, you invert the normal order. So the highest level in backward design is-- that's the and over there.

So this is a hierarchical structure. Basically, you choose your course goals. You find ways to make them operational. So in other words, how would you measure, for example, being able to solve Newton's laws problems in designed bridges, for example, might be one of your goals. Well, you would actually give them some bridges to design, things like that. And then given that you want them to be able to do that, what do you do in class?

So this was last time. That's this time. And this is next time.

And as I said before, I start here, just because I find this is at the right level, balanced between concrete and abstract. So we did that last time. But now we're here. How do you choose course goals?

And that, the paper by Middlebrook-- so who read Middlebrook's paper? A few people. OK. So if you haven't read it, I didn't want to overload you with reading, but I think it's also a brilliant paper. And I know Middlebrook because he was my teacher at Caltech. And I took the course that he describes in the paper.

And he identified a fundamental problem and a fundamental solution to it. In other words, how do you choose the goals? And I myself have actually come to pretty much the same conclusion through years of hard experience. So this is all to answer, what do you do about it? Which is, you choose course goals such that the course itself will produce transfer.

So let me explain the problem that Middlebrook identified and what his solution is. So his particular class is an analog circuit design class. And generally, the way analog circuit design is taught is you have a circuit. And you ask the students to analyze it.

For example, you give them a circuit that filters out high frequency noise, for example, in an audio recorder in a CD recording studio. And you analyze it. And you see that its behavior does do-- what is its behavior? OK. It throws away high frequencies.

OK. So now that's the general run of the mill. You'll just get tons and tons and tons of problems like that. And you learn a wave of analysis that allows you to go from here to there. But then what happens when the student graduates?

What happens is what Middlebrook calls a fall off a cliff. So either they go into graduate school or into a company. But either way, what happens is they're actually given that. Somebody says, well, actually, I need a circuit that doesn't cost too much but is quite exact and cuts off all frequencies beyond 22 kilohertz and doesn't mess up the frequencies below 20 kilohertz.

This is actually a real problem. That's how you need to do that to record. The people who developed the CD standard had to make something like this. So basically you're making a, quote, "antialias" filter. So I want that.

Now here, the problem is that the student has learned methods of analysis that cannot be inverted. For example, one method of analysis is you have a circuit and you just simulate the thing. You just feed it to your computer, which does all the analysis for you.

An analogy for that is if someone gives you a function to graph, you can use your graphing calculator to see what the graph looks like. But suppose someone says, well, I want a graph that looks like this. The graphing calculator can't answer that question. So they've learned methods of analysis that are not invertible.

So they're actually useless, because the real world problem is exactly opposite of the school problem. The school problem, the circuit is the start. In the real world, the specification is the start. So to have any hope of going from there to there-- this is a one-way arrow, it's like a diet-- you need a different method. You need to learn invertible methods of analysis.

So if you have invertible methods of analysis, you can go both ways. And this problem I think is central to many, many, many, many courses, not just to this particular analog circuit design course. So then, what Middlebrook slowly evolved to was not just teaching invertible methods of analysis, but he made those the foreground of the course.

As he described, "I found in teaching this stuff for a long time, I've actually had to put names and words to ideas for the students to get hold of them. This came about because for a long time I taught by example. I used a lot of methods, saying we're going to look into the properties of a certain circuit. And I would use some shortcut method or trick that I thought was a neater way to do it. And I just showed them this without actually emphasizing the method.

That was fine. But when they did the homework problems and exams, their minds just automatically reset to the old way of doing it. So that didn't work very well. And I finally realized I had to give names to these things."

And those are the names that he introduces in the paper. So those names, some of those are specific to analog circuits. Some are general. But it's the giving of names that is the central [INAUDIBLE].

So you give names to the big ideas in the course. And you don't just use them in passing, but you make them the focus of the course. So let's see. So in answer to what should the course goals be, roughly speaking-- OK. Questions about that, because it's very different from how it's normally done?

So if you look, for example, at a biology course, it might be organized by, say in cell biology, organized by the organ to organelle, the little system inside the cell. But those aren't necessarily transferable to anything except that particular subject. A physics class might be organized by a whole bunch of kinematics problems and dynamics problems. But a big organizational principle that is implicit there but isn't said is that, well, actually what we're doing is we're approximating down to something simple enough that we can do. That's an organizational principle that you can transfer.

But that's never said. It's just implicit. If you look at Chapter 1, the systems are simpler. And then Chapter 2, they get harder. Maybe they have two dimensions in them. In Chapter 5, they have two particles in them. In Chapter 7, they have n particles in them.

So it's this process of successive complication. And the beginning part of it, where we simplified real-world things into just one particle moving in one dimension at constant acceleration, that was never told to the students. So the transferable principle just went straight by them, because we never made it explicit.

So you have to, if you ever want to have any hope of teaching these ideas, you have to make them explicit and give them names. Giving things names is absolutely essential to have any chance of learning them. And this was known in all the old legends.

The gods did not want the mortals to have names for the gods, because it would give the mortals too much power over the gods. So some gods were He-who-has-no-name. And maybe only the high priest knew the name.

Well, you don't want to be the high priest. You want to share the names with the students. And that involves a lot and a lot and a lot of thinking. That's where most of the work in course design goes, is figuring out those main ideas, those transferable goals and ways of reasoning that students can use no matter what they do.

And then to teach them, you need to have it from as wide a set of examples as possible. So for example, to give you an illustration, when I teach dimensional analysis, which is the idea, and I want them to be able to use it wherever, the first example-- I forget whether it's the first or second. i sort of flip it around. The first example is in economics, because you can identify a lot of bogus stuff in economics, because it has the wrong dimensions. So I want them to see it there.

The second example is solid geometry, volumes of tetrahedrons and cones and things. And then we do sort of a more standard physics one. So I want them to see the broad sweep of this idea so that the idea comes clear. So the thing to do is to identify large ideas. So these are the--

OK. So to practice, choose right now a course in your own mind that either you are interested in teaching or teaching now or would like to teach. And see if you can come up with some big ideas for that course, just to practice what some big ideas are.

And I will show you as an example the large ideas for my-- just to give you an idea of what they are. I'll show you the tree for my approximation course just so you can see what level these ideas kind of live at. So the whole course is designed around how do you deal with complex problems and how do you handle the fact that the world is really messy?

And I broke that down into either you manage it or you throw it away. Those are still too high-level to actually do anything with. But I wanted to organize the big ideas as well. So you can manage it by divide and conquer reasoning, so subdividing into smaller problems. Or you can make abstractions. Abstractions is basically the process of giving names to things, which is what we're talking about now.

To throw away complexity, you can either do it losslessly-- so this is methods of reasoning that find complexity that isn't there but look like it is there and get rid of it. So symmetry is number one. And examples of symmetry reasoning are proportional reasoning and dimensional analysis.

So those are all lossless ways. They don't actually cost you anything. If they work, they make your life so much easier. And you haven't thrown any information away.

Or you can make lossy compression. So you can do extreme cases, look at only the simple cases of things so you're throwing out information. Or you can make spring models. So even though things aren't-- like chemical bonds, you can just say, well, they're just like springs. So that's a very useful but lossy way.

Or the final one is you can lump. So you can lump changing processes into one. For example, a curve, you want to find the area under the curve, well, replace it by one rectangle. So that general pattern-- those are what I've found are the most useful ways of dealing with complex problems.

So these are the big ideas in that course. And the whole course is organized that way. And I show this tree once a week usually, just so people know where they are.

So now, depending on the course, what stuff you have in your big ideas will be different. But I just want to give you an idea of what some large ideas, some structuring principles are. And then the course itself is actually organized Unit 1, Unit 2, 3, 4, 5, 6, 7, 8.

So what I've found is, just like Middlebrook found, that not only do you have to give names to the things, but I found that I had to actually organize the course around them, rather than organizing them around the topics that they apply to, if I really wanted the techniques, the ways of reasoning, to transfer. OK. So questions about what these large ideas, these reasoning principles, are? What kind of things to look for? Yes.

AUDIENCE: What's the topic of the course?

PROFESSOR: The course is called The Art of Approximation in Science and Engineering. So actually, it's carte blanche to talk about anything. So that's why these are probably higher-level objects than in most courses which will be narrower.

But even there, like for example in physics, in the Intro Physics, one of the core ideas is successive approximation. And that's not brought out normally. But you could bring it out. You could say, look, we're using successive approximation again. Or here's the structure of the course, so that it's plain for the student. Does that help?

OK. So pick that course in your mind. And talk to your neighbor. And see if you come up with, say, two or three large ideas. And now what we're going to do is we're going to put them on the board, the reason being that you'll find that people's big ideas and large ideas from other courses you'll be able to use in your course, not every single one, but many of them. That's almost by definition what makes something a large idea, that it's not just limited to that course. So it helps to see what other people's are.

So I'll make some space on the board to draw those up. So I'll give you a chance to develop your list farther in the homework. Oh, I should say something about the homework, which is that you guys are all grad students, or almost all of you. You're all taking the course for fun. It's kind of pointless of me to police whether you're doing stuff or not.

But all the same, I will give you an assignment to do-- and I'll just trust that you'll do it-- which is to work with each other on making large ideas for a course and then figuring out examples for those large ideas, so basically a sketch of a course design And the reason to do this, starting now and also later, is that basically that is the hardest work of course design. Once you've done that, everything else sort of flows. The syllabus writing, all that stuff, just flows.

But if you haven't done this, you're just kind of spinning around. And you don't actually know what you're doing. It's really hard to actually figure out how you should start and what you should do. So, big ideas. Any big idea in any course?

AUDIENCE: Separation of scale.

PROFESSOR: OK. Separation of scale, so that behavior at one scale and another scale, you can often separate them and analyze them differently and without interfering with each other.

AUDIENCE: [INAUDIBLE] past problems and then [INAUDIBLE].

PROFESSOR: OK. Separation of scale. So for example, I've seen that done in neurobiology. It's done in physics all the time. So in neurobiology, the calcium flux, the calcium concentration changes slowly, so you do the calcium concentration separate from, say, the sodium and the potassium, which is done with the action potential spikes. So separation scales, very important principle across many fields. Yes?

AUDIENCE: Equilibrium.

PROFESSOR: Equilibrium. Is that right? So equilibrium, right away you can see if you really want students to understand the value of equilibrium, you have a whole set of examples to use, to steal from.

You can use chemistry, physics, mechanical engineering. And they all have slightly different senses to equilibrium, statistical mechanics. But there's some core idea that's the same. And so it also is an argument for team teaching, or actually talking to colleagues from different fields to steal examples from them. Yes?

AUDIENCE: I don't know how to describe this, but it describes a process which we all do research, and that is how to deal with scientific problems. [INAUDIBLE].

PROFESSOR: OK. So identifying a scientific question and how to start studying it.

AUDIENCE: [INAUDIBLE].

PROFESSOR: Yeah. Identifying and studying. Right. And there's many things that are common across many fields. And there's many things that are different in each field. And you can actually have a very fruitful-- I could imagine a really fruitful interesting-- you could almost teach a whole class on this. Actually, that sounds kind of fun.

You could teach a history of science class on scientific questions that were identified, how they were studied, what principles still are done today. You could have people read papers from across many fields and read the intro and see how they would study that question, see what questions there are. Yes. Sharon.

AUDIENCE: Resonance.

PROFESSOR: Pardon?

AUDIENCE: Resonance.

PROFESSOR: Resonance. Right. Resonance you can have a field day with. It's in electric circuits. It's in mechanical systems, the earth-moon system, the tides are driven almost on resonance by the moon and the sun.

Resonance is everywhere. It's in mechanical systems. There's beautiful films you can show of the Tacoma Narrows Bridge. I think the London Millennium Footbridge as well got hit by resonance. So resonance, another very transferable idea. Yes?

AUDIENCE: How to develop 3D models.

PROFESSOR: OK. So how to build models. Right. And that's something that there are general patterns to it. And some of it's specific to a particular course. But there are general patterns. And that's something the students can use no matter what they do later.

Whenever you go to a new field-- so there's a Nobel Prize winner at Harvard, Walter Gilbert. So he's worked in I think chemistry, physics, and biology, many different fields. And he said, oh, yeah. Whenever you go to a new field, there's only a few key ideas in that new field. And the hard work is figuring out what those are. But that's what you have to do.

So that is one of the transferable skills as well. Others? Yes.

AUDIENCE: Problem solving requires communication.

PROFESSOR: OK. So communication. Right. If you can't communicate, you haven't solved the problem, which reminds me of my Caltech students, when I had them keep a journal. And then I told them I wanted them to explain-- the journal they liked. But then I said, on the homeworks, I want you to explain your reasoning so that it reads. It's not just hence and thus and therefore.

And they said, well, why we have to do that? We're physics students. I said, well, later on, you're going to have to communicate your answers to someone. They said, oh, we'll just hire someone who does that.

So then I said to them, "well, how are you going to explain it to that person? How will they know?" I said, "oh you're going--" "oh, I'll explain it to them." I said, "oh, so you are explaining something?"

They said "oh, no." Somebody said, "oh, well, actually that person will understand it already." So then I said to them, "well, so that person understands it already and can communicate it. And you just understand it. Who are people going to turn? "

So only then did they agree that it was a good thing to actually write problem sets so they could communicate. So that's something you can teach in any field and across fields. You could look at scientific writing. You could look at writing scientific papers. You can look at giving presentations.

Doing it on problem sets so they communicate, you can have students exchange problem sets with each other. If they can't understand the other problem set, clearly it wasn't communicated. Other transferable ideas? Yes.

AUDIENCE: Coordinates.

PROFESSOR: Coordinates. Yes. So coordinates and-- it's also fun to make this kind of list, because you start to see commonalities and they start to suggest other ones. So when you say coordinates, I also think, oh, yeah. That's a general example of representations.

So there's different-- this is probably what you meant is coordinate representations, what you were thinking of. But you can also think of the general idea of choosing good representations for solving particular problems. Like for some problems, a symbolic representation may be right. And some, there's a pictorial representation. You want to use in circuits, the time domain representation, or the frequency domain representation.

So coordinates, I'll put dot, dot, dot, to say that almost leads to the next one, which is representation in general. Others? Yes.

AUDIENCE: Symmetry.

PROFESSOR: Symmetry. Yes. Symmetry, incredibly powerful. Yes. Adrian, and then you're next. Yeah?

AUDIENCE: Then you have approximation [INAUDIBLE].

PROFESSOR: OK. So linearization. And that is of course also huge. I mean, symmetry is used in almost every field. Linearization, basically you can't do anything that's nonlinear. So every field does linearization. So that core idea would be beautiful to convey that. Yes?

AUDIENCE: Error.

PROFESSOR: OK. Errors . How do you deal with errors? There are common principles, probability theory, statistics, that deal with errors in any field. What are those? Well, that's what you want to understand so you can move to any other field.

I'll put a dot, dot here, so symmetry, another one that's related to symmetry is conservation. Noether's theorem is that every conservation law corresponds to a symmetry and vice versa. I think I've stated that correctly. But they flow from each other. So actually I do conservation here as part of symmetry.

So let me stop the list right there, just to say that right away-- so let me put the other one which I talked about before, which is successive approximation, that you don't want to solve a hard problem all at once. It's sort of like divide and conquer. But you want to try to solve it approximately and then less approximately, then less less approximately, so you get closer and closer, as close as you need. That's a general principle of engineering design as well.

So these, just to show you that all of a sudden you thought of a whole bunch of examples. And probably you can use that in any course you take. So now you all have like 11. Right now I'm not saying you have to use all 11 in your course. But you can start to restructure courses around those ideas and make them explicit and choose which you think are the most important in your field in the ways of thinking and illustrate them with things from your field but nearby as well.

So this evening I'll post a reading for next week. And I'll post a short assignment basically to continue this line of thinking, just to give you some more practice in course design, fleshing it out farther. But what you've done is the key step, the hardest step to course design.

So if you could fill out the question sheet, as always very helpful. And also there was a question before. Could I make sure to list on the website things I mentioned in class?

Yes. I should do that. So if I could request people to, if you have in your notes any of the books that I mentioned in class but didn't put on the website, if you could email me those, I will put them on the website. So as a sort of divide and conquer approach, I'm sure actually collectively you have all of them.

OK. So just when you're done, you can put the sheets up here. And then outside of that door, we'll have office hours as usual so that the next class doesn't get delayed by us being in here.