The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
ERIC GRIMSON: OK, welcome back or welcome,
depending on whether you've been away or not.
I'm going to start with two simple announcements.
There is a reading assignment for this lecture,
actually for the next two lectures, which is chapter 18.
And on a much happier note, there
is no lecture Wednesday because we
hope that you're going to be busy
preparing to get that tryptophan poisoning
as you eat way too much turkey and you fall asleep.
More importantly, I hope you have
a great break over Thanksgiving, whether you're here
or you're back home or wherever you are.
But no lecture Wednesday.
Topic for today, I'm going to start with seems like--
sorry, what's going to seem like a really obvious statement.
We're living in a data intensive world.
Whether you're a scientist, an engineer, social scientist,
financial worker, politician, manager of a sports team,
you're spending increasingly larger amounts
of time dealing with data.
And if you're in one of those positions,
that often means that you're either writing code
or you're hiring somebody to write code for you
to figure out that data.
And this section of the course is
focusing on exactly that issue.
We want to help you understand what
you can try to do with software that manipulates data, how you
can write code that would do that manipulation of data
for you, and especially what you should believe about what
that software tells you about data, because sometimes it
tells you stuff that isn't exactly what you need to know.
And today we're going to start that
by looking at particularly the case where we get data
from experiments.
So think of this lecture and the next one
as sort of being statistics meets experimental science.
So what do I mean by that?
Imagine you're doing a physics lab, biology lab, a chemistry
lab, or even something in sociology or anthropology,
you conduct an experiment to gather some data.
It could be measurements in a lab.
It could be answers on a questionnaire.
You get a set of data.
Once you've got the data, you want
to think about what can I do with it,
and that usually will involve using some model, some theory
about the underlying process to generate questions
about the data.
What does this data and the model associated with it
tell me about future expectations,
help me predict other results that
will come out of this data.
In the social case, it could be how
do I think about how people are going
to respond to a poll about who are
you voting for in the next election, for example.
Given the data, given the model, the third thing
we're typically going to want to do
is then design a computation to help us answer questions
about the data, run a computational experiment
to complement the physical experiment
or the social experiment we used to gather
the data in the first place.
And that computation could be something deep.
It could be something a little more interesting,
depending on how you're thinking about it.
But we want to think about how do we
use computation to run additional experiments for us.
So I'm going to start by using an example of gathering
experimental data, and I want to start
with the idea of a spring.
How would I model a spring?
How would I gather data about a spring?
And how would I write software to help me answer questions
about a spring?
So what's spring?
Well, there's one kind of spring, a little hard to model,
although it could be interesting what's swimming around in there
and how do I think about the ecological implications
of that spring.
Here's a second kind of spring.
It's about four or five months away,
but eventually we'll get through this winter
and get to that spring and that would be nice,
but I'm not going to model that one either.
And yes, my jokes are really bad,
and yes, you can't do a darn thing about them
because I am tenured because--
while I'd like to model these two springs,
we're going to stick with the one
that you see in physics labs, these kinds of springs,
so-called linear springs.
And these are springs that have the property that you
can stretch or compress them by applying a force to it.
And when you release them, they literally
spring back to the position they were originally.
So we're going to deal with these kinds of springs.
And the distinguishing characteristics
of these two springs and others in this class
is that that force you require to compress it or stretch
it a certain amount--
the amount of force you require varies linearly
in the distance.
So if it takes some amount of force
to compress it some amount of distance,
it takes twice as much force to compress it twice as much
of a distance.
It's linearly related.
So each one of these springs--
these kinds of springs has that property.
The amount of force needed to stretch
or compress it's linear in that distance.
Associated with these springs there
is something called a spring constant--
usually represented by the number k--
that determines how much force do
you need to stretch or compress the spring.
Now, it turns out that that spring constant can vary a lot.
The slinky actually has a very low spring constant.
It's one newton per meter.
That spring on the suspension of a motorcycle
has a much bigger spring constant.
It's a lot stiffer, 35,000 newtons per meter.
And just in case you don't remember,
a newton is the amount of force you
need to accelerate a one-kilogram mass one
meter per second squared.
We'll come back to that in a second.
But the idea is we'd like to think about how do we
model these kinds of springs.
Well, turns out, fortunately for us,
that that was done about 300-plus years ago
by a British physicist named Robert Hooke.
Back in 1676 he formulated Hooke's law of elasticity.
Simple expression that says the force you
need to compress or stretch a spring
is linearly related to the distance, d,
that you've actually done that compression in,
or another way of saying it is, if I compress
a spring some amount, the force that's stored in it
is linearly related to that distance.
And the negative sign here basically says
it's pointing in the opposite direction.
So if I compress, the force is going to push it back out.
If I stretch it, the force is going to push back
into that resting position.
Now, this law holds for a wide range of springs,
which is kind of nice.
It's going to hold both in biological systems
as well as in physical systems.
It doesn't hold perfectly.
There's a limit to how much you can stretch,
in particular, a spring before the law breaks down,
and maybe you did this as a kid, right.
If you take a slinky and pull it too far apart,
it stops working because you've exceeded
what's called the elastic limit of the spring.
Similarly, if you compress it too far,
although I think you have to compress it a long ways,
it'll stop working as well.
So it doesn't hold completely, and it also
doesn't hold for all springs.
Only those springs that satisfy this linear law,
which are a lot of them.
So, for example, it doesn't apply to rubber bands,
it doesn't apply to recurved bows.
Those are two examples of springs that do not
obey this linear relationship.
But nonetheless, there's Hooke's law.
And one of the things we can do is say, well,
let's use it to do a little bit of reasoning about this spring.
So we can ask the question, how much
does a rider have to weigh to compress this spring by one
centimeter?
And we've got Hooke's law, and I also gave you
a little bit of hint here.
So I told you that this spring has a spring constant
of 35,000 newtons per meter.
So I could just plug this in, right, one centimeter,
it's 1/100 of a meter times--
so that's the-- there's the spring constant.
There's the amount we're going to compress it.
Do a little math, and that says that the force I need
is 350 newtons.
So what's a newton?
A small town in Massachusetts, an interesting cookie,
and a force that we want to think about.
I keep telling you guys, the jokes are really bad.
So how do I get force?
Well, you know that.
Mass times acceleration, right, F equals ma.
For acceleration here, I'm going to make an assumption, which
is that the spring is basically oriented perpendicular
to the earth, so that the acceleration is just
the acceleration of gravity, which is roughly
9.8 meters per second squared.
It's basically pulling it down.
So I could plug that back in because remember
what I want to do is figure out what's the mass I need.
So for the force, I'm substituting that in.
I've got that expression, mass times 9.8 meters divided
by seconds squared is 350 newtons,
divide through by 9.8 both sides, do a little bit of math.
And it says that the mass I need is 350 kilograms divided
by 9.8.
And that k refers to kilograms, not to the spring constant.
Poor choice of example, but there I am.
And if I do the math, it says I need a rider that
weighs 35.68 kilos.
And if you're not big on the metric system,
it's actually a fairly light rider.
That's about 79 pounds.
So a 79-pound rider would compress that spring one
centimeter.
So we can figure out how to use Hooke's law.
We're thinking about what we want to do with springs.
That's kind of nice.
How will we actually get the spring constant?
It's really valuable to know what the spring constant is.
And just to give you a sense of that,
it's not just to deal with things like slinkies.
Atomic force microscopes, need to know the spring
constants of the components in order
to calibrate them properly.
The force you need to deform a strand of DNA
is directly related to the spring constants
of the biological structures themselves.
So I'd really like to figure out how do I get them.
How many of you have done this experiment in physics and hated
it?
Right.
Well, I don't know if you hated it or not,
but you've done it, right?
Standard way to do it is I'd take a spring,
I suspend it from some point.
Let it come to a resting position.
And then I put a mass on the bottom of the spring.
It kind of bounces around.
And when it settles, I measure the distance
from where it was before I put the mass
on to the distance of where it is after I've added the mass.
I measure that distance.
And then I just plug in.
I plug into that formula there.
The force is minus k times d.
So k the spring constant is the force, forget the minus sign,
divided by the distance, and the force here
would be 9.8 meters per second squared or-- kilograms
per second squared times the mass divided by d.
So I could just plug it in.
In an ideal world, I'd plug it in, I'm done, one measurement.
Not so much, right.
Masses aren't always perfectly calibrated.
Maybe the spring has got not perfect materials in it.
So ideally I'd actually do multiple trials.
I would take different weights, put them on the spring,
make the measurements, and just record those.
So that's what I'm going to do, and I've actually done that.
I'm not going to make you do it.
But I get out a set of measurements.
What have I done here?
I've used different masses, all increasing by now 0.05
kilograms, and I've measured the distance
that the spring has deformed.
And ideally, these would all have
that nice linear relationship, so I could just plug them in
and I could figure out what the spring constant is.
So let's take this data and let's plot it.
And by the way, all the code you'll
be able to see when you download the file,
I'm going to walk through some of it quickly.
This is a simple way to deal with it,
and I'm going to back up for a second.
There's my data, and I actually have done this
in some ways the wrong order.
These are my independent measures, different masses.
I'm going to plot those along the x-axis,
the horizontal axis.
These are the dependent things.
These are the things I'm measuring.
I'm going to plot those along the y-axis.
So I really should have put them in the other order.
So just cross your eyes and make this column go over
to that column, and we'll be in good shape.
Let's plot this.
So here's a little file.
Having stored those away in a file,
I'm just going to read them in, get data.
Just going to do the obvious thing of read in these things
and return two tuples or lists, one for the x values--
or if you like, again going back to it,
this set of values, and one for the y values.
Now I'm going to play a little trick
that you may have seen before that's going to be handy to me.
I'm going to actually call this function out
of the PyLab library called array.
I pass in that tuple, and what it
does is it converts it into an array, which
is a data structure that has a fixed number of slots in it
but has a really nice property I want to take advantage of.
I could do all of this with lists.
But by converting that into array
and then giving it the same name xVals and similarly
for the yVals, I can now do math on the array
without having to write loops.
And in particular right here, notice what I'm doing.
I'm taking xVals, which is an array, multiplying it
by a number.
And what that does is it takes every entry in the array,
multiplies that entry, and puts it
into basically a new version of the array, which I then
store into xVals.
If you've programmed in Matlab, this
is the same kind of feeling, right.
I can take an array, do something to it,
and that's really nice.
So I'm going to scale all of my values,
and then I'm going to plot them out some appropriate things.
And if I do it, I get that.
I thought we said Hooke's law was a linear relationship.
So in an ideal world, all of these points
ought to lay along a line somewhere,
where the slope of the line would
tell me the spring constant.
Not so good, right.
And in fact, if you look at it, you can kind of
see-- in here you can kind of imagine there's a line there,
something funky is going on up here.
And we're going to come back to that at the end of the lecture.
But how do we think about actually finding the line?
Well, we know there's noise in the measurement,
so our best thing to do is to say, well,
could we just fit a line to this data?
And how would we do that?
And that's the first big thing we want to do today.
We want to try and figure out, given
that we've got measurement noise, how
do we fit a line to it.
So how do we fit a curve to data?
Well, what we're basically going to try and do
is find a way to relate an independent variable, which
were the masses, the y values, to the dependent-- sorry,
wrong way.
The independent values, which are the x-axis,
to the dependent value, what is the actual displacement we're
going to see?
So another way of saying it is if I go back to here,
I want to know for every point along here,
how do I fit something that predicts what the y value is?
So I need to figure out how to do that fit.
To decide-- even if I had a curve, a line that I
thought was a good fit to that, I
need to decide how good it is.
So imagine I was lucky and somebody said,
here's a line that I think describes
Hooke's law in this case.
Great.
I could draw the line on that data.
I could draw it on this chunk of data here.
I still need to decide how do I know if it's a good fit.
And for that, we need something we call an objective function,
and it's going to measure how close is
the line to the data to which I'm trying to fit it.
Once we've defined the objective function, then what we say
is, OK, now let's find the line that minimizes it,
the best possible line, the line that makes that objective
function as small as possible, because that's going
to be the best fit to the data.
And so that's what I'd like to do.
We're going to see--
we're going to do it for general curves,
but we're going to start just with lines,
with linear function.
So in this case, we want to say what's
the line such that some function of the sum
of the distances from the line to the measured points
is minimized.
And I'm going to come back in a second to how
do we find the line.
But first we've got to think about what does it
mean to measure it.
So I've got a point.
Imagine I got a line that I think
is a good match for the thing fitting the data.
How do I measure distance?
Well, there's one option.
I could measure just the displacement along the x-axis.
There's a second option.
I could measure the displacement vertically.
Or a third option is I could actually measure the distance
to the closest point on the line, which
would be that perpendicular distance there.
You're way too quiet, which is always dangerous.
What do you think?
I'm going to look for a show of hands here.
How many people think we should use x as the thing
that we measure here?
Hands up.
Please don't use a single finger when you put your hand up.
All right.
Good.
How many people think we should use p, the perpendicular
distance?
Reasonable number of hands.
And how about y?
And I see actually about split between p and y.
And that's actually really good.
X doesn't make a lot of sense, right,
because I know that my values along the x-axis
are independent measurements.
So the displacement in that direction
doesn't make a lot of sense.
P makes a lot of sense, but unfortunately isn't
what I want.
We're going to see examples later
on where, in fact, minimizing things
where you minimize that distance is the right thing to do.
When we do machine learning, that
is how you find what's called a classifier or a separator.
But actually here we're going to pick y,
and the reason is important.
I'm trying to predict the dependent value, which
is the y value, given an independent new x value.
And so the displacement, the uncertainty
is, in fact, the vertical displacement.
And so I'm going to use y.
That displacement is the thing I'm going
to measure as the distance.
How do I find this?
I need an objective function that's
going to tell me what is the closeness of the fit.
So here's how I'm going to do it.
I'm going to have some set of observed values.
Think of it as an array.
I've got some index into them, so the indices
are giving me the x values.
And the observed values are the things I've actually measured.
If you want to think of it this way,
I'm going to go back to this slide really quickly.
The observed values are the displacements
or the values along the y-axis.
Sorry about that.
Let's assume that I have some hypothesized line that I
think fits this data, y equals ax plus b.
I know the a and the b.
I've hypothesized it.
Then predicted will basically say given the x value,
the line predicts here's what the y value should be.
And so I'm going to take the difference between those two
and square them.
So the difference makes sense.
It tells me how far away is the observed value from what
the line predicts it should be.
Why am I squaring it?
Well, there are two reasons.
The first one is that squaring is
going to get rid of the sign.
It shouldn't matter if my observed value
is some amount above the predicted value
or some amount below-- the same amount
below the predicted value.
The displacement in direction shouldn't matter.
It's how far away is it.
Now, you could say, well, why not just use absolute value?
And the answer is you could, but we're
going to see in a couple of slides that by using the square
we get a really nice property that helps
us find the best fitting line.
So my objective function here basically says,
given a bunch of observed values,
use the hypothesized line to predict what the value should
be, measure the difference in the y direction--
which is what I'm doing because I'm measuring predicted
and observed y values--
square them, sum them all up.
It's called least squares.
That's going to give me a measure of how
close that line is to a fit.
In a second, I'll get to how you find the best line.
But this hopefully looks familiar.
Anybody recognize this?
You've seen it earlier in this class.
Boy, that's a terrible thing to ask because you don't even
remember the last thing you did in this class other
than the problem set.
AUDIENCE: [INAUDIBLE]
ERIC GRIMSON: Sorry?
AUDIENCE: Variance.
ERIC GRIMSON: Variance.
Thank you.
Absolutely.
Sorry, I didn't bring any candy today.
That's Professor Guttag.
I got a better arm than he does, but I still
didn't bring any candy today.
Yeah, it's variance, not quite.
It's almost variance.
That's the variance times the number of observations,
or another way of saying it is if I divided this
by the number of observations, that would be the variance.
If I took the square root, it would
be the standard deviation.
Why is that valuable?
Because that tells you something about how badly things
are dispersed, how much variation
there is in this measurement.
And so if it says, if I can minimize this expression,
that's great because it not only will
find what I hope is the best fit,
but it's going to minimize the variance between what I predict
and what I measure, which makes intuitive sense.
That's exactly the thing I would like to minimize.
This was built on the assumption that I
had a line that I thought was a good fit,
and this lets me measure how good a fit I have.
But I still have to do a little bit more.
I have to now figure out, OK, how do
I find the best-fitting line?
And for that, we need to come up with a minimization technique.
So to minimize this objective function,
I want to find the curve for the predicted values--
this thing here--
some way of representing that that leads
to the best possible solution.
And I'm going to make a simple assumption.
I'm going to assume that my model for this
predicted curve--
I've been using the example of a line,
but we're going to say curve--
is a polynomial.
It's a polynomial and one variable.
The one variable is what are the x values of the samples.
And I'm going to assume that the curve is a polynomial.
In the simplest case, it's a line in case order, and two,
it's going to be a parabola.
And I'm going to use a technique called linear regression
to find the polynomial that best fits the data, that minimizes
that objective function.
Quick aside, just to remind you, I'm sure you remember,
so polynomial--
polynomials, either the value is zero, which is really boring,
or it is a finite sum of non-zero terms
that all have the form c times x to the p.
C is a constant, a real number.
P is a power, a non-negative integer.
And this is basically-- x is the free variable that's
going to capture this.
So easy way to say it is a line would
be represented as a degree one polynomial ax plus b.
A parabola is a second-degree polynomial,
ax squared plus bx plus c.
And we can go up to higher order terms.
We're going to refer to the degree of the polynomial
as the largest degree of any term in that polynomial.
So again, degree one, linear degree two, quadratic.
Now how do I use that?
Well, here's the basic idea.
Let's take a simple example.
Let's assume I'm still just trying to fit a line.
So my assumption is I want to find a degree one
polynomial, y equals ax plus b, as our model of the day.
That means for every sample, I'm going to plug in x,
and if I know a and b, it gives me the predicted value.
I've already seen that's going to give me
a good measure of the closeness of the fit.
And the question is, how do I find a and b.
My goal is find a and b such that when
we use this polynomial to compute those y values,
that sum squared difference is minimized.
So the sum squared difference is my measure of fit.
All I have to do is find a and b.
And that's where linear regression comes in,
and I want to just give you a visualization of this.
If a line is described by ax plus b,
then I can represent every possible line
in a two-dimensional space.
One axis is possible values for a.
The other axis is possible values for b.
So if you think about it, I take any point in that space.
It gives me an a and a B value.
That describes a line.
Why should you care about that?
Because I can put a two-dimensional surface
over that space.
In other words, for every a and b, that gives me a line,
and I could, therefore, compute this function,
given the observed values and the predicted values,
and it would give me a value, which
is the height of the surface in that space.
If you're with me with the visualization,
why is that nice?
Because linear regression gives me a very easy way
to find the lowest point on that surface,
which is exactly the solution I want,
because that's the best fitting line.
And it's called linear regression
not because we're solving for a line,
but because of how you do that solution.
If you think of this as being--
take a marble on this two-dimensional surface,
you want to place the marble on it,
you want to let it run down to the lowest
point in the surface.
And oh, yeah, I promised you why do we use sum squares,
because if we used the sum of the squares,
that surface always has only one minimum.
So it's not a really funky, convoluted surface.
It has exactly one minimum.
It's called linear regression because the way
to find it is to start at some point and walk downhill.
I linearly regress or walk downhill along the gradient
some distance, measure the new gradient,
and do that until I get down to the lowest
point in the surface.
Could you write code to do it?
Sure.
Are we going to ask you to do it?
No, because fortunately--
I was hoping to get a cheer out of that.
Too bad.
OK, maybe we will ask you to do it on the exam.
What the hell.
You could do it.
In fact, you've seen a version of this.
The typical algorithm for doing it
is very similar to Newton's method
that we used way back in the beginning of 60001
when we found square roots.
You could write that kind of a solution,
but the good news is that the nice people who
wrote Python, or particularly PyLab,
have given you code to do it.
And we're going to take advantage of it.
So in PyLab there is a built-in function called polyFit.
It takes a collection of x values,
takes a collection of equal length
of y values-- they need to be the same length.
I'm going to assume they're arrays.
And it takes an integer n, which is the degree of fit,
that I want to apply.
And what polyFit will do is it will
find the coefficients of a polynomial of that degree that
provides the best least squares fit.
So think of it as polyFit walking along
that surface to find the best a and b that will come back.
So if I give it a value of n equals one,
it'll give me back the a and b that gives me the best line.
If I get a value of n equal two, it
gives me back a, b, and c that would
fit an ax squared plus bx plus c parabola to best fit the data.
And I could pick n to be any non-negative integer,
and it would actually come up with a good fit.
So let's use it.
I'm going to write a little function called fitData.
The first part up here just comes from plotData.
It's exactly the same thing.
I read in the data.
I convert them into arrays.
I convert this because I want to get out the force.
I go ahead and plot it.
And then notice what I do, I use polyFit right here
to take the inputted x values and y values and a degree one,
and it's going to give me back a tuple,
an a and a b that are the best fit line.
Finds that point in the space that best fits it.
Once I've got that, I could go ahead and actually compute now
what are the estimated or predicted values.
The line's going to tell me what I
should have seen as those values,
and I'm going to do the same thing.
I'm going to take x values, convert it into array,
multiply it by a, which says every entry in the array
is scaled by a.
Add b to every entry.
So I'm just computing ax plus b for all possible x's.
And that then gives me an estimated set of y values,
and I can plot those out.
I'm cheating here.
Sorry.
I'm misdirecting you.
I never cheat.
I actually don't need to do the conversion to an array
there because I did it up here.
But because I've borrowed this from plot lab,
I wanted to show you that I can redundantly
do it here to remind you that I want to convert it
into array to make sure I can do that kind of algebra on it.
The last thing I could do is say even if I can--
once I show you the fit of this line,
I also want to get out the spring constant.
Now, the slope of this line is difference in force
over difference in distance.
The spring constant is the opposite of it.
So I could simply take the slope of the line, which
is a, invert it, and that gives me the spring constant.
So let's see what happens if we actually run this.
So I'm going to go over to my code,
hoping that it works properly.
Here's my Python.
I've loaded this in.
I'm going to run it.
And there you go.
Fits a line, and it prints out the value
of a, which is about 0.46, and the value of b.
And if I go back and look at this,
there we go, spring constant is about 21 and a half,
which is about the reciprocal of 0.046
if you can figure that out.
And you can see, it's not a bad fit
to a line through that data.
Again, there's still something funky going on over here
that we're going to come back to.
But it's a pretty good fit to the data.
Great.
So now I've got a fit.
I'm going to show you a variation of this
that we're going to use in a second.
I could do the same thing, but after I've done polyFit here,
I'm going to use another built-in function called
polyval.
It's going to take a polynomial, which
is captured by that model of the thing that I returned,
and I'm going to show you the difference again.
Back sure we returned this as a tuple.
Since it's coming back as a tuple,
I can give it a name model.
Polyval will take that tuple plus the x values
and do the same thing.
It will give me back an array of predicted values.
But the nice thing here is that this model could be a line.
It could be a parabola.
It could be a quartic.
It could be a quintic.
It could be any order polynomial.
If you like the abstraction here-- which
we're going to see in a little bit,
that it allows me to use the same code
for different orders of model.
And if I ran this, it would do exactly the same thing.
I'm going to come back to thinking about what's going
on in that spring in a second.
But I want to show you another example.
So here's another set of data.
In a little bit, I'll show you where
that mystery data came from.
But here's another set of data that I've plotted out.
I could run the same thing.
I could run exactly the same code and fit a line to it.
And if I do it, I get that.
What do you think?
Good fit?
Show of hands, how many people like this fit to the data?
Show of hands, how many people don't
like this fit to the data?
Show of hands, how many hope that I'll
stop asking you questions?
Don't put your hands up.
Yeah, thank you.
I know.
Too bad.
It's a lousy fit.
And you kind of know it, right.
It's clear that this doesn't look
like it's coming from a line, or if it
is, it's a really noisy line.
So let's think about this.
What if I were to try a higher order degree.
Let's change the one to a two.
So I'm going to come back to it in a second.
I've changed the one to a two.
That says I'm still using the polynomial fit,
but now I'm going to ask what's the best fitting parabola, ax
squared plus bx plus c.
Simple change.
Because I was using polyval, exactly the same code
will work.
It's going to do the fit to it.
This is, by the way, still an example of linear regression.
So think of what I'm doing now.
I have a three-dimensional space.
One axis is a values.
Second axis is b values.
Third axis is c values.
Any point in that space describes a parabola,
and every point in that space describes
every possible parabola.
And now you've got to twist your head a little bit.
Put a four-dimensional surface on
that three-dimensional basis, where the point in that surface
is the value of that objective function.
Play the same game.
And you can.
It's just a higher-dimensional thing.
So you're, again, going to walk down the gradient
to find the solution, and be glad you don't
have to write this code because PyLab will do it
for you freely.
But it's still an example of regression, which is great.
And if we do that, we get that fit.
Actually just to show you that, I'm going to run it,
but it will do exactly the same thing.
If I go over to Python--
wherever I have it here--
I'm going to change that order of the model.
Oops, it went a little too far for me.
Sorry about that.
Let me go back and do this again.
There's the first one, and there's the second one.
So I could fit different models to it.
Quadratic clearly looks like it's a better fit.
I hope you'll agree.
So how do I decide which one's better
other than eyeballing it?
And then if I could fit a quadratic to it,
what about other orders of polynomials?
Maybe there's an even better fit out there.
So how do I figure out what's the best way to do the fit?
And that leads to the second big thing for this lecture.
How good are these fits?
What's the first big thing?
The idea of linear regression, a way of finding
fits of curves to data.
But now I've got to decide how good are these.
And I could ask this question two ways.
One is just relative to each other,
how do I measure which one's better other than looking
at it by eye?
And then the second part of it is in an absolute sense,
how do I know where the best solution is?
Is quadratic the best I could do?
Or should I be doing something else
to try and figure out a better solution, a better
fit to the data?
The relative fit.
What are we doing here?
We're fitting a curve, which is a function
of the independent variable to the dependent variable.
What does it mean by that?
I've got a set of x values.
I'm trying to predict what the y values should be,
the displacement should be.
I want to get a good fit to that.
The idea is that given an independent value,
it gives me an estimate of what it should be,
and I really want to know which fit provides the better
estimates.
And since I was simply minimizing mean squared error,
average square error, an obvious thing to do
is just to use the goodness of fit by looking at that error.
Why not just measure where am I on that surface
and see which one does better?
Or actually it would be two surfaces,
one for a linear fit, one for a quadratic one.
We'll do what we always do.
Let's write a little bit of code.
I can write something that's going
to get the average, mean squared error.
Takes in a set of data points, a set of predicted values,
simply measures the difference between them,
squares them, adds them all up in a little loop
here and returns that divided by the number of samples I have.
So it gives me the average squared error.
And I could do it for that first model I built,
which was for a linear fit, and I
could do it for the second model I built,
which is a quadratic fit.
And if I run it, I get those values.
Looks pretty good.
You knew by eye that the quadratic was a better fit.
And look, this says it's about six times better,
that the residual error is six times smaller
with the quadratic model than it is the linear model.
But with that, I still have a problem, which is--
OK, so it's useful for comparing two models.
But is 1524 a good number?
Certainly better than 9,000-something or other.
But how do I know that 1524 is a good number?
How do I know there isn't a better fit out there somewhere?
Well, good news is we're going to be able to measure that.
It's hard to know because there's no bound on the values.
And more importantly, this is not scale independent.
What do I mean by that?
If I take all of the values and multiply them by some factor,
I would still fit the same models to them.
They would just scale.
But that measure would increase by that amount.
So I could make the error as big or as small
as I want by just changing the size of the values.
That doesn't make any sense.
I'd like a way to measure goodness of fit
that is scale independent and that tells me
for any fit how close it comes to being
the perfect fit to the data.
And so for that, we're going to use something
called the coefficient of determination
written as r squared.
So let me show you what this does, and then we're
going to use it.
The y's are measured values.
Those are my samples I got from my experiment.
The p's are the predicted values.
That is, for this curve, here's what I
predict those values should be.
So the top here is basically measuring
as we saw before the sum squared error in those pieces.
Mu down here is the average, or mean, of the measured values.
It's the average of the y's.
So what I've got here is in the numerator--
this is basically the error in the estimates from my curve
fit.
And in the denominator I've got the amount of variation
in the data itself.
This is telling me how much does the data change from just being
a constant value, and this is telling me how much
do my errors vary around it.
That ratio is scale independent because it's a ratio.
So even if I increase all of the values
by some amount, that's going to divide out,
which is kind of nice.
So I could compute that, and there it is.
R squared is, again, that expression.
I'll take in a set of observed values,
a set of predicted values, and I'll measure the error--
again, these are arrays.
So I'm going to take the difference between the arrays.
That's going to give me piecewise or pairwise
that difference.
I'll square it.
That's going to give me at every point in the array
the square of that distance.
And then because it's an array, I
can just use the built-in sum function to add them all up.
So this is going to give me the--
if you like, the values up there.
And then I'm going to play a little trick.
I'm going to compute the mean error, which
is that thing divided by the number of observations.
Why would I do that?
Well, because then I can compute this really simply.
I could write a little loop to compute it.
But in fact, I've already said what is that?
If I take that sum and divide it by the number of samples,
that's the variance.
So that's really nice.
Right here I can say, get the variance
using the non-p version of the observed data.
And because that has associated with it division
by the number of samples, the ratio
of the mean error to the variance
is exactly the same as the ratio of that to that.
Little trick.
It lets me save doing a little bit of computation.
So I can compute r squared values.
So what does r squared actually tell us?
What we're doing is we're trying to compare the estimation
errors, the top part, with the variability
in the original values, the bottom part.
So r squared, as you're going to see there,
it's intended to capture what portion
of the variability in the data is accounted for by my model.
My model's a really good fit.
It should account for almost all of that data.
So what we see then is if we do a fit with a linear regression,
r squared is always going to be between zero and one.
And I want to just show you some examples.
If r squared is equal to one, this is great.
It says the model explains all of the variability in the data.
And you can see it if we go back here.
How do we make r squared equal to one?
We need this to be zero, which says
that the variability in the data is perfectly
predicted by my model.
Every point lies exactly along the curve.
That's great.
Second option at the other extreme
is if r squared is equal to zero,
you basically got bupkis, which is a well-known technical term,
meaning there's no relationship between the values predicted
by the model and the actual data.
That basically says that all of the variability
here is exactly the same as all the variability in the data.
The model doesn't capture anything,
and it's making this one, which is making the whole thing zero.
And then in between an r squared of about a half
says you're capturing about half the variability.
So what you would like is a system
in which your fit is as close to an r squared value of one
as possible because it says my model is
capturing all the variability in the data really well.
So two functions that will do this for us.
We're going to come back to these in the next lecture.
The first one called generate fits, or genFits,
will take a set of x values, a set of y values,
and a list or a tuple of degrees,
and these will be the different degrees
of models I'd like to fit.
I could just give it one.
I could give it two.
I could give a 1, 2, 4, 8, 16, whatever.
And I'll just run through a little loop
here where I'm going to build up a set of models
for each degree-- or d in degrees.
I'll do the fit exactly as I had before.
It's going to return a model, which
is a tuple of coefficients.
And I'm going to store that in models and then return it.
And then I'm going to use that, because in testFits I
will take the models that come from genFits,
I'll take the set of degrees that I also passed
in there as well as the values.
I'll plot them out, and then I'll
simply run through each of the models and generate a fit,
compute the r squared value, plot it, and then print out
some data.
With that in mind, let's see what happens if we run this.
So I'm going to take, again, that example
of that data that I started with,
assuming I picked the right one here, which I think
is this one.
I'm going to do a fit with a degree one and a degree
two curve.
So I'm going to fit the best line.
I'm going to fit the best quadratic, the best parabola,
and I want to see how well that comes out.
So I do that.
I got some data there.
Looks good.
And what does the data tell me?
Data says, oh, cool--
I know you don't believe it, but it
is because notice what it says, it
says the r squared value for the line is horrible.
It accounts for less than 0.05% of the data.
You could say, OK, I can see that.
I look at it.
It does a lousy job.
On the other hand, the quadratic is really pretty good.
It's accounting for about 84% of the variability in the data.
This is a nice high value.
It's not one, but it's a nice high value.
So this is now reinforcing what I already
knew, but in a nice way.
It's telling me that that r squared value tells me
that the quadratic is a much better fit than the linear fit
was.
But then you say maybe, wait a minute.
I could have done this by just comparing the fits themselves.
I already saw that.
Part of my goal is how do I know if I've got
the best fit possible or not.
So I'm going to do the same thing,
but now I'm going to run it with another set of degrees.
I'm going to go over here.
I'm going to take exactly the same code.
But let's try it with a quadratic,
with a quartic, an order eight, and an order 16 fit.
So I'm going to take different size polynomials.
As a quick aside, this is why I want
to use the PyLab kind of code because now I'm
simply optimizing over a 16-dimensional space.
Every point in that 16-dimensional space
defines a 16th-degree polynomial.
And I can still use linear regression,
meaning walking down the gradient,
to find the best solution.
I'm going to run this.
And I get out a set of values.
Looks good.
And let's go look at them.
Here is the r squared value for quadratic, about 84%.
Degree four does a little bit better.
Degree eight does a little bit better.
But wow, look at that, degree 16--
16th order polynomial does a really good job,
accounts for almost 97% of the variability in the data.
That sounds great.
Now, to quote something that your parents probably
said to you when you were much younger, just because something
looks good doesn't mean we should do it.
And in fact, just because this has a really high r
squared value doesn't mean that we want to use the order
16th polynomial.
And I will wonderfully leave you waiting in suspense
because we're going to answer that question next Monday.
And with that, I'll let you out a few minutes early.
Have a great Thanksgiving break.