CHRISTINE BREINER: Welcome back to recitation.
In this video, what I'd like you to do
is use least squares to fit a line to the following
data, which includes three points: the point (0, 1),
the point (2, 1), and the point (3, 4).
And I've drawn a rough picture where these points are
on a graph, and I'll be talking a little bit about that
after you try this problem.
So why don't you try to solve the problem based
on the technique you learned in class,
and then bring the video back up.
I'll show you again what you're actually doing,
and then I will also solve the problem and you can see,
you can compare your answer to mine.
OK, welcome back.
So the first thing I want to do when we're talking about least
squares to fit a line to these data, what I'd like to do
is I'm going to draw a line on here,
and we're going to talk about what the least squares actually
is trying to do.
And so I'm going to draw this line, say, something like this.
That's my first guess on what might
be the actual least squares line for these data.
And if you recall, what you're actually trying to do
is you're trying to minimize a certain quantity,
and the quantity you're trying to minimize
is the difference between the actual value you get
and the expected value you get, the square of that.
So this is one thing we're going to square.
So even pictorially, we could say
we have something that contributes some area, OK,
and then this is another thing that's
the difference between the expected value in y
and the actual value you get.
So let me try and make that squared,
because we square that value.
And then this last one is pretty small.
The difference between the expected value
and the actual value is very small over in this case
from the picture I drew.
And so what you're trying to do is
you're trying to find the line that
minimizes the area of the sum of these three,
in this case, three squares.
So if I, you know, if I tip the line down further here,
but I kept it fixed there, the area of this square
would get bigger, because this distance would get bigger.
But, of course, the area of this square would get smaller.
And so you're trying to figure out
a way to optimize, in this case, minimize,
the area of these-- sorry, the sum
of the areas of these squares by figuring out
where to put the line.
Now, I mentioned if I could fix this here and move it
up and down, then that would change the slope
but fix the intercept.
But I could also move the intercept up and down,
and I could slide the line up and down,
and that would fix the slope but vary the intercept,
or I could do both.
And ultimately, that's what you're doing.
You're trying to find the best y-intercept and the best slope
at the same time, and that's why you learned this process
in multivariable calculus, because you're
trying to optimize something that has two
quantities that you can change.
You can change slope and you can change the y-intercept,
so that's why you learned it now instead
of in single-variable calculus.
Now, if we come over here, I wrote down ahead
of time what the equations are you get when you optimize,
when you're trying to optimize the least squares line.
So if you write down, you know, the function
as a function of a and b, and you
take the derivative with respect to a
and you set it equal to zero, you get the first equation.
When you take the derivative with respect
to b and you set it equal to zero,
you get the second equation.
And so I'm just going to fill in the details
and then tell you what the solution is,
and then I'm going to mention a little more sophisticated thing
we can do, or I guess maybe the next level
of least squares we can do, that's not a linear problem.
So I'll do that last.
So what I want to do is I put all the values I need over here
on the right.
So all the values we need are on the right.
We're going to need the sum of all the x_i's and that's 5.
The sum of all the y_i's is 6.
The sum of the x_i squareds is 13,
and the sum of the x_i*y_i product is 14.
Now, where are these coming from?
Remember, what is x_i and what is y_i?
If we come back over here, this is (x_1, y_1), (x_2, y_2),
(x_3, y_3), OK?
So that's sort of how we get-- if we switch the whole pairs
around, it doesn't matter.
Obviously, if we switched x- and y-values together,
it does matter, and that you see actually from the equation
does matter.
Right here you have a product of them.
So let me write this out.
And again, we're solving for a and b, so in this case,
a is the slope and b is the intercept, right?
So let me write everything in.
So we actually get that we want to solve the following system:
13a plus 5b is equal to 14.
Make sure I put in everything correctly.
And then the second one is 5a plus-- n in this case
is 3, because there were three points-- plus 3b is equal to 6.
OK, this is a system of equations.
It has two equations and two unknowns,
and if you solve this-- I should look at my notes,
because I didn't memorize it and I'm not
going to do all the algebra.
If you go to solve this, what you actually get
is that a is equal to 6/7 and b is equal to 4/7.
So you get that the line-- let me write it here.
You get a is equal to 6/7 and b is equal to 4/7, OK?
And so what that tells you is if you come back over here,
the line we actually want-- I'll write the solution right here--
the line we actually want is the line y equals 6/7 x plus 4/7.
That is our least squares line that is our solution.
OK?
So that is actually-- solving for a line to fit these data,
you get the following solution.
But I want to point out, and what we're going to do,
after this video there will be a challenge problem, to have
you actually finish this.
I want to point out that you can actually
also fit a higher-degree polynomial to these data.
For instance, you could try and use
the technique of least squares to fit a parabola
to these data.
And let me point out what the function would
look like in this case.
So this is for the challenge problem.
For the challenge problem, it now
will be a function of three variables,
so it will look something like this.
We'll say a comma b comma c, and the point that we'll
be wanting to minimize, or the function
we'll be wanting to minimize, we're summing over i, again
from 1 to 3-- I didn't write that down above,
but it was fairly obvious that that
was what we were summing over-- the following thing.
We have y_i minus this big quantity:
a x_i squared plus b*x_i plus c and we square the whole thing.
And so what this is actually giving
you is, just as in the picture before,
this is giving you the difference
between the expected value and the actual value.
So this is your actual y-value, and based on the parabola
you're looking for, what's in parentheses is
your expected y-value, right?
So this would actually be my parabola.
When I evaluate it at x_i, I get a number.
I get a value, and that value will be on the parabola.
This will be the y-value associated
to the x-value from the data.
So this is the actual value.
This is the expected value.
We take the difference and square it.
And because we want to minimize how
we're going to solve this problem
and this is what you'll have to do
is, just like with the line situation,
you're going to take the derivative of this function
with respect to each of these three variables separately.
So you'll take f sub a, and then you'll set that equal to zero.
And then you'll take f sub b, you'll set that equal to zero.
And then you'll take f sub c, and you'll
set that equal to zero.
Again, we set them all equal to zero,
because this is really an optimization problem.
We want to minimize something.
We want to minimize this quantity.
So you take each of those three derivatives,
partial derivatives, set them equal to zero,
and you have a system of three equations with three variables.
And you know how to solve such a system, by using matrices,
for instance.
That would be one way to solve such a system.
And so then you can actually come up
with a formula for the sort of parabola of best fit,
you could think of it as.
And so I'm going to stop there, because I think from here,
you guys will do it.
But I just want to point out this
is where you're going to be starting
this next sort of challenge problem
to find the quadratic of best fit for these three data
points.
So that's where I'll end it.