The following content is provided under a Creative
Commons license. Your support will help MIT
OpenCourseWare continue to offer high quality educational
resources for free. To make a donation or to view
additional materials from hundreds of MIT courses,
visit MIT OpenCourseWare at ocw.mit.edu.
Today we are going to see how to use what we saw last time
about partial derivatives to handle minimization or
maximization problems involving functions of several variables.
Remember last time we said that when we have a function,
say, of two variables, x and y, then we have actually two
different derivatives, partial f, partial x,
also called f sub x, the derivative with respect to
x keeping y constant. And we have partial f,
partial y, also called f sub y, where we vary y and we keep x
as a constant. And now, one thing I didn't
have time to tell you about but hopefully you thought about in
recitation yesterday, is the approximation formula
that tells you what happens if you vary both x and y.
f sub x tells us what happens if we change x a little bit,
by some small amount delta x. f sub y tells us how f changes,
if you change y by a small amount delta y.
If we do both at the same time then the two effects will add up
with each other, because you can imagine that
first you will change x and then you will change y.
Or the other way around. It doesn't really matter.
If we change x by a certain amount delta x,
and if we change y by the amount delta y,
and let's say that we have z= f(x, y) then that changes by an
amount which is approximately f sub x times delta x plus f sub y
times delta y. And that is one of the most
important formulas about partial derivatives.
The intuition for this, again, is just the two effects
of if I change x by a small amount and then I change y.
Well, first changing x will modify f, how much does it
modify f? The answer is the rate change
is f sub x. And if I change y then the rate
of change of f when I change y is f sub y.
So all together I get this change as a value of f.
And, of course, that is only an approximation
formula. Actually, there would be higher
order terms involving second and third derivatives and so on.
One way to justify this -- Sorry.
I was distracted by the microphone.
OK. How do we justify this formula?
Well, one way to think about it is in terms of tangent plane
approximation. Let's think about the tangent
plane with regard to a function f.
We have some pictures to show you.
It will be easier if I show you pictures.
Remember, partial f, partial x was obtained by
looking at the situation where y is held constant.
That means I am slicing the graph of f by a plane that is
parallel to the x, z plane.
And when I change x, z changes, and the slope of
that is going to be the derivative with respect to x.
Now, if I do the same in the other direction then I will have
similarly the slope in a slice now parallel to the y,
z plane that will be partial f, partial y.
In fact, in each case, I have a line.
And that line is tangent to the surface.
Now, if I have two lines tangent to the surface,
well, then together they determine for me the tangent
plane to the surface. Let's try to see how that works.
We know that f sub x and f sub y are the slopes of two tangent
lines to this plane, two tangent lines to the graph.
And let's write down the equations of these lines.
I am not going to write parametric equations.
I am going to write them in terms of x, y,
z coordinates. Let's say that partial f of a
partial x at the given point is equal to a.
That means that we have a line given by the following
conditions. I am going to keep y constant
equal to y0. And I am going to change x.
And, as I change x, z will change at the rate that
is equal to a. That would be z = 0 a(x - x0).
That is how you would describe a line that, I guess,
the one that is plotted in green here, been dissected with
the slice parallel to the x, z plane.
I hold y constant equal to y0. And z is a function of x that
varies with a rate of a. And now if I look similarly at
the other slice, let's say that the partial with
respect to y is equal to b, then I get another line which
is obtained by the fact that z now will depend on y.
And the rate of change with respect to y will be b.
While x is held constant equal to x0.
These two lines are both going to be in the tangent plane to
the surface.
They are both tangent to the graph of f and together they
determine the plane.
And that plane is just given by the formula z = z0 a( x - x0) b
( y - y0). If you look at what happens --
This is the equation of a plane. z equals constant times x plus
constant times y plus constant. And if you look at what happens
if I hold y constant and vary x, I will get the first line.
If I hold x constant and vary y, I get the second line.
Another way to do it, of course,
would provide actually parametric equations of these
lines, get vectors along them and then
take the cross-product to get the normal vector to the plane.
And then get this equation for the plane using the normal
vector. That also works and it gives
you the same formula. If you are curious of the
exercise, do it again using parametrics and using
cross-product to get the plane equation.
That is how we get the tangent plane.
And now what this approximation formula here says is that,
in fact, the graph of a function is close to the tangent
plane. If we were moving on the
tangent plane, this would be an actual
equality. Delta z would be a linear
function of delta x and delta y. And the graph of a function is
near the tangent plane, but is not quite the same,
so it is only an approximation for small delta x and small
delta y. The approximation formula says
the graph of f is close to its tangent plane.
And we can use that formula over here now to estimate how
the value of f changes if I change x and y at the same time.
Questions about that? Now that we have caught up with
what we were supposed to see on Tuesday, I can tell you now
about max and min problems.
That is going to be an application of partial
derivatives to look at optimization problems.
Maybe ten years from now, when you have a real job,
your job might be to actually minimize the cost of something
or maximize the profit of something or whatever.
But typically the function that you will have to strive to
minimize or maximize will depend on several variables.
If you have a function of one variable, you know that to find
its minimum or its maximum you look at the derivative and set
that equal to zero. And you try to then look at
what happens to the function. Here it is going to be kind of
similar, except, of course, we have several
derivatives. For today we will think about a
function of two variables, but it works exactly the same
if you have three variables, ten variables,
a million variables. The first observation is that
if we have a local minimum or a local maximum then both partial
derivatives, so partial f partial x and
partial f partial y, are both zero at the same time.
Why is that? Well, let's say that f of x is
zero. That means when I vary x to
first order the function doesn't change.
Maybe that is because it is going through...
If I look only at the slice parallel to the x-axis then
maybe I am going through the minimum.
But if partial f, partial y is not 0 then
actually, by changing y, I could still make a value
larger or smaller. That wouldn't be an actual
maximum or minimum. It would only be a maximum or
minimum if I stay in the slice. But if I allow myself to change
y that doesn't work. I need actually to know that if
I change y the value will not change either to first order.
That is why you also need partial f, partial y to be zero.
Now, let's say that they are both zero.
Well, why is that enough? It is essentially enough
because of this formula telling me that if both of these guys
are zero then to first order the function doesn't change.
Then, of course, there will be maybe quadratic
terms that will actually turn that, you know,
this won't really say that your function is actually constant.
It will just tell you that maybe it will actually be
quadratic or higher order in delta x and delta y.
That is what you expect to have at a maximum or a minimum.
The condition is the same thing as saying that the tangent plane
to the graph is actually going to be horizontal.
And that is what you want to have.
Say you have a minimum, well, the tangent plane at this
point, at the bottom of the graph is going to be horizontal.
And you can see that on this equation of a tangent plane,
when both these coefficients are 0 that is when the equation
becomes z equals constant: the horizontal plane.
Does that make sense? We will have a name for this
kind of point because, actually,
what we will see very soon is that these conditions are
necessary but are not sufficient.
There are actually other kinds of points where the partial
derivatives are zero. Let's give a name to this.
We say the definition is (x0, y0) is a critical point of f --
-- if the partial derivative, with respect to x,
and partial derivative with respect to y are both zero.
Generally, you would want all the partial derivatives,
no matter how many variables you have, to be zero at the same
time. Let's see an example.
Let's say I give you the function f(x;y)= x^2 - 2xy 3y^2
2x - 2y. And let's try to figure out
whether we can minimize or maximize this.
What we would start doing immediately is taking the
partial derivatives. What is f sub x?
It starts with 2x - 2y 0 2. Remember that y is a constant
so this differentiates to zero. Now, if we do f sub y,
that is going to be 0-2x 6y-2. And what we want to do is set
these things to zero. And we want to solve these two
equations at the same time. An important thing to remember,
and maybe I should have told you a couple of weeks ago
already, if you have two equations to
solve, well, it is very good to try to
simplify them by adding them together or whatever,
but you must keep two equations. If you have two equations,
you shouldn't end up with just one equation out of nowhere.
For example here, we can certainly simplify
things by summing them together. If we add them together,
well, the x's cancel and the constants cancel.
In fact, we are just left with 4y for zero.
That is pretty good. That tells us y should be zero.
But then we should, of course, go back to these and
see what else we know. Well, now it tells us,
if you put y = 0 it tells you 2x 2 = 0.
That tells you x = - 1. We have one critical point that
is (x, y) = (- 1; 0).
Any questions so far? No.
Well, you should have a question.
The question should be how do we know if it is a maximum or a
minimum? Yeah.
If we had a function of one variable, we would decide things
based on the second derivative. And, in fact,
we will see tomorrow how to do things based on the second
derivative. But that is kind of tricky
because there are a lot of second derivatives.
I mean we already have two first derivatives.
You can imagine that if you keep taking partials you may end
up with more and more, so we will have to figure out
carefully what the condition should be.
We will do that tomorrow. For now, let's just try to look
a bit at how do we understand these things by hand?
In fact, let me point out to you immediately that there is
more than maxima and minima. Remember, we saw the example of
x^2 y^2. That has a critical point.
That critical point is obviously a minimum.
And, of course, it could be a local minimum
because it could be that if you have a more complicated function
there is indeed a minimum here, but then elsewhere the function
drops to a lower value. We call that just a local
minimum to say that it is a minimum if you stick two values
that are close enough to that point.
Of course, you also have local maximum, which I didn't plot,
but it is easy to plot. That is a local maximum.
But there is a third example of critical point,
and that is a saddle point. The saddle point,
it is a new phenomena that you don't really see in single
variable calculus. It is a critical point that is
neither a minimum nor a maximum because, depending on which
direction you look in, it's either one or the other.
See the point in the middle, at the origin,
is a saddle point. If you look at the tangent
plane to this graph, you will see that it is
actually horizontal at the origin.
You have this mountain pass where the ground is horizontal.
But, depending on which direction you go,
you go up or down. So, we say that a point is a
saddle point if it is neither a minimum or a maximum.
Possibilities could be a local min, a local max or a saddle.
Tomorrow we will see how to decide which one it is,
in general, using second derivatives.
For this time, let's just try to do it by
hand. I just want to observe,
in fact, I can try to, you know,
these examples that I have here,
they are x^2 y^2, y^2 - x^2, they are sums or differences of
squares. And, if we know that we can put
things as sum of squares for example, we will be done.
Let's try to express this maybe as the square of something.
The main problem is this 2xy. Observe we know something that
starts with x^2 - 2xy but is actually a square of something
else. It would be x^2 - 2xy y^2,
not plus 3y2. Let's try that.
So, we are going to complete the square.
I am going to say it is x minus y squared, so it gives me the
first two terms and also the y2. Well, I still need to add two
more y^2, and I also need to add, of course,
the 2x and - 2y. It is still not simple enough
for my taste. I can actually do better.
These guys look like a sum of squares, but here I have this
extra stuff, 2x - 2y. Well, that is 2 (x - y).
It looks like maybe we can modify this and make this into
another square. So, in fact,
I can simplify this further to (x - y 1)^2.
That would be (x - y)^2 2( x - y), and then there is a plus
one. Well, we don't have a plus one
so let's remove it by subtracting one.
And I still have my 2y^2. Do you see why this is the same
function? Yeah.
Again, if I expand x minus y plus one squared,
I get (x - y)^2 2 (x - y) 1. But I will have minus one that
will cancel out and then I have a plus 2y^2.
Now, what I know is a sum of two squared minus one.
And this critical point, (x,y) = (-1;0),
that is actually when this is zero and that is zero,
so that is the smallest value. This is always greater or equal
to zero, the same with that one, so that is always at least
minus one. And minus one happens to be the
value at the critical point. So, it is a minimum.
Now, of course here I was very lucky.
I mean, generally, I couldn't expect things to
simplify that much. In fact, I cheated.
I started from that, I expanded, and then that is
how I got my example. The general method will be a
bit different, but you will see it will
actually also involve completing squares.
Just there is more to it than what we have seen.
We will come back to this tomorrow.
Sorry? How do I know that this equals
-- How do I know that the whole function is greater or equal to
negative one? Well, I wrote f of x,
y as something squared plus 2y^2 - 1.
This squared is always a positive number and not a
negative. It is a square.
The square of something is always non-negative.
Similarly, y^2 is also always non-negative.
So if you add something that is at least zero plus something
that is at least zero and you subtract one,
you get always at least minus one.
And, in fact, the only way you can get minus
one is if both of these guys are zero at the same time.
That is how I get my minimum. More about this tomorrow.
In fact, what I would like to tell you
about now instead is a nice application of min,
max problems that maybe you don't think of as a min,
max problem that you will see. I mean you will think of it
that way because probably your calculator can do it for you or,
if not, your computer can do it for you.
But it is actually something where the theory is based on
minimization in two variables. Very often in experimental
sciences you have to do something called least-squares
intercalation. And what is that about?
Well, it is the idea that maybe you do some experiments and you
record some data. You have some data x and some
data y. And, I don't know,
maybe, for example, x is -- Maybe your measuring
frogs and you're trying to measure how bit the frog leg is
compared to the eyes of the frog,
or you're trying to measure something.
And if you are doing chemistry then it could be how much you
put of some reactant and how much of the output product that
you wanted to synthesize generated.
All sorts of things. Make up your own example.
You measure basically, for various values of x,
what the value of y ends up being.
And then you like to claim these points are kind of
aligned. And, of course,
to a mathematician they are not aligned.
But, to an experimental scientist, that is evidence that
there is a relation between the two.
And so you want to claim -- And in your paper you will actually
draw a nice little line like that.
The functions depend linearly on each of them.
The question is how do we come up with that nice line that
passes smack in the middle of the points?
The question is, given experimental data xi,
yi -- Maybe I should actually be more precise.
You are given some experimental data.
You have data points x1, y1, x2, y2 and so on,
xn, yn, the question would be find the
"best fit" line of a form y equals ax b
that somehow approximates very well this data.
You can also use that right away to predict various things.
For example, if you look at your new
homework, actually the first problem asks
you to predict how many iPods will be on this planet in ten
years looking at past sales and how they behave.
One thing, right away, before you lose all the money
that you don't have yet, you cannot use that to predict
the stock market. So, don't try to use that to
make money. It doesn't work.
One tricky thing here that I want to draw your attention to
is what are the unknowns here? The natural answer would be to
say that the unknowns are x and y.
That is not actually the case. We are not going to solve for
some x and y. I mean we have some values
given to us. And, when we are looking for
that line, we don't really care about the perfect value of x.
What we care about is actually these coefficients a and b that
will tell us what the relation is between x and y.
In fact, we are trying to solve for a and b that will give us
the nicest possible line for these points.
The unknowns, in our equations,
will have to be a and b, not x and y.
The question really is find the "best"
a and b. And, of course,
we have to decide what we mean by best.
Best will mean that we minimize some function of a and b that
measures the total errors that we are making when we are
choosing this line compared to the experimental data.
Maybe, roughly speaking, it should measure how far these
points are from the line. But now there are various ways
to do it. And a lot of them are valid
they give you different answers. You have to decide what it is
that you prefer. For example,
you could measure the distance to the line by projecting
perpendicularly. Or you could measure instead,
for a given value of x, the difference between the
experimental value of y and the predicted one.
And that is often more relevant because these guys actually may
be expressed in different units. They are not the same type of
quantity. You cannot actually combine
them arbitrarily. Anyway, the convention is
usually we measure distance in this way.
Next, you could try to minimize the largest distance.
Say we look at who has the largest error and we make that
the smallest possible. The drawback of doing that is
experimentally very often you have one data point that is not
good because maybe you fell asleep in front of the
experiment. And so you didn't measure the
right thing. You tend to want to not give
too much importance to some data point that is far away from the
others. Maybe instead you want to
measure the average distance or maybe you want to actually give
more weight to things that are further away.
And then you don't want to do the distance with a square of
the distance. There are various possible
answers, but one of them gives us actually a particularly nice
formula for a and b. And so that is why it is the
universally used one. Here it says list squares.
That's because we will measure, actually, the sum of the
squares of the errors. And why do we do that?
Well, part of it is because it looks good.
When you see this plot in scientific papers they really
look like the line is indeed the ideal line.
And the second reason is because actually the
minimization problem that we will get is particularly simple,
well-posed and easy to solve. So we will have a nice formula
for the best a and the best b. If you have a method that is
simple and gives you a good answer then that is probably
good. We have to define best.
Here it is in the sense of minimizing the total square
error. Or maybe I should say total
square deviation instead. What do I mean by this?
The deviation for each data point is the difference between
what you have measured and what you are predicting by your
model. That is the difference between
y1 and axi plus b. Now, what we will do is try to
minimize the function capital D, which is just the sum for all
the data points of the square of a deviation.
Let me go over this again. This is a function of a and b.
Of course there are a lot of letters in here,
but xi and yi in real life there will be numbers given to
you. There will be numbers that you
have measured. You have measured all of this
data. They are just going to be
numbers. You put them in there and you
get a function of a and b. Any questions?
How do we minimize this function of a and b?
Well, let's use your knowledge. Let's actually look for a
critical point. We want to solve for partial d
over partial a= 0, partial d over partial b = 0.
That is how we look for critical points.
Let's take the derivative of this with respect to a.
Well, the derivative of a sum is sum of the derivatives.
And now we have to take the derivative of this quantity
squared. Remember, we take the
derivative of the square. We take twice this quantity
times the derivative of what we are squaring.
We will get 2(yi - axi) b times the derivative of this with
respect to a. What is the derivative of this
with respect to a? Negative xi, exactly.
And so we will want this to be 0.
And partial d over partial b, we do the same thing,
but different shading with respect to b instead of with
respect to a. Again, the sum of squares twice
yi minus axi equals b times the derivative of this with respect
to b is, I think, negative one.
Those are the equations we have to solve.
Well, let's reorganize this a little bit.
The first equation. See, there are a's and there
are b's in these equations. I am going to just look at the
coefficients of a and b. If you have good eyes,
you can see probably that these are actually linear equations in
a and b. There is a lot of clutter with
all these x's and y's all over the place.
Let's actually try to expand things and make that more
apparent. The first thing I will do is
actually get rid of these factors of two.
They are just not very important.
I can simplify things. Next, I am going to look at the
coefficient of a. I will get basically a times xi
squared. Let me just do it and should be
clear. I claim when we simplify this
we get xi squared times a plus xi times b minus xiyi.
And we set this equal to zero. Do you agree that this is what
we get when we expand that product?
Yeah. Kind of? OK. Let's do the other one.
We just multiply by minus one, so we take the opposite of that
which would be axi plus b. I will write that as xia plus b
minus yi. Sorry. I forgot the n here.
And let me just reorganize that by actually putting all the a's
together. That means I will have sum of
all the xi2 times a plus sum of xib minus sum of xiyi equal to
zero.
If I rewrite this, it becomes sum of xi2 times a
plus sum of the xi's time b, and let me move the other guys
to the other side, equals sum of xiyi.
And that one becomes sum of xi times a.
Plus how many b's do I get on this one?
I get one for each data point. When I sum them together,
I will get n. Very good.
N times b equals sum of yi. Now, this quantities look
scary, but they are actually just numbers.
For example, this one, you look at all your
data points. For each of them you take the
value of x and you just sum all these numbers together.
What you get, actually, is a linear system in
a and b, a two by two linear system.
And so now we can solve this for a and b.
In practice, of course, first you plug in
the numbers for xi and yi and then you solve the system that
you get. And we know how to solve two by
two linear systems, I hope.
That's how we find the best fit line.
Now, why is that going to be the best one instead of the
worst one? We just solved for a critical
point. That could actually be a
maximum of this error function D.
We will have the answer to that next time, but trust me.
If you really want to go over the second derivative test that
we will see tomorrow and apply it in this case,
it is quite hard to check, but you can see it is actually
a minimum. I will just say -- -- we can
show that it is a minimum. Now, the event with the linear
case is the one that we are the most familiar with.
Least-squares interpolation actually works in much more
general settings. Because instead of fitting for
the best line, if you think it has a different
kind of relation then maybe you can fit in using a different
kind of formula. Let me actually illustrate that
with an example. I don't know if you are
familiar with Moore's law. It is something that is
supposed to tell you how quickly basically computer chips become
smarter faster and faster all the time.
It's a law that says things about the number of transistors
that you can fit onto a computer chip.
Here I have some data about -- Here is data about the number of
transistors on a standard PC processor as a function of time.
And if you try to do a best-line fit,
well, it doesn't seem to follow a linear trend.
On the other hand, if you plug the diagram in the
log scale, the log of a number of
transitions as a function of time,
then you get a much better line. And so, in fact,
that means that you had an exponential relation between the
number of transistors and time. And so, actually that's what
Moore's law says. It says that the number of
transistors in the chip doubles every 18 months or every two
years. They keep changing the
statement. How do we find the best
exponential fit? Well, an exponential fit would
be something of a form y equals a constant times exponential of
a times x. That is what we want to look at.
Well, we could try to minimize a square error like we did
before. That doesn't work well at all.
The equations that you get are very complicated.
You cannot solve them. But remember what I showed you
on this log plot. If you plot the log of y as a
function of x then suddenly it becomes a linear relation.
Observe, this is the same as ln of y equals ln of c plus ax.
And that is the linear best fit. What you do is you just look
for the best straight line fit for the log of y.
That is something we already know.
But you can also do, for example,
let's say that we have something more complicated.
Let's say that we have actually a quadratic law.
For example, y is of the form ax^2 bx c.
And, of course, you are trying to find somehow
the best. That would mean here fitting
the best parabola for your data points.
Well, to do that, you would need to find a,
b and c. And now you will have actually
a function of a, b and c, which would be the sum
of the old data points of the square deviation.
And, if you try to solve for critical points,
now you will have three equations involving a,
b and c, in fact, you will find a three
by three linear system. And it works the same way.
Just you have a little bit more data.
Basically, you see that this best fit problems are an example
of a minimization problem that maybe you didn't expect to see
minimization problems come in. But that is really the way to
handle these questions. Tomorrow we will go back to the
question of how do we decide whether it is a minimum or a
maximum. And we will continue exploring
in terms of several variables.