1 00:00:01,550 --> 00:00:03,920 The following content is provided under a Creative 2 00:00:03,920 --> 00:00:05,310 Commons license. 3 00:00:05,310 --> 00:00:07,520 Your support will help MIT OpenCourseWare 4 00:00:07,520 --> 00:00:11,610 continue to offer high-quality educational resources for free. 5 00:00:11,610 --> 00:00:14,180 To make a donation or to view additional materials 6 00:00:14,180 --> 00:00:18,140 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:18,140 --> 00:00:19,026 at ocw.mit.edu. 8 00:00:22,300 --> 00:00:24,850 GILBERT STRANG: So I've worked hard over the weekend. 9 00:00:24,850 --> 00:00:27,150 I figured out what I was doing last time 10 00:00:27,150 --> 00:00:31,260 and what I'm doing this time and improved the notes. 11 00:00:31,260 --> 00:00:36,060 So you'll get a new set of notes on the last lecture 12 00:00:36,060 --> 00:00:38,720 and on this one. 13 00:00:38,720 --> 00:00:43,770 And I kind of got a better picture of what we're doing. 14 00:00:43,770 --> 00:00:47,370 And that board is aiming to describe 15 00:00:47,370 --> 00:00:52,000 the large picture of what we're doing last time and this time. 16 00:00:52,000 --> 00:01:01,250 So last time was about changes in A inverse when A changed. 17 00:01:01,250 --> 00:01:06,710 This time is about changes in eigenvalues and changes 18 00:01:06,710 --> 00:01:08,890 in singular values when A change. 19 00:01:08,890 --> 00:01:13,340 As you can imagine, this is a natural important situation. 20 00:01:13,340 --> 00:01:20,060 Matrices move, and therefore, their inverses change, 21 00:01:20,060 --> 00:01:25,360 their eigenvalues change, their singular values change. 22 00:01:25,360 --> 00:01:28,870 And you hope for a formula. 23 00:01:28,870 --> 00:01:34,620 Well, so we did have a formula for last time 24 00:01:34,620 --> 00:01:39,730 for the change in the inverse matrix. 25 00:01:39,730 --> 00:01:45,300 And I didn't get every u and v transpose in the right place 26 00:01:45,300 --> 00:01:52,560 in the video or in the first version of the notes, 27 00:01:52,560 --> 00:01:59,040 but I hope that that formula, that that Woodbury Morrison 28 00:01:59,040 --> 00:02:02,040 formula will be correct this time. 29 00:02:06,080 --> 00:02:10,740 So I won't go back over that part. 30 00:02:10,740 --> 00:02:14,520 But I realize also there is another question 31 00:02:14,520 --> 00:02:19,050 that we can answer when the change is very small, 32 00:02:19,050 --> 00:02:27,510 when the change in A is dA or delta A, a small change. 33 00:02:27,510 --> 00:02:30,590 And that's, of course, what calculus is about. 34 00:02:30,590 --> 00:02:37,370 So I have to sort of parallel topics here. 35 00:02:37,370 --> 00:02:46,770 What is the derivative when the change is infinitesimal? 36 00:02:46,770 --> 00:02:55,590 And what is the actual change when the change is finite size? 37 00:02:55,590 --> 00:02:59,250 So now, let me say what we can do and what we can't do. 38 00:03:02,890 --> 00:03:08,140 Oh, I'll start out by figuring out what the derivative is 39 00:03:08,140 --> 00:03:08,980 for the inverse. 40 00:03:08,980 --> 00:03:11,890 So that's like completing the last time 41 00:03:11,890 --> 00:03:14,320 for infinitesimal changes. 42 00:03:14,320 --> 00:03:19,960 Then I'll move on to changes in the eigenvalues and singular 43 00:03:19,960 --> 00:03:21,220 values. 44 00:03:21,220 --> 00:03:26,330 And there, you cannot expect an exact formula. 45 00:03:26,330 --> 00:03:33,110 We had a formula that was exact, apart from any typos, for this. 46 00:03:33,110 --> 00:03:34,790 And we'll find a formula for this, 47 00:03:34,790 --> 00:03:37,520 and we'll find a formula for that and for that. 48 00:03:37,520 --> 00:03:39,940 Well, that one will come from this one. 49 00:03:39,940 --> 00:03:43,580 So this will be a highlight today. 50 00:03:43,580 --> 00:03:46,505 How do the eigenvalues change when the matrix changes? 51 00:03:49,170 --> 00:03:52,710 But we won't be able to do parallel to this, 52 00:03:52,710 --> 00:03:54,310 we won't be able to-- 53 00:03:54,310 --> 00:03:58,770 oh, we will be able to do something for finite changes. 54 00:03:58,770 --> 00:03:59,490 That's important. 55 00:03:59,490 --> 00:04:03,090 Mathematics would have to keep hitting that problem 56 00:04:03,090 --> 00:04:04,860 until it got somewhere. 57 00:04:04,860 --> 00:04:07,770 So I won't get an exact formula for that change. 58 00:04:07,770 --> 00:04:09,000 That's too much. 59 00:04:09,000 --> 00:04:12,030 But I'll get inequalities. 60 00:04:12,030 --> 00:04:13,710 How big that change could be. 61 00:04:13,710 --> 00:04:15,580 What can I say about it? 62 00:04:15,580 --> 00:04:19,769 So these are highly interesting. 63 00:04:19,769 --> 00:04:23,670 May I start with completing the last lecture? 64 00:04:23,670 --> 00:04:26,400 What is the derivative of the inverse? 65 00:04:26,400 --> 00:04:30,420 So I'm thinking here, so what's the setup? 66 00:04:30,420 --> 00:04:38,430 The setup is my matrix A depends on time, on t. 67 00:04:41,290 --> 00:04:45,030 And it has an inverse. 68 00:04:45,030 --> 00:04:49,940 A inverse depends on t. 69 00:04:49,940 --> 00:04:52,230 And if I know this dependence, in other words, 70 00:04:52,230 --> 00:04:57,950 if I know dA dt, how the matrix is depending on t, 71 00:04:57,950 --> 00:04:59,840 then I hope I could figure out what 72 00:04:59,840 --> 00:05:04,010 the derivative of A inverse is. 73 00:05:04,010 --> 00:05:06,360 We should be able to do this. 74 00:05:06,360 --> 00:05:08,630 So let me just start with-- 75 00:05:08,630 --> 00:05:13,310 it's not hard and it complements this one 76 00:05:13,310 --> 00:05:18,770 by doing the calculus case, the infinitesimal change. 77 00:05:18,770 --> 00:05:21,650 So I want to get to that. 78 00:05:27,170 --> 00:05:29,420 I can figure out the change in A. 79 00:05:29,420 --> 00:05:43,010 And my job is to find the derivative of A inverse. 80 00:05:43,010 --> 00:05:50,720 So here's a handy identity. 81 00:05:50,720 --> 00:05:53,090 Can I just put this here? 82 00:05:53,090 --> 00:05:55,210 So here's my usual identity. 83 00:06:00,150 --> 00:06:03,890 So as last time, I start with a finite change 84 00:06:03,890 --> 00:06:05,935 because calculus always does that, right. 85 00:06:05,935 --> 00:06:10,000 It starts with a delta t and then it goes to 0. 86 00:06:10,000 --> 00:06:13,820 So here I am up at a full size change. 87 00:06:13,820 --> 00:06:24,550 So I think that this is equal to B inverse A minus B A inverse. 88 00:06:24,550 --> 00:06:30,340 And if it's true, it's a pretty cool formula. 89 00:06:30,340 --> 00:06:34,190 And, look, it is true, because over on this right-hand side, 90 00:06:34,190 --> 00:06:36,880 I have B inverse times A A inverse. 91 00:06:36,880 --> 00:06:37,780 That's the identity. 92 00:06:37,780 --> 00:06:39,640 So that's my B inverse. 93 00:06:39,640 --> 00:06:43,180 And I have the minus, the B inverse B is the identity. 94 00:06:43,180 --> 00:06:44,230 There's A inverse. 95 00:06:44,230 --> 00:06:46,690 It's good, right? 96 00:06:46,690 --> 00:06:52,180 So from that, well, I could actually 97 00:06:52,180 --> 00:07:04,240 learn from that the rank of this equals the rank of this. 98 00:07:08,320 --> 00:07:11,340 That's a point that I made from the big formula. 99 00:07:11,340 --> 00:07:13,470 But now, we can see it from an easy formula. 100 00:07:17,310 --> 00:07:19,440 Everywhere here, I'm assuming that A and B 101 00:07:19,440 --> 00:07:21,300 are invertible matrices. 102 00:07:21,300 --> 00:07:24,300 So when I multiply by an invertible matrix, 103 00:07:24,300 --> 00:07:26,530 that does not change the rank. 104 00:07:26,530 --> 00:07:28,770 So those have the same ranks. 105 00:07:28,770 --> 00:07:31,650 But I want to get further than that. 106 00:07:31,650 --> 00:07:35,220 I want to find this. 107 00:07:35,220 --> 00:07:37,380 So how do I go? 108 00:07:37,380 --> 00:07:40,820 How do I go forward with that job 109 00:07:40,820 --> 00:07:43,350 to find the derivative of the inverse? 110 00:07:43,350 --> 00:07:51,660 Well, I'm going to call this a change in A inverse. 111 00:07:51,660 --> 00:07:56,015 And over here, I'll have B will be-- 112 00:07:58,810 --> 00:08:02,920 yeah, OK, let's see, am I right? 113 00:08:02,920 --> 00:08:03,670 Yeah. 114 00:08:03,670 --> 00:08:07,630 So B inverse will be-- 115 00:08:07,630 --> 00:08:14,680 this is A plus delta A inverse. 116 00:08:14,680 --> 00:08:18,220 And this is-- well, that's A minus B. So 117 00:08:18,220 --> 00:08:29,680 that's really minus delta A. From A to B is the change. 118 00:08:29,680 --> 00:08:33,429 Here, I'm looking at the difference A minus B. 119 00:08:33,429 --> 00:08:35,049 So it's minus a change. 120 00:08:35,049 --> 00:08:37,960 And here, I have A inverse. 121 00:08:37,960 --> 00:08:45,640 I haven't done anything except to introduce this delta 122 00:08:45,640 --> 00:08:49,240 and get B out of it and brought delta in. 123 00:08:49,240 --> 00:08:52,130 Now, I'm going to do calculus. 124 00:08:52,130 --> 00:09:00,220 So I'm thinking of B as there's a sort of a delta t. 125 00:09:00,220 --> 00:09:04,370 And I'm going to divide by both sides by delta t. 126 00:09:04,370 --> 00:09:05,830 I have to do this if I want-- 127 00:09:09,200 --> 00:09:14,840 and now, I'll let delta t go to 0. 128 00:09:14,840 --> 00:09:17,030 So calculus appears. 129 00:09:19,790 --> 00:09:27,390 Finally, our-- I won't say enemy calculus, 130 00:09:27,390 --> 00:09:31,730 but there is a sort of like competition 131 00:09:31,730 --> 00:09:34,520 between linear algebra and calculus 132 00:09:34,520 --> 00:09:39,700 for college mathematics. 133 00:09:39,700 --> 00:09:44,260 Calculus has had far, far too much time and attention. 134 00:09:44,260 --> 00:09:46,930 It like it gets three or four semesters 135 00:09:46,930 --> 00:09:50,440 of calculus for people who don't get any linear algebra. 136 00:09:50,440 --> 00:09:53,260 I'm glad this won't be on the video, but I'm afraid it will. 137 00:09:55,960 --> 00:09:59,680 Anyway, of course, calculus is fine in its place. 138 00:09:59,680 --> 00:10:01,450 So here's its place. 139 00:10:01,450 --> 00:10:04,270 Now let delta t go to 0. 140 00:10:04,270 --> 00:10:06,010 So what does this equation become? 141 00:10:11,750 --> 00:10:16,510 Then everybody knows that as the limit of delta t goes to 0, 142 00:10:16,510 --> 00:10:20,560 I replace deltas by-- 143 00:10:20,560 --> 00:10:25,270 so this delta A divided by delta t that has a meaning. 144 00:10:25,270 --> 00:10:28,940 The top has a meaning and the bottom has for me. 145 00:10:28,940 --> 00:10:33,580 But then the limit, it's the ratio that has a meaning. 146 00:10:33,580 --> 00:10:37,255 So dA by itself, I don't attach a meaning to that. 147 00:10:37,255 --> 00:10:38,790 That's infinitesimal. 148 00:10:38,790 --> 00:10:42,720 It's the limit, so that's why I wanted a delta over a delta 149 00:10:42,720 --> 00:10:44,160 so I could do calculus. 150 00:10:44,160 --> 00:10:47,070 So what happens now is delta t goes to 0. 151 00:10:47,070 --> 00:10:49,500 And, of course, as delta t goes to 0, 152 00:10:49,500 --> 00:10:51,720 that carries delta A to 0. 153 00:10:51,720 --> 00:10:53,160 So that becomes A inverse. 154 00:10:55,980 --> 00:11:01,390 And what does this approach as delta t goes to 0? 155 00:11:01,390 --> 00:11:03,710 dA dt with that minus sign. 156 00:11:03,710 --> 00:11:06,650 Oh, I've got to remember the minus sign. 157 00:11:06,650 --> 00:11:09,890 The minus sign is in here. 158 00:11:09,890 --> 00:11:12,060 So I'm bringing out the minus sign. 159 00:11:12,060 --> 00:11:15,380 Then this was A inverse, as we had. 160 00:11:15,380 --> 00:11:16,930 And that's dA dt. 161 00:11:20,000 --> 00:11:22,310 And that's A inverse. 162 00:11:22,310 --> 00:11:26,660 That's our formula, a nice formula, 163 00:11:26,660 --> 00:11:32,150 which sort of belongs in people's knowledge. 164 00:11:32,150 --> 00:11:38,250 You recognize that if A was a 1 by 1 matrix, 165 00:11:38,250 --> 00:11:45,060 we could call it x, instead of A. If A was a 1 by 1 matrix x, 166 00:11:45,060 --> 00:11:49,830 then I'm saying the formula for the derivative of 1 167 00:11:49,830 --> 00:11:51,570 over x, right? 168 00:11:51,570 --> 00:11:56,010 A inverse just 1 by 1 case is just 1 over x. 169 00:11:56,010 --> 00:12:00,120 So the derivative of 1-- or maybe t, I should be saying. 170 00:12:00,120 --> 00:12:05,580 If A is just t, then the derivative of 1 over t with 171 00:12:05,580 --> 00:12:09,790 respect to t is....? 172 00:12:09,790 --> 00:12:11,680 Is minus 1 over t squared. 173 00:12:15,580 --> 00:12:18,010 The 1 by 1 case we know. 174 00:12:18,010 --> 00:12:19,330 That's what calculus does. 175 00:12:19,330 --> 00:12:23,640 And now we're able to do the n by n case. 176 00:12:23,640 --> 00:12:26,520 So that's just like good. 177 00:12:26,520 --> 00:12:33,330 And then it's sort of parallel to formulas like this, where 178 00:12:33,330 --> 00:12:37,020 this delta A has not gone to 0. 179 00:12:37,020 --> 00:12:39,990 It's full size, but low rank. 180 00:12:39,990 --> 00:12:41,280 That was the point. 181 00:12:41,280 --> 00:12:44,880 Actually, the formula would apply if the rank wasn't low. 182 00:12:44,880 --> 00:12:48,870 But the interest is in low rank here. 183 00:12:48,870 --> 00:12:50,130 Are we good for this? 184 00:12:50,130 --> 00:12:53,370 That's really the completion of last time's lecture 185 00:12:53,370 --> 00:12:57,500 with derivatives. 186 00:12:57,500 --> 00:13:02,530 OK, come back to here, to the new thing now, lambdas. 187 00:13:02,530 --> 00:13:05,280 Let's focus on lambdas, eigenvalues. 188 00:13:05,280 --> 00:13:09,960 How does the eigenvalue change when the matrix changes? 189 00:13:09,960 --> 00:13:12,960 How does the eigenvalue change when the matrix changes? 190 00:13:12,960 --> 00:13:16,150 So I have two possibilities. 191 00:13:16,150 --> 00:13:19,860 One is small change when I'm doing calculus 192 00:13:19,860 --> 00:13:22,530 and I'm letting a delta t go to 0. 193 00:13:22,530 --> 00:13:27,570 The other is full size, order 1 change, 194 00:13:27,570 --> 00:13:29,130 where I will not be able to give you 195 00:13:29,130 --> 00:13:32,010 a formula for the new lambdas, but I'll 196 00:13:32,010 --> 00:13:36,120 be able to tell you important facts about them. 197 00:13:36,120 --> 00:13:39,240 So this is today's lecture now. 198 00:13:39,240 --> 00:13:42,510 You could say that's the completion of Friday's lecture. 199 00:13:42,510 --> 00:13:45,600 What about d lambda dt? 200 00:13:45,600 --> 00:13:46,590 It's a nice formula. 201 00:13:49,560 --> 00:13:51,300 Its proof is fun too. 202 00:13:51,300 --> 00:13:54,920 I was very happy about this proof. 203 00:13:54,920 --> 00:13:58,320 OK, so I guess calculus is showing up here 204 00:13:58,320 --> 00:13:59,760 on this middle board. 205 00:13:59,760 --> 00:14:03,910 So how do I start with the eigenvalues? 206 00:14:03,910 --> 00:14:07,260 Well, start with what I know. 207 00:14:07,260 --> 00:14:14,750 So these are facts, you could say, 208 00:14:14,750 --> 00:14:17,020 that I have to get the eigenvalues into it. 209 00:14:17,020 --> 00:14:20,230 And, of course, eigenvalues have to come with eigenvectors. 210 00:14:20,230 --> 00:14:24,240 So I'll again use A of t. 211 00:14:24,240 --> 00:14:26,350 It will be depending on t. 212 00:14:26,350 --> 00:14:29,290 And an eigenvector that depends on t 213 00:14:29,290 --> 00:14:32,350 is an eigenvalue that depends on t times 214 00:14:32,350 --> 00:14:35,036 and eigenvector that depends on t. 215 00:14:35,036 --> 00:14:36,990 Good? 216 00:14:36,990 --> 00:14:42,580 That's fact one that we plan to take the derivative of somehow. 217 00:14:42,580 --> 00:14:47,650 There's also a second fact that comes into play here. 218 00:14:47,650 --> 00:14:52,520 What's the deal on the eigenvalues of A transpose? 219 00:14:52,520 --> 00:14:54,270 They are the same. 220 00:14:54,270 --> 00:14:56,490 The eigenvalues of A transpose are the same 221 00:14:56,490 --> 00:14:58,110 as the eigenvalues of A. 222 00:14:58,110 --> 00:15:00,690 Are the eigenvectors the same? 223 00:15:00,690 --> 00:15:01,860 Not usually. 224 00:15:01,860 --> 00:15:05,500 Of course, if the matrix was symmetric, 225 00:15:05,500 --> 00:15:09,150 then A and A transpose are just the same thing. 226 00:15:09,150 --> 00:15:13,620 So A transpose would have that eigenvalue-- 227 00:15:13,620 --> 00:15:14,160 eigenvector. 228 00:15:14,160 --> 00:15:17,850 But, generally, it has a different eigenvector. 229 00:15:17,850 --> 00:15:22,890 And really to keep a sort of separate from this one, 230 00:15:22,890 --> 00:15:24,530 let's call it y. 231 00:15:24,530 --> 00:15:26,970 It will have the same eigenvalue. 232 00:15:26,970 --> 00:15:28,590 I'm going to call it y. 233 00:15:28,590 --> 00:15:33,760 But I'm going to make it a row vector, because A transpose is 234 00:15:33,760 --> 00:15:34,270 what-- 235 00:15:34,270 --> 00:15:36,340 instead of writing down A transpose, 236 00:15:36,340 --> 00:15:39,130 I'm going to stay with A, but put the eigenvalue 237 00:15:39,130 --> 00:15:40,240 on the left side. 238 00:15:40,240 --> 00:15:43,000 So here's is the eigenvalue-- 239 00:15:43,000 --> 00:15:48,260 eigenvector for A on the left. 240 00:15:48,260 --> 00:15:53,680 And it has the same eigenvalue times that eigenvector. 241 00:15:53,680 --> 00:15:58,100 But that eigenvector is a row eigenvector, of course. 242 00:15:58,100 --> 00:16:01,230 This is an equality between rows. 243 00:16:01,230 --> 00:16:05,150 A row times my matrix gives a row. 244 00:16:05,150 --> 00:16:08,450 So that's the eigenvalues of-- 245 00:16:08,450 --> 00:16:10,210 and it has the same eigenvalues. 246 00:16:10,210 --> 00:16:14,650 So this is totally parallel to that, totally parallel. 247 00:16:14,650 --> 00:16:17,770 And maybe sort of less-- 248 00:16:17,770 --> 00:16:25,450 definitely less seen, but it's just 249 00:16:25,450 --> 00:16:27,970 the same thing for A transpose. 250 00:16:27,970 --> 00:16:30,910 Everybody sees that if I transpose this equation, 251 00:16:30,910 --> 00:16:33,400 then I've got something that looks like that. 252 00:16:33,400 --> 00:16:35,320 But I'd rather have it this way. 253 00:16:35,320 --> 00:16:38,960 Now, one more fact I need. 254 00:16:38,960 --> 00:16:46,090 There is-- there has to be some normalization. 255 00:16:46,090 --> 00:16:47,800 What should be the length of these? 256 00:16:47,800 --> 00:16:50,530 Right now, x could have any length. 257 00:16:50,530 --> 00:16:52,750 y could have any length. 258 00:16:52,750 --> 00:16:55,360 And there's a natural normalization, 259 00:16:55,360 --> 00:17:00,890 which is y transpose times x equal to 1. 260 00:17:03,520 --> 00:17:05,415 That normalizes the two. 261 00:17:05,415 --> 00:17:09,160 It doesn't tell me the length of x or the length of y. 262 00:17:09,160 --> 00:17:13,089 But it tells me, the key thing, the length of both. 263 00:17:13,089 --> 00:17:20,050 So what I've got there is tracking along one eigenvalue 264 00:17:20,050 --> 00:17:22,480 and its pair of eigenvectors. 265 00:17:22,480 --> 00:17:25,480 And you're always welcome to think of the symmetric case 266 00:17:25,480 --> 00:17:27,400 when y and x are the same. 267 00:17:27,400 --> 00:17:29,740 And then I would call them q. 268 00:17:29,740 --> 00:17:32,360 Oh, well, I would call them q if it was a symmetric matrix. 269 00:17:36,040 --> 00:17:39,850 So if it's a symmetric matrix, both eigenvectors 270 00:17:39,850 --> 00:17:41,350 would be called q. 271 00:17:41,350 --> 00:17:44,413 And this would be saying that q is a-- 272 00:17:44,413 --> 00:17:45,330 AUDIENCE: Unit vector. 273 00:17:45,330 --> 00:17:47,070 GILBERT STRANG: Unit vector, right. 274 00:17:47,070 --> 00:17:49,640 So this is all stuff we know. 275 00:17:49,640 --> 00:17:55,910 And actually, maybe I should write it in matrix notation, 276 00:17:55,910 --> 00:17:57,140 because it's important. 277 00:18:04,390 --> 00:18:06,350 That's for one eigenvector. 278 00:18:06,350 --> 00:18:09,650 This is for all of them at once. 279 00:18:09,650 --> 00:18:11,120 Everybody's with it? 280 00:18:11,120 --> 00:18:15,120 The x's are the columns of x. 281 00:18:15,120 --> 00:18:19,620 And lambda is the diagonal matrix of lambdas. 282 00:18:19,620 --> 00:18:21,750 And it has to sit on the right so that it 283 00:18:21,750 --> 00:18:24,190 will multiply those columns. 284 00:18:24,190 --> 00:18:27,120 So this is like all eigenvectors at once. 285 00:18:27,120 --> 00:18:29,310 What would this one be? 286 00:18:29,310 --> 00:18:39,390 This would be like y transpose A equals A-- 287 00:18:39,390 --> 00:18:41,690 AUDIENCE: y transpose inverse? 288 00:18:41,690 --> 00:18:43,110 GILBERT STRANG: y transpose, yes-- 289 00:18:43,110 --> 00:18:47,100 equals-- and probably these are multiplied-- 290 00:18:50,460 --> 00:18:53,460 I feel wrong if I write y transpose here. 291 00:18:53,460 --> 00:18:58,260 Like here, the x was on the right and on the left. 292 00:18:58,260 --> 00:19:06,310 And I'll-- oh, yeah, y transpose, yeah. 293 00:19:06,310 --> 00:19:10,110 OK, so what do I put? 294 00:19:10,110 --> 00:19:12,407 Lambda y transpose. 295 00:19:12,407 --> 00:19:12,907 Thanks. 296 00:19:16,730 --> 00:19:18,970 And what do I put here? 297 00:19:18,970 --> 00:19:23,200 What does this translate to if this was for one eigenvector? 298 00:19:23,200 --> 00:19:25,060 For all of them at once, it's just 299 00:19:25,060 --> 00:19:28,747 going to translate to y transpose x equal the identity. 300 00:19:32,330 --> 00:19:33,540 This is pretty basic stuff. 301 00:19:33,540 --> 00:19:37,770 But stuff somehow we don't always necessarily see. 302 00:19:37,770 --> 00:19:38,840 Those are the key facts. 303 00:19:41,980 --> 00:19:46,900 And now, I plan to take the derivative, 304 00:19:46,900 --> 00:19:49,690 take the derivative of respect to lambda. 305 00:19:49,690 --> 00:19:53,500 Oh, I can derive one more fact. 306 00:19:53,500 --> 00:19:56,800 So this would be a formula. 307 00:19:56,800 --> 00:19:57,870 This is formula 1. 308 00:20:01,680 --> 00:20:06,210 Formula 1 just says, what do I get if I hit this on the left 309 00:20:06,210 --> 00:20:07,530 by y transpose? 310 00:20:07,530 --> 00:20:08,560 Can I do that? 311 00:20:08,560 --> 00:20:18,960 y transpose of t A of t x of t equals lambda of t. 312 00:20:18,960 --> 00:20:20,530 That's a number. 313 00:20:20,530 --> 00:20:30,090 So I can always bring that out in front of the inner product 314 00:20:30,090 --> 00:20:32,684 of vector notation. 315 00:20:32,684 --> 00:20:37,470 Are you good for that? 316 00:20:37,470 --> 00:20:43,780 I'm pleading like everything I've done is totally OK. 317 00:20:43,780 --> 00:20:48,660 And now, I have a improvement to make on this right-hand side, 318 00:20:48,660 --> 00:20:51,110 which is...? 319 00:20:51,110 --> 00:20:53,690 So what is y transpose times x? 320 00:20:53,690 --> 00:20:54,190 AUDIENCE: 1. 321 00:20:54,190 --> 00:20:55,440 GILBERT STRANG: It's 1. 322 00:20:55,440 --> 00:20:56,890 So let's remember that. 323 00:20:56,890 --> 00:20:57,390 It's 1. 324 00:21:02,070 --> 00:21:07,720 So in other words, I have got a formula for lambda of t. 325 00:21:07,720 --> 00:21:11,070 As time changes, the matrix changes. 326 00:21:11,070 --> 00:21:14,550 Its eigenvalues change according to this formula. 327 00:21:14,550 --> 00:21:18,130 Its eigenvectors change according of this formula. 328 00:21:18,130 --> 00:21:23,760 And its left eigenvectors change according to that formula. 329 00:21:23,760 --> 00:21:27,990 So everything here is above board. 330 00:21:27,990 --> 00:21:32,610 And now, what's the point? 331 00:21:32,610 --> 00:21:38,450 The point is I'm going to find this, the derivative. 332 00:21:38,450 --> 00:21:41,620 So I'm going to take the derivative of that equation 333 00:21:41,620 --> 00:21:43,330 and see what I get. 334 00:21:43,330 --> 00:21:47,260 That'll be the formula for the derivative of an eigenvalue. 335 00:21:47,260 --> 00:21:52,450 And amazingly, it's not that widely known. 336 00:21:52,450 --> 00:21:56,500 Of course, it's classical, but it's not always 337 00:21:56,500 --> 00:21:58,210 part of courses. 338 00:21:58,210 --> 00:22:02,160 So this is as time varies, the matrix varies, A. 339 00:22:02,160 --> 00:22:05,200 And therefore, its eigenvalues vary, 340 00:22:05,200 --> 00:22:07,360 and its eigenvectors vary. 341 00:22:07,360 --> 00:22:10,960 So we're going to find d lambda dt. 342 00:22:10,960 --> 00:22:13,210 It's one level of difficulty more 343 00:22:13,210 --> 00:22:18,670 to find dx dt, the derivative of the eigenvector 344 00:22:18,670 --> 00:22:21,300 or the second derivative of the eigenvalue. 345 00:22:21,300 --> 00:22:22,570 Those kind of come together. 346 00:22:22,570 --> 00:22:24,470 And I'm not going to go there. 347 00:22:24,470 --> 00:22:27,280 I'm just going to do the one great thing here-- 348 00:22:27,280 --> 00:22:30,020 take the derivative of that equation. 349 00:22:30,020 --> 00:22:31,080 Shall I do it over there? 350 00:22:33,680 --> 00:22:35,060 So here we go. 351 00:22:35,060 --> 00:22:37,480 So I want to compute d lambda dt. 352 00:22:40,290 --> 00:22:44,378 And I'm using this formula for lambda there. 353 00:22:50,720 --> 00:22:53,726 So I've got three things that depend on t. 354 00:22:53,726 --> 00:22:56,290 And I'm taking the derivative of their product. 355 00:22:56,290 --> 00:23:00,400 So I'm going to use the product rule. 356 00:23:00,400 --> 00:23:04,440 I'll apply the product rule to that derivative. 357 00:23:04,440 --> 00:23:13,230 Take the derivative of the first guy times A times x. 358 00:23:13,230 --> 00:23:34,330 Take the derivative of the second guy times the second guy 359 00:23:34,330 --> 00:23:36,010 and the third guy. 360 00:23:36,010 --> 00:23:42,990 y transpose of t A of t dx dt. 361 00:23:51,420 --> 00:23:51,920 OK? 362 00:23:51,920 --> 00:23:57,160 We are one minute away from a great formula. 363 00:23:57,160 --> 00:24:00,790 And I'm really happy if you allow me to say it. 364 00:24:00,790 --> 00:24:05,710 That that formula comes by just taking those facts 365 00:24:05,710 --> 00:24:09,670 we know, putting them together into this expression 366 00:24:09,670 --> 00:24:13,450 that we also know, and this is like lambda 367 00:24:13,450 --> 00:24:16,470 equals x inverse Ax. 368 00:24:16,470 --> 00:24:21,740 That's a diagonalizing thing and then taking the derivative. 369 00:24:21,740 --> 00:24:24,820 So what do I get if I take that derivative? 370 00:24:24,820 --> 00:24:31,677 Well, this term I'm going to keep. 371 00:24:31,677 --> 00:24:33,010 I'm not going to play with that. 372 00:24:40,790 --> 00:24:42,520 Everybody is clear? 373 00:24:42,520 --> 00:24:43,710 That's a number. 374 00:24:43,710 --> 00:24:44,770 Here's a matrix. 375 00:24:44,770 --> 00:24:48,350 dA dt is a matrix. 376 00:24:48,350 --> 00:24:52,250 I take the derivative of every entry in A. 377 00:24:52,250 --> 00:24:57,510 Here's its column vector, its eigenvector. 378 00:24:57,510 --> 00:24:58,770 And here's a row vector. 379 00:24:58,770 --> 00:25:03,150 So row times matrix times column is a number, 1 by 1. 380 00:25:03,150 --> 00:25:07,040 And actually, that's my answer. 381 00:25:07,040 --> 00:25:07,910 That's my answer. 382 00:25:07,910 --> 00:25:15,080 So I'm saying that these two terms cancel each other out 383 00:25:15,080 --> 00:25:17,000 as those two terms added to zero. 384 00:25:17,000 --> 00:25:23,060 This is the right answer for the derivative. 385 00:25:23,060 --> 00:25:25,640 That's a nice formula. 386 00:25:25,640 --> 00:25:31,250 So to find the derivative of an eigenvalue, 387 00:25:31,250 --> 00:25:36,500 the matrix is changing, you multiply by the eigenvector 388 00:25:36,500 --> 00:25:38,260 and by the left eigenvector. 389 00:25:38,260 --> 00:25:39,960 It gives you a number. 390 00:25:39,960 --> 00:25:44,220 And that's the d lambda dt. 391 00:25:44,220 --> 00:25:49,060 So why do those two guys add to 0? 392 00:25:49,060 --> 00:25:50,280 That's all that remains here. 393 00:25:50,280 --> 00:25:54,240 And then this topic is ended with this nice formula. 394 00:25:57,380 --> 00:26:00,840 So I want to simplify that, simplify that, and show 395 00:26:00,840 --> 00:26:03,150 that they cancel each other. 396 00:26:03,150 --> 00:26:05,950 So what is Ax? 397 00:26:05,950 --> 00:26:07,520 It's lambda x. 398 00:26:07,520 --> 00:26:13,910 So this guy is nothing but it's lambda 399 00:26:13,910 --> 00:26:23,430 that depends on time of course times dy dt dy transpose dt. 400 00:26:23,430 --> 00:26:24,685 I'm just copying that. 401 00:26:24,685 --> 00:26:27,520 Ax is lambda x. 402 00:26:27,520 --> 00:26:29,590 Sorry, I didn't mean to make that look hard. 403 00:26:32,810 --> 00:26:35,310 You OK with that? 404 00:26:35,310 --> 00:26:37,110 Ax is lambda x. 405 00:26:37,110 --> 00:26:41,010 And I am perfectly safe, because lambda is just a number 406 00:26:41,010 --> 00:26:42,670 to bring it out to the left. 407 00:26:42,670 --> 00:26:45,870 So it doesn't look like it's in the way. 408 00:26:45,870 --> 00:26:47,220 And what about this other term? 409 00:26:51,330 --> 00:26:54,150 So I have y transpose-- 410 00:26:54,150 --> 00:26:56,090 oh, y transpose A, what's that? 411 00:26:59,540 --> 00:27:00,980 What's y transpose A? 412 00:27:00,980 --> 00:27:03,770 That's the combination that I know. 413 00:27:03,770 --> 00:27:09,320 y transpose A, y is that left eigenvalue. y transpose 414 00:27:09,320 --> 00:27:11,150 A brings out a lambda. 415 00:27:11,150 --> 00:27:19,390 So this also brings out a lambda times y transpose times dx dt. 416 00:27:27,240 --> 00:27:28,020 OK? 417 00:27:28,020 --> 00:27:30,400 I just use Ax equal lambda x there. 418 00:27:30,400 --> 00:27:32,670 It was really nothing. 419 00:27:32,670 --> 00:27:33,990 Now, what do I do? 420 00:27:33,990 --> 00:27:35,460 I want this to be 0. 421 00:27:35,460 --> 00:27:37,170 Can you see it happening? 422 00:27:37,170 --> 00:27:40,600 It's a great pleasure to see it happening. 423 00:27:40,600 --> 00:27:42,060 So what do I have here? 424 00:27:42,060 --> 00:27:44,248 What's my first step now? 425 00:27:44,248 --> 00:27:45,490 AUDIENCE: Like take lambda-- 426 00:27:45,490 --> 00:27:48,170 GILBERT STRANG: Bring lambda outside. 427 00:27:48,170 --> 00:27:49,130 That's not 0. 428 00:27:49,130 --> 00:27:50,780 We don't know what that is. 429 00:27:50,780 --> 00:27:54,240 Bring lambda outside there times the whole thing. 430 00:27:57,650 --> 00:28:01,040 So for some wonderful reason I believe 431 00:28:01,040 --> 00:28:05,240 that this number, which is a row times a column, a row 432 00:28:05,240 --> 00:28:08,540 times a column, two terms there, I 433 00:28:08,540 --> 00:28:14,170 believe they knock each other out and that result is 0. 434 00:28:14,170 --> 00:28:16,024 And why? 435 00:28:16,024 --> 00:28:18,070 Why? 436 00:28:18,070 --> 00:28:20,980 Because I come back to-- 437 00:28:20,980 --> 00:28:23,710 this board has all that I know. 438 00:28:23,710 --> 00:28:27,490 And here's y transpose times x equal 1. 439 00:28:27,490 --> 00:28:30,330 And how does that help me? 440 00:28:30,330 --> 00:28:34,782 Because what I'm seeing in that square, in those brackets is? 441 00:28:34,782 --> 00:28:36,490 AUDIENCE: The derivative of y transpose-- 442 00:28:36,490 --> 00:28:39,850 GILBERT STRANG: The derivative of the y transpose x. 443 00:28:39,850 --> 00:28:41,750 So it's the derivative of? 444 00:28:41,750 --> 00:28:42,250 AUDIENCE: 1 445 00:28:42,250 --> 00:28:43,000 GILBERT STRANG: 1. 446 00:28:45,620 --> 00:28:47,170 Therefore, 0. 447 00:28:47,170 --> 00:28:50,430 So this is the derivative of 1. 448 00:28:50,430 --> 00:28:51,490 It equals 0. 449 00:28:51,490 --> 00:28:54,130 Those two terms knock each other out 450 00:28:54,130 --> 00:28:57,650 and leave just the nice term that we're seeing. 451 00:28:57,650 --> 00:28:59,380 So the derivative of the eigenvalue, 452 00:28:59,380 --> 00:29:04,690 just to have one more look at it before we leave it. 453 00:29:04,690 --> 00:29:08,120 The derivative of the eigenvalue is this formula. 454 00:29:08,120 --> 00:29:09,640 It's the rate at which the matrix 455 00:29:09,640 --> 00:29:15,120 is changing times the eigenvectors on right and left. 456 00:29:15,120 --> 00:29:17,170 Sometimes they're called the right eigenvector 457 00:29:17,170 --> 00:29:23,880 and the left eigenvector at the time t. 458 00:29:23,880 --> 00:29:28,260 So we're not saying in this d lambda dt. 459 00:29:28,260 --> 00:29:33,930 In other words, I get a nice formula, 460 00:29:33,930 --> 00:29:37,170 which doesn't involve the derivative of the eigenvector. 461 00:29:37,170 --> 00:29:39,120 That's the beauty of it. 462 00:29:39,120 --> 00:29:43,650 If I want to go up to take the next step-- 463 00:29:43,650 --> 00:29:47,580 I tried this weekend, but it's a mess. 464 00:29:47,580 --> 00:29:49,310 It would be to take the-- 465 00:29:49,310 --> 00:29:53,640 so this is my formula then, d lambda dt equals this. 466 00:29:53,640 --> 00:29:57,240 And I can take the next derivative of that, 467 00:29:57,240 --> 00:30:01,350 and it will involve d second dt squared. 468 00:30:01,350 --> 00:30:05,400 But it will also involve dx dt and dy dt. 469 00:30:05,400 --> 00:30:10,230 And in fact, a pseudo inverse even shows up. 470 00:30:10,230 --> 00:30:18,060 It's another step, and I'm not going that far, because we've 471 00:30:18,060 --> 00:30:20,570 got the best formula there. 472 00:30:23,890 --> 00:30:29,620 So now that has answered this question. 473 00:30:29,620 --> 00:30:33,440 And I could answer that question the same way. 474 00:30:33,440 --> 00:30:37,750 It would involve A transpose A and the singular vectors, 475 00:30:37,750 --> 00:30:42,190 instead of involving A and the eigenvectors. 476 00:30:42,190 --> 00:30:45,068 Maybe that's a suitable exercise. 477 00:30:45,068 --> 00:30:45,610 I don't know. 478 00:30:45,610 --> 00:30:47,380 I haven't done it myself. 479 00:30:47,380 --> 00:30:53,560 What I want to do is this, now say, 480 00:30:53,560 --> 00:30:58,590 what can we say about the change in the eigenvalue-- 481 00:30:58,590 --> 00:31:02,670 and I'll just stay first of all with eigenvalue-- 482 00:31:02,670 --> 00:31:08,040 when the change is like rank 1? 483 00:31:08,040 --> 00:31:11,830 This is a perfect example when the change is rank 1. 484 00:31:18,360 --> 00:31:26,350 So what can we say about the eigenvalues-- 485 00:31:26,350 --> 00:31:31,330 let's take the top, the largest eigenvalue, or all of them, 486 00:31:31,330 --> 00:31:34,390 all of them, lambda j, all of them-- 487 00:31:34,390 --> 00:31:44,490 of A plus a rank 1 matrix uv transpose. 488 00:31:44,490 --> 00:31:53,110 Oh, let's do the nice case here, the nice case, 489 00:31:53,110 --> 00:31:57,000 because if I allow a general matrix A, I have to worry about 490 00:31:57,000 --> 00:31:59,100 does it have enough eigenvectors? 491 00:31:59,100 --> 00:32:00,150 Can it diagonalize? 492 00:32:00,150 --> 00:32:02,220 All that stuff. 493 00:32:02,220 --> 00:32:04,815 Let's make it a symmetric matrix. 494 00:32:07,620 --> 00:32:12,500 And let's make the rank 1 change symmetric too. 495 00:32:12,500 --> 00:32:18,320 So the question is, what can I say about the eigenvalues 496 00:32:18,320 --> 00:32:20,670 after a rank 1 change? 497 00:32:20,670 --> 00:32:22,520 So again, this isn't calculus now, 498 00:32:22,520 --> 00:32:25,370 because the change that I'm making 499 00:32:25,370 --> 00:32:32,520 is a true vector and not a differential. 500 00:32:32,520 --> 00:32:35,340 And I'm not going to have an exact formula 501 00:32:35,340 --> 00:32:38,670 for the new eigenvalues, as I said. 502 00:32:38,670 --> 00:32:46,380 But what I am going to do is write down the beautiful facts 503 00:32:46,380 --> 00:32:49,100 that are known about that. 504 00:32:49,100 --> 00:32:50,760 And here they are. 505 00:32:50,760 --> 00:32:53,930 So, first of all, the eigenvalues 506 00:32:53,930 --> 00:32:58,130 are in descending order. 507 00:32:58,130 --> 00:33:01,040 We use descending order for singular values. 508 00:33:01,040 --> 00:33:03,170 Let's use them also for eigenvalues. 509 00:33:03,170 --> 00:33:07,400 So lambda 1 is greater or equal to lambda 2, 510 00:33:07,400 --> 00:33:10,230 greater or equal to lambda 3, and so on. 511 00:33:16,640 --> 00:33:20,870 Oh, give me-- give me an idea. 512 00:33:20,870 --> 00:33:26,690 What do you expect from that rank 1 change? 513 00:33:26,690 --> 00:33:29,870 So that change is rank 1. 514 00:33:29,870 --> 00:33:33,330 Can you tell me any more about that change, u u transpose? 515 00:33:33,330 --> 00:33:36,337 What kind of a matrix is u u transpose? 516 00:33:39,320 --> 00:33:43,300 It's rank 1, but we can say more. 517 00:33:43,300 --> 00:33:44,480 It is...? 518 00:33:44,480 --> 00:33:46,110 AUDIENCE: Symmetrical. 519 00:33:46,110 --> 00:33:48,290 GILBERT STRANG: Symmetric, of course. 520 00:33:48,290 --> 00:33:49,140 And it is...? 521 00:33:49,140 --> 00:33:49,640 Yeah? 522 00:33:49,640 --> 00:33:50,973 AUDIENCE: Positive semidefinite. 523 00:33:50,973 --> 00:33:53,100 GILBERT STRANG: Positive semidefinite. 524 00:33:53,100 --> 00:33:54,480 Positive semidefinite. 525 00:33:54,480 --> 00:33:56,465 This is a positive change. 526 00:33:56,465 --> 00:34:03,265 u u transpose is the typical rank 1 positive semidefinite. 527 00:34:03,265 --> 00:34:04,640 It couldn't be positive definite, 528 00:34:04,640 --> 00:34:06,950 because it's only got rank 1. 529 00:34:06,950 --> 00:34:10,460 What's the eigenvector of that matrix? 530 00:34:10,460 --> 00:34:13,000 Let's just-- why not here? 531 00:34:13,000 --> 00:34:16,199 We can do this in two seconds. 532 00:34:16,199 --> 00:34:18,830 So u u transpose, that's the matrix 533 00:34:18,830 --> 00:34:21,020 I'm asking you to think about. 534 00:34:21,020 --> 00:34:25,219 And it's a full n by n matrix, column times a row. 535 00:34:25,219 --> 00:34:27,690 Tell me an eigenvector of that matrix. 536 00:34:27,690 --> 00:34:28,190 Yes? 537 00:34:28,190 --> 00:34:28,820 AUDIENCE: u. 538 00:34:28,820 --> 00:34:29,570 GILBERT STRANG: u. 539 00:34:29,570 --> 00:34:35,190 If I multiply my matrix by u, I get-- what do I get? 540 00:34:35,190 --> 00:34:37,969 I get some number times u. 541 00:34:37,969 --> 00:34:39,770 And what is that number lambda? 542 00:34:39,770 --> 00:34:41,600 AUDIENCE: u transpose u. 543 00:34:41,600 --> 00:34:44,570 GILBERT STRANG: That lambda happens to be u transpose u. 544 00:34:47,330 --> 00:34:49,280 So that's different from u u transpose. 545 00:34:49,280 --> 00:34:50,760 This is a matrix. 546 00:34:50,760 --> 00:34:53,030 This is 18.065 now. 547 00:34:53,030 --> 00:34:54,650 That's a number. 548 00:34:54,650 --> 00:34:58,540 And what can you tell me about that number? 549 00:34:58,540 --> 00:34:59,290 It is...? 550 00:34:59,290 --> 00:35:01,060 AUDIENCE: Greater than or equal to 0. 551 00:35:01,060 --> 00:35:03,290 GILBERT STRANG: Greater-- well, even more. 552 00:35:03,290 --> 00:35:04,670 Greater than 0. 553 00:35:04,670 --> 00:35:08,090 Greater, because this is a true vector here. 554 00:35:08,090 --> 00:35:10,250 So this is greater than 0. 555 00:35:10,250 --> 00:35:12,920 It's the only eigenvalue-- all the other eigenvalues 556 00:35:12,920 --> 00:35:16,440 of that rank 1 matrix are zero. 557 00:35:16,440 --> 00:35:20,740 But the one non-zero eigenvalue is over on the plus side. 558 00:35:20,740 --> 00:35:22,580 It's u transpose u. 559 00:35:22,580 --> 00:35:26,510 We all recognize that as the length of u squared. 560 00:35:26,510 --> 00:35:29,540 It's certainly positive. 561 00:35:29,540 --> 00:35:32,690 So we do have a positive semidefinite definite matrix. 562 00:35:32,690 --> 00:35:39,760 What would your guess be of the effect on the eigenvalues of A? 563 00:35:39,760 --> 00:35:42,230 So I'm coming back to my real problem-- 564 00:35:42,230 --> 00:35:45,080 eigenvalues of S, sorry, S. Symmetric 565 00:35:45,080 --> 00:35:47,240 matrices, I'm saying symmetric. 566 00:35:47,240 --> 00:35:51,090 What is your guess if I have a symmetric matrix 567 00:35:51,090 --> 00:35:53,960 and I add on u u transpose? 568 00:35:53,960 --> 00:35:57,840 What do you imagine that does to the eigenvalues? 569 00:35:57,840 --> 00:35:59,340 You're going to get it right. 570 00:35:59,340 --> 00:36:01,120 Just say it. 571 00:36:01,120 --> 00:36:05,320 What happens to the eigenvalues of S if I add on u u transpose? 572 00:36:05,320 --> 00:36:06,100 They will...? 573 00:36:06,100 --> 00:36:07,380 AUDIENCE: More positive. 574 00:36:07,380 --> 00:36:09,330 GILBERT STRANG: They'll be more positive. 575 00:36:09,330 --> 00:36:10,300 They'll go up. 576 00:36:10,300 --> 00:36:11,970 This is a positive thing. 577 00:36:11,970 --> 00:36:15,320 It's like adding 17 to something. 578 00:36:15,320 --> 00:36:16,920 It moves up. 579 00:36:16,920 --> 00:36:20,010 So therefore, what I believe is-- 580 00:36:26,150 --> 00:36:30,590 so I've got two sets of eigenvalues now. 581 00:36:30,590 --> 00:36:33,170 One is the eigenvalues of s. 582 00:36:33,170 --> 00:36:36,650 The other is the different eigenvalues of S. 583 00:36:36,650 --> 00:36:40,190 So I can't call them both lambdas or I'm in trouble. 584 00:36:46,280 --> 00:36:50,540 So do you have a favorite other Greek letter 585 00:36:50,540 --> 00:36:52,310 for the eigenvalues of S? 586 00:36:52,310 --> 00:36:53,430 AUDIENCE: Gamma. 587 00:36:53,430 --> 00:36:56,000 GILBERT STRANG: Gamma. 588 00:36:56,000 --> 00:36:57,215 OK, gamma. 589 00:36:57,215 --> 00:36:58,590 As long as you say a Greek letter 590 00:36:58,590 --> 00:37:00,630 that I have some idea how to write. 591 00:37:00,630 --> 00:37:04,710 Zeta, it seems to me, is like the world's toughest letter 592 00:37:04,710 --> 00:37:05,220 to write. 593 00:37:05,220 --> 00:37:09,300 And electrical engineers can coolly flush off a zeta. 594 00:37:12,390 --> 00:37:13,440 I've never succeeded. 595 00:37:13,440 --> 00:37:16,260 So I'll write-- what did you say? 596 00:37:16,260 --> 00:37:17,070 AUDIENCE: Gamma. 597 00:37:17,070 --> 00:37:22,980 GILBERT STRANG: Gamma j of the original. 598 00:37:22,980 --> 00:37:26,150 So those are the eigenvalues of the original. 599 00:37:26,150 --> 00:37:29,580 These are the eigenvalues of the modified. 600 00:37:29,580 --> 00:37:33,130 And we're expecting the lambdas to be bigger than the gammas. 601 00:37:36,040 --> 00:37:38,200 So that's just a qualitative statement. 602 00:37:38,200 --> 00:37:39,410 And it's true. 603 00:37:39,410 --> 00:37:44,190 Each lambda is bigger than the gamma. 604 00:37:44,190 --> 00:37:48,600 Sorry, yeah, yeah, each lambda, by adding this stuff, 605 00:37:48,600 --> 00:37:51,390 the lambdas are bigger than-- so I'll just write that. 606 00:37:51,390 --> 00:37:53,040 Lambdas are bigger than gammas. 607 00:38:00,920 --> 00:38:07,840 And that's a fundamental fact, which we could prove. 608 00:38:07,840 --> 00:38:09,880 But a little more is known. 609 00:38:09,880 --> 00:38:13,450 Of course, the question is, how much bigger? 610 00:38:13,450 --> 00:38:17,770 How much can they be way bigger? 611 00:38:17,770 --> 00:38:19,540 Well, I don't believe they could be bigger 612 00:38:19,540 --> 00:38:23,680 by more than that number myself. 613 00:38:23,680 --> 00:38:28,250 But there's just better news than that. 614 00:38:28,250 --> 00:38:30,550 So the lambdas are bigger than the gammas. 615 00:38:30,550 --> 00:38:34,540 So lambda 1 is bigger than gamma 1. 616 00:38:34,540 --> 00:38:41,800 So this is the S plus u u transpose matrix. 617 00:38:41,800 --> 00:38:45,160 And these are the eigenvalues of the S matrix. 618 00:38:45,160 --> 00:38:47,260 Lambda 1 is bigger than gamma 1. 619 00:38:47,260 --> 00:38:52,450 But look what's happening in this line of text here. 620 00:38:52,450 --> 00:38:54,640 I'm saying that gamma 1-- 621 00:38:54,640 --> 00:38:58,150 that lambda 2 is smaller than gamma 1. 622 00:38:58,150 --> 00:38:59,410 Isn't that neat? 623 00:38:59,410 --> 00:39:01,960 The eigenvalues go up. 624 00:39:05,200 --> 00:39:07,752 But they don't just like go anywhere. 625 00:39:13,060 --> 00:39:15,135 And that's called interlacing. 626 00:39:22,570 --> 00:39:25,240 So this is one of those wonderful theorems that 627 00:39:25,240 --> 00:39:34,140 makes your heart happy, that if I do I rank one change 628 00:39:34,140 --> 00:39:38,580 and it's a positive change, then the eigenvalues increase, 629 00:39:38,580 --> 00:39:41,880 but they don't increase-- 630 00:39:41,880 --> 00:39:45,800 the new eigenvalue is below the new second eigenvalue. 631 00:39:45,800 --> 00:39:50,280 It doesn't pass up the old, first eigenvalues. 632 00:39:50,280 --> 00:39:54,150 And the new third eigenvalue doesn't pass up 633 00:39:54,150 --> 00:39:57,380 the old second eigenvalue. 634 00:39:57,380 --> 00:40:00,690 So that's the interlacing theorem 635 00:40:00,690 --> 00:40:06,490 that's associated with the names of famous math guys. 636 00:40:06,490 --> 00:40:10,900 And of course you have to say that's beautiful. 637 00:40:10,900 --> 00:40:14,350 While we're writing down such a theorem, 638 00:40:14,350 --> 00:40:16,930 make a guess of what the theorem would 639 00:40:16,930 --> 00:40:23,280 be if I do a rank 2 change. 640 00:40:23,280 --> 00:40:31,220 Suppose I do an S, staying symmetric. 641 00:40:31,220 --> 00:40:34,200 And I do a rank 1 change. 642 00:40:34,200 --> 00:40:37,770 But then I also do a rank 2 change, say w w transpose. 643 00:40:41,370 --> 00:40:43,830 So what's the deal here? 644 00:40:43,830 --> 00:40:50,370 What do I know about the change matrix, the delta S here? 645 00:40:50,370 --> 00:40:53,840 I know its rank is 2. 646 00:40:53,840 --> 00:40:57,230 I'm assuming u and w are not in the same direction. 647 00:40:57,230 --> 00:40:59,990 So that's a rank 2 matrix. 648 00:40:59,990 --> 00:41:03,120 And what can you tell me about the eigenvalues of that rank 2 649 00:41:03,120 --> 00:41:03,620 matrix? 650 00:41:06,480 --> 00:41:09,630 So it's got n eigenvalues because it's an n by n matrix. 651 00:41:09,630 --> 00:41:14,160 But how many non-zero eigenvalues has it got? 652 00:41:14,160 --> 00:41:17,926 Two, because its rank is 2. 653 00:41:17,926 --> 00:41:21,140 The rank tells you the number of non-zero eigenvalues 654 00:41:21,140 --> 00:41:23,540 when matrices are symmetric. 655 00:41:23,540 --> 00:41:24,990 It doesn't tell you enough. 656 00:41:24,990 --> 00:41:28,720 If matrices are unsymmetric, eigenvalues can be weird. 657 00:41:28,720 --> 00:41:32,710 So stay symmetric here. 658 00:41:32,710 --> 00:41:38,010 So this has two non-zero eigenvalues. 659 00:41:38,010 --> 00:41:41,110 And can you tell me their sign. 660 00:41:41,110 --> 00:41:44,140 Is that matrix positive semidefinite? 661 00:41:44,140 --> 00:41:45,700 Yes, of course, it is. 662 00:41:45,700 --> 00:41:46,600 Of course. 663 00:41:46,600 --> 00:41:48,350 So this was and this was. 664 00:41:48,350 --> 00:41:50,860 And together it certainly is. 665 00:41:50,860 --> 00:41:59,160 So now, I've added a rank 2 positive semidefinite matrix. 666 00:41:59,160 --> 00:42:01,890 And now, I'm not going to rewrite this line, 667 00:42:01,890 --> 00:42:06,530 but what would you expect to be true? 668 00:42:06,530 --> 00:42:09,140 You would expect that the eigenvalues increase. 669 00:42:11,730 --> 00:42:16,610 But how big could gamma-- 670 00:42:16,610 --> 00:42:20,360 yeah, so gamma 2, let's follow gamma 2. 671 00:42:22,950 --> 00:42:25,470 Well, maybe I should use another-- 672 00:42:25,470 --> 00:42:29,070 do the Greeks have any other letters than lambda and gamma. 673 00:42:29,070 --> 00:42:30,253 They must had-- 674 00:42:30,253 --> 00:42:30,960 AUDIENCE: Zeta. 675 00:42:30,960 --> 00:42:31,793 GILBERT STRANG: Who? 676 00:42:31,793 --> 00:42:33,420 C? 677 00:42:33,420 --> 00:42:34,760 Hell with that. 678 00:42:34,760 --> 00:42:37,150 Who knows one I can write? 679 00:42:37,150 --> 00:42:37,920 AUDIENCE: Alpha. 680 00:42:37,920 --> 00:42:38,837 GILBERT STRANG: Alpha. 681 00:42:38,837 --> 00:42:39,600 Good, alpha. 682 00:42:39,600 --> 00:42:40,620 Yes, alpha. 683 00:42:40,620 --> 00:42:41,220 Right. 684 00:42:41,220 --> 00:42:50,615 So alpha is the eigenvalues of this rank 2 change. 685 00:42:53,950 --> 00:42:55,420 OK. 686 00:42:55,420 --> 00:42:59,000 Now, what am I going to be able to say? 687 00:42:59,000 --> 00:43:01,400 Can I say anything about the-- 688 00:43:01,400 --> 00:43:08,640 well, of course, alpha 1 is bigger than lambda 1, 689 00:43:08,640 --> 00:43:10,860 which was bigger than-- 690 00:43:10,860 --> 00:43:13,140 eigenvalues are going up, right? 691 00:43:13,140 --> 00:43:16,270 I'm adding positive definite or positive semidefinite stuff. 692 00:43:16,270 --> 00:43:19,500 There's no way eigenvalues can start going down on me. 693 00:43:19,500 --> 00:43:24,000 So alpha 1 is a greater or equal to the lambda 694 00:43:24,000 --> 00:43:27,180 1, which had just a rank 1 change, which 695 00:43:27,180 --> 00:43:29,280 is greater or equal to the-- 696 00:43:32,010 --> 00:43:33,230 mu, was it mu? 697 00:43:33,230 --> 00:43:34,353 AUDIENCE: Gamma. 698 00:43:34,353 --> 00:43:35,270 GILBERT STRANG: Gamma. 699 00:43:35,270 --> 00:43:42,140 Gamma 1, and so on. 700 00:43:42,140 --> 00:43:50,370 OK, now, let's see, is gamma 1 bigger than alpha-- 701 00:43:50,370 --> 00:43:54,050 what am I struggling to write down here? 702 00:43:54,050 --> 00:43:57,160 What could I say? 703 00:43:57,160 --> 00:44:00,170 Well, what can I say that reflects 704 00:44:00,170 --> 00:44:07,440 the fact that this lambda 2-- 705 00:44:07,440 --> 00:44:12,200 or sorry, so gamma 1 went up. 706 00:44:12,200 --> 00:44:14,150 Gamma 1 was bigger than lambda 2. 707 00:44:14,150 --> 00:44:15,890 That was the point here. 708 00:44:15,890 --> 00:44:17,050 Gamma 1 is bigger. 709 00:44:17,050 --> 00:44:20,540 So this was a sort of easy, because I'm adding stuff. 710 00:44:20,540 --> 00:44:22,830 I expected the lambda to go up. 711 00:44:22,830 --> 00:44:26,660 This is where the theorem is that it didn't go up 712 00:44:26,660 --> 00:44:28,830 so far as to pass-- 713 00:44:28,830 --> 00:44:32,240 or sorry, the lambda 2, which went up, 714 00:44:32,240 --> 00:44:35,180 didn't pass up gamma 1. 715 00:44:35,180 --> 00:44:37,580 Lambda 2 didn't pass up gamma 1. 716 00:44:37,580 --> 00:44:40,850 And now let me write those words down. 717 00:44:40,850 --> 00:44:45,170 Now the alpha 2-- 718 00:44:45,170 --> 00:44:48,260 well, could alpha 2 pass up lambda 1? 719 00:44:51,690 --> 00:44:53,000 And what about alpha 3? 720 00:44:57,800 --> 00:45:00,680 Let me say what I believe. 721 00:45:00,680 --> 00:45:08,020 I think alpha 2, which is like 1 behind, but I'm adding rank 2, 722 00:45:08,020 --> 00:45:12,270 I think alpha 2 could pass up lambda 1. 723 00:45:12,270 --> 00:45:13,440 It could pass lambda 1. 724 00:45:13,440 --> 00:45:15,270 But alpha 3 can't. 725 00:45:15,270 --> 00:45:21,260 I believe that alpha 3 is smaller than lambda 1-- 726 00:45:21,260 --> 00:45:26,300 smaller than gamma 1, the original. 727 00:45:26,300 --> 00:45:26,960 Got it. 728 00:45:26,960 --> 00:45:28,900 Yeah, yeah, yeah. 729 00:45:35,020 --> 00:45:38,680 Anyway, I'll get it right in the notes. 730 00:45:38,680 --> 00:45:41,290 You know what question I'm asking. 731 00:45:41,290 --> 00:45:44,860 And for me, that's the important thing. 732 00:45:44,860 --> 00:45:50,560 Now, there is a little matter of why is this true? 733 00:45:50,560 --> 00:45:52,460 This is the good case. 734 00:45:52,460 --> 00:45:54,700 Let me give you another example of interlacing. 735 00:45:54,700 --> 00:45:55,810 Can I do that? 736 00:45:55,810 --> 00:45:59,830 It really comes from this, but let 737 00:45:59,830 --> 00:46:02,220 me give you another example that's just striking. 738 00:46:05,560 --> 00:46:10,390 So I have a symmetric matrix, n by n. 739 00:46:10,390 --> 00:46:18,260 Call it S. And then I throw away the last row and column. 740 00:46:18,260 --> 00:46:20,875 So in here is S n minus 1. 741 00:46:20,875 --> 00:46:24,250 The big matrix was Sn. 742 00:46:24,250 --> 00:46:26,516 This one is of size n minus 1. 743 00:46:29,230 --> 00:46:31,900 So it's got sort of less degrees of freedom, 744 00:46:31,900 --> 00:46:36,370 because the last degree of freedom got removed. 745 00:46:36,370 --> 00:46:39,040 And what do you think about the eigenvalues 746 00:46:39,040 --> 00:46:42,670 of the n minus 1 eigenvalues of this 747 00:46:42,670 --> 00:46:45,670 and the n eigenvalues of that? 748 00:46:45,670 --> 00:46:46,740 They interlace. 749 00:46:49,830 --> 00:46:53,370 So this has eigenvalue lambda 1. 750 00:46:53,370 --> 00:46:57,990 This would have an eigenvalue smaller than that. 751 00:46:57,990 --> 00:47:01,710 This would have an eigenvalue lambda 2. 752 00:47:01,710 --> 00:47:06,900 This would have an eigenvalue smaller than that and so on. 753 00:47:06,900 --> 00:47:09,120 Just the same interlacing and basically 754 00:47:09,120 --> 00:47:11,850 for the same reason, that when you-- 755 00:47:15,110 --> 00:47:19,440 this reduction to size n minus 1 is 756 00:47:19,440 --> 00:47:23,070 like I'm saying xn has to be 0 in the energy 757 00:47:23,070 --> 00:47:25,380 or any of those expression. 758 00:47:25,380 --> 00:47:32,370 And the fact of making xn be 0 is like one constraint, taking 759 00:47:32,370 --> 00:47:34,380 one degree of freedom away. 760 00:47:34,380 --> 00:47:38,220 It reduces the eigenvalues, but not by two. 761 00:47:38,220 --> 00:47:44,180 OK, now I have one final mystery. 762 00:47:44,180 --> 00:47:47,650 And let me try to tell you what. 763 00:47:47,650 --> 00:47:48,750 It worried me. 764 00:47:51,420 --> 00:47:55,070 Now what is it that worried me? 765 00:47:55,070 --> 00:48:06,350 Yes, suppose this change, this u, this change that I'm making, 766 00:48:06,350 --> 00:48:11,930 suppose it's actually the second eigenvector of S. 767 00:48:11,930 --> 00:48:14,325 So can I write this down? 768 00:48:17,970 --> 00:48:26,180 Suppose u is actually the second eigenvector of S. 769 00:48:26,180 --> 00:48:29,370 What do I mean by that? 770 00:48:29,370 --> 00:48:33,350 So I mean that S times u is lambda 2 times u. 771 00:48:37,670 --> 00:48:40,530 Now, I'm going to change it. 772 00:48:40,530 --> 00:48:45,860 S plus u u transpose, that's what I've been looking at. 773 00:48:45,860 --> 00:48:48,580 And that moves the eigenvalues up. 774 00:48:48,580 --> 00:48:51,010 But what worries me is like if I multiply this 775 00:48:51,010 --> 00:48:57,550 by 20, some big number, I'm going 776 00:48:57,550 --> 00:49:03,340 to move that eigenvalue way up, way past. 777 00:49:03,340 --> 00:49:08,900 I got worried about this inequality. 778 00:49:08,900 --> 00:49:17,090 When I add this, that same u is lambda 2 plus 20 u. 779 00:49:19,720 --> 00:49:22,120 Su is lambda 2u. 780 00:49:22,120 --> 00:49:26,350 And this 20 is 20. 781 00:49:26,350 --> 00:49:28,490 u is a unit vector. 782 00:49:28,490 --> 00:49:30,380 So you see my worry? 783 00:49:30,380 --> 00:49:33,460 Here, I'm doing a rank 1 change. 784 00:49:33,460 --> 00:49:39,080 But it's moved an eigenvalue way, way up. 785 00:49:39,080 --> 00:49:45,650 So how could this statement be true? 786 00:49:45,650 --> 00:49:50,330 So I've just figured out here what gamma-- 787 00:49:50,330 --> 00:49:54,170 well, do you see my question? 788 00:49:54,170 --> 00:49:57,680 I could leave it as a question to answer next time. 789 00:49:57,680 --> 00:50:00,170 Let me do that. 790 00:50:00,170 --> 00:50:04,620 And I'll put it online so you'll see it clearly. 791 00:50:04,620 --> 00:50:11,290 It looks like and it happens this eigenvector now has 792 00:50:11,290 --> 00:50:13,710 eigenvalue lambda 2 plus 20. 793 00:50:13,710 --> 00:50:19,170 Why doesn't that blow away this statement? 794 00:50:19,170 --> 00:50:24,360 I'll put that, because it's sort of coming with minus 10 seconds 795 00:50:24,360 --> 00:50:29,280 to go in the class, so let's leave that and a discussion 796 00:50:29,280 --> 00:50:32,100 of this for next time. 797 00:50:32,100 --> 00:50:35,320 But I'm happy with this lecture if you are. 798 00:50:35,320 --> 00:50:38,670 Last lecture I got u's and v's mixed up. 799 00:50:38,670 --> 00:50:40,500 And it's not reliable. 800 00:50:40,500 --> 00:50:44,940 Here, I like the proof of the lambda dt 801 00:50:44,940 --> 00:50:47,800 and we're started on this topic 2. 802 00:50:47,800 --> 00:50:48,300 Good. 803 00:50:48,300 --> 00:50:50,150 Thank you.