1 00:00:01,550 --> 00:00:03,920 The following content is provided under a Creative 2 00:00:03,920 --> 00:00:05,310 Commons license. 3 00:00:05,310 --> 00:00:07,520 Your support will help MIT OpenCourseWare 4 00:00:07,520 --> 00:00:11,610 continue to offer high quality educational resources for free. 5 00:00:11,610 --> 00:00:14,180 To make a donation or to view additional materials 6 00:00:14,180 --> 00:00:18,140 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:18,140 --> 00:00:19,026 at ocw.mit.edu. 8 00:00:22,758 --> 00:00:23,550 GILBERT STRANG: OK. 9 00:00:23,550 --> 00:00:25,770 Why don't I start? 10 00:00:25,770 --> 00:00:32,130 So I was hoping that we would have the next online assignment 11 00:00:32,130 --> 00:00:37,110 ready, but Julia 0.6, the new version, is slowing us down, 12 00:00:37,110 --> 00:00:42,140 and it'll probably be next time. 13 00:00:42,140 --> 00:00:47,110 But my lectures, of course, are-- 14 00:00:47,110 --> 00:00:49,030 well, what I want to say, perhaps, 15 00:00:49,030 --> 00:00:52,690 is this isn't intended to be a course 16 00:00:52,690 --> 00:00:55,750 in numerical linear algebra, but I 17 00:00:55,750 --> 00:01:00,550 thought I couldn't let the whole semester go by without saying 18 00:01:00,550 --> 00:01:04,780 something about how to compute eigenvalues and singular 19 00:01:04,780 --> 00:01:07,420 values. 20 00:01:07,420 --> 00:01:14,350 Of course, you're going to call eig or SVD or the equivalent 21 00:01:14,350 --> 00:01:17,560 in Python or Julia. 22 00:01:17,560 --> 00:01:23,020 But actually, the QR factorization 23 00:01:23,020 --> 00:01:27,640 that we spoke about, that we spoke entirely about last time, 24 00:01:27,640 --> 00:01:29,470 is the key-- 25 00:01:29,470 --> 00:01:31,870 unexpectedly, unexpectedly. 26 00:01:31,870 --> 00:01:38,450 You have a matrix A whose eigenvalues you want. 27 00:01:38,450 --> 00:01:41,590 So let's start with eigenvalues. 28 00:01:41,590 --> 00:01:44,580 And it might be a symmetric matrix. 29 00:01:44,580 --> 00:01:49,060 I'll stay with A rather than S because it doesn't have to be, 30 00:01:49,060 --> 00:01:51,040 but you get special-- 31 00:01:51,040 --> 00:01:55,630 always you get something special if the matrix is symmetric. 32 00:01:58,240 --> 00:02:03,130 So this method of computing eigenvalues, to me at least 33 00:02:03,130 --> 00:02:05,950 and I think to many people, came out 34 00:02:05,950 --> 00:02:12,640 of the blue a while ago, but not that long ago. 35 00:02:12,640 --> 00:02:14,840 And it worked very well. 36 00:02:14,840 --> 00:02:16,420 So here's the idea. 37 00:02:16,420 --> 00:02:21,730 It's called the QR method because you start by factoring 38 00:02:21,730 --> 00:02:23,530 your matrix into QR. 39 00:02:23,530 --> 00:02:29,000 So here's A. Can we call it A0? 40 00:02:29,000 --> 00:02:33,400 That's the matrix we start with whose eigenvalues we want. 41 00:02:33,400 --> 00:02:37,354 And I'll call these Q0 and R0. 42 00:02:37,354 --> 00:02:40,940 And you remember what that means, 43 00:02:40,940 --> 00:02:44,620 what's hiding behind those letters that I've written? 44 00:02:44,620 --> 00:02:49,930 You have the columns of A, possibly symmetric as I said, 45 00:02:49,930 --> 00:02:54,100 but not orthogonal usually. 46 00:02:54,100 --> 00:02:59,380 So you find an orthogonal basis. 47 00:02:59,380 --> 00:03:01,840 You orthogonalize-- you line them up 48 00:03:01,840 --> 00:03:03,700 perpendicular to each other. 49 00:03:03,700 --> 00:03:07,390 And then there is a matrix R, which 50 00:03:07,390 --> 00:03:11,800 happens to be upper triangular, that 51 00:03:11,800 --> 00:03:15,400 connects the not-orthogonal basis 52 00:03:15,400 --> 00:03:18,070 with the orthogonality base. 53 00:03:18,070 --> 00:03:18,810 Right. 54 00:03:18,810 --> 00:03:26,310 We constructed R step-by-step. 55 00:03:26,310 --> 00:03:34,390 And then, the idea is write these in the reverse order, 56 00:03:34,390 --> 00:03:42,300 and that will be A1, the next A. And then do it again and again 57 00:03:42,300 --> 00:03:44,690 and again. 58 00:03:44,690 --> 00:03:50,670 And so we're hoping, of course, that the eigenvalues didn't 59 00:03:50,670 --> 00:03:51,730 change. 60 00:03:51,730 --> 00:03:56,190 We're hoping that we can forget A0, start again with A1, 61 00:03:56,190 --> 00:03:59,730 and produce A2 and continue. 62 00:03:59,730 --> 00:04:00,960 So we're hoping two things. 63 00:04:00,960 --> 00:04:02,070 One will not be a hope. 64 00:04:02,070 --> 00:04:08,580 We can see that the eigenvalues of A1 and A0 are the same. 65 00:04:08,580 --> 00:04:11,190 So we have not changed the eigenvalues. 66 00:04:11,190 --> 00:04:12,840 How do we see that? 67 00:04:12,840 --> 00:04:15,900 If you had to show that two matrices had 68 00:04:15,900 --> 00:04:20,490 the same eigenvalues, would you compute all the eigenvalues 69 00:04:20,490 --> 00:04:21,810 and compare them? 70 00:04:21,810 --> 00:04:23,190 Certainly not. 71 00:04:23,190 --> 00:04:23,940 What would you do? 72 00:04:23,940 --> 00:04:29,190 What's the best test, the usual test, the usual thing 73 00:04:29,190 --> 00:04:31,860 you would want to show to-- 74 00:04:31,860 --> 00:04:33,090 AUDIENCE: They're similar. 75 00:04:33,090 --> 00:04:35,007 GILBERT STRANG: They're similar, that's right. 76 00:04:35,007 --> 00:04:39,060 So the claim would be that these two matrices are similar. 77 00:04:39,060 --> 00:04:40,570 Right. 78 00:04:40,570 --> 00:04:42,270 So maybe we should show that. 79 00:04:42,270 --> 00:04:45,330 So we want to write A1 in a different way. 80 00:04:45,330 --> 00:04:51,820 So the claim is A1 is similar to A. 81 00:04:51,820 --> 00:04:55,180 So we just have to figure out-- 82 00:04:55,180 --> 00:04:58,720 we have to get A1 and turn it to A0. 83 00:04:58,720 --> 00:04:59,620 So here is A1. 84 00:05:03,250 --> 00:05:05,320 So A1. 85 00:05:05,320 --> 00:05:07,300 I want to show that that's right. 86 00:05:07,300 --> 00:05:11,830 So that's R0, Q0. 87 00:05:11,830 --> 00:05:14,570 But what is Q0? 88 00:05:14,570 --> 00:05:17,090 From here, Q0 is what we-- 89 00:05:20,000 --> 00:05:22,460 Q0 is-- what do I want? 90 00:05:22,460 --> 00:05:24,960 I want to put R0 inverse over there. 91 00:05:24,960 --> 00:05:27,800 So R0. 92 00:05:27,800 --> 00:05:31,938 Now for Q0, I'm going to substitute. 93 00:05:31,938 --> 00:05:35,050 So what is Q0? 94 00:05:35,050 --> 00:05:39,400 I multiply both sides of that equation by R0 inverse. 95 00:05:39,400 --> 00:05:51,550 So Q0 is A0 is-- sorry, I said A and wrote Q. A0, R0 inverse. 96 00:05:55,320 --> 00:05:59,570 For Q0, I've put in what it equals. 97 00:05:59,570 --> 00:06:02,290 And that's done it. 98 00:06:02,290 --> 00:06:09,670 That's exactly telling me that A1 which is this, equals this, 99 00:06:09,670 --> 00:06:12,410 and that's a similarity transformation. 100 00:06:12,410 --> 00:06:14,860 I have not changed the eigenvalues. 101 00:06:14,860 --> 00:06:17,010 So that's OK. 102 00:06:17,010 --> 00:06:20,460 The other thing, which is the, you could say, 103 00:06:20,460 --> 00:06:24,360 the miracle in this thing, is that when 104 00:06:24,360 --> 00:06:29,330 I continue to do that for almost every matrix, 105 00:06:29,330 --> 00:06:35,660 the matrices begin to lose stuff off the diagonal, 106 00:06:35,660 --> 00:06:41,180 especially below the diagonal by the ordering that QR has done. 107 00:06:41,180 --> 00:06:43,440 So it would tend to-- 108 00:06:43,440 --> 00:06:45,010 you start with a matrix A0. 109 00:06:49,230 --> 00:06:54,980 You got a matrix A1, which is a little smaller here. 110 00:06:59,080 --> 00:07:02,760 This part being especially smaller. 111 00:07:02,760 --> 00:07:03,690 You do it again. 112 00:07:03,690 --> 00:07:06,000 This is even smaller. 113 00:07:06,000 --> 00:07:07,240 Even smaller. 114 00:07:07,240 --> 00:07:08,460 A2. 115 00:07:08,460 --> 00:07:10,080 Even smaller. 116 00:07:10,080 --> 00:07:11,820 And you keep going. 117 00:07:11,820 --> 00:07:18,730 And for most matrices, the result is that-- 118 00:07:18,730 --> 00:07:22,420 I don't know how many steps we want to think of taking, 119 00:07:22,420 --> 00:07:28,500 but we get quite small numbers here, especially-- 120 00:07:28,500 --> 00:07:29,000 yeah. 121 00:07:29,000 --> 00:07:32,940 So once we get small numbers here, little epsilons-- 122 00:07:32,940 --> 00:07:37,610 that's everybody's shorthand for small numbers-- 123 00:07:37,610 --> 00:07:41,330 what would you expect to see on the diagonal? 124 00:07:41,330 --> 00:07:42,460 AUDIENCE: The eigenvalues. 125 00:07:42,460 --> 00:07:45,550 GILBERT STRANG: The eigenvalues, because this 126 00:07:45,550 --> 00:07:47,740 has the same eigenvalues as this, 127 00:07:47,740 --> 00:07:50,530 as this, as this, and these little epsilons 128 00:07:50,530 --> 00:07:53,590 are not going to change the eigenvalues too much. 129 00:07:53,590 --> 00:07:56,440 So these will be, on the diagonal, 130 00:07:56,440 --> 00:08:02,020 will be close to the eigenvalues. 131 00:08:02,020 --> 00:08:07,270 And actually, what happens is this one comes first. 132 00:08:07,270 --> 00:08:10,600 That one is quite accurate first. 133 00:08:10,600 --> 00:08:16,640 I guess we should probably do a simple example 134 00:08:16,640 --> 00:08:19,870 to see this happen. 135 00:08:19,870 --> 00:08:22,980 Actually, I do an example in the notes. 136 00:08:22,980 --> 00:08:25,570 And let me say what happens. 137 00:08:25,570 --> 00:08:29,860 So I do a 2 by 2 example which has something 138 00:08:29,860 --> 00:08:33,624 like cos theta, sine theta. 139 00:08:37,728 --> 00:08:38,520 I don't know what-- 140 00:08:38,520 --> 00:08:41,940 I've forgotten what I took, and I don't have that page of notes 141 00:08:41,940 --> 00:08:42,440 here. 142 00:08:42,440 --> 00:08:51,370 Something here, something here, something here as A0. 143 00:08:51,370 --> 00:08:55,810 And, then A1, after just one step, 144 00:08:55,810 --> 00:09:04,410 has sine cubed theta there and numbers there 145 00:09:04,410 --> 00:09:08,160 that are getting much closer to the eigenvalues. 146 00:09:08,160 --> 00:09:11,820 Sorry that this isn't a full-scale example, 147 00:09:11,820 --> 00:09:14,460 but the point of the example is here, 148 00:09:14,460 --> 00:09:22,880 that this off-diagonal entry gets cubed. 149 00:09:22,880 --> 00:09:27,770 And the next step will be its 9th power, and then 150 00:09:27,770 --> 00:09:29,330 the 27th power. 151 00:09:29,330 --> 00:09:32,540 So it's really quickly going to 0. 152 00:09:32,540 --> 00:09:39,800 And this happens-- so cubic convergence is-- 153 00:09:39,800 --> 00:09:45,410 that's a price in numerical linear algebra. 154 00:09:45,410 --> 00:09:48,880 So that happens. 155 00:09:48,880 --> 00:09:53,560 So this arrived on the scene and quickly 156 00:09:53,560 --> 00:09:57,400 blew away all other methods that were being 157 00:09:57,400 --> 00:10:00,560 used to compute eigenvalues. 158 00:10:00,560 --> 00:10:04,610 But numerical people, being what they are, 159 00:10:04,610 --> 00:10:06,020 they wanted to improve it-- 160 00:10:06,020 --> 00:10:08,780 like, is there a way to make it faster? 161 00:10:08,780 --> 00:10:12,050 And it turned out there is. 162 00:10:12,050 --> 00:10:14,585 It turned out that the better way-- 163 00:10:17,120 --> 00:10:21,440 basically, the same idea, but just you're 164 00:10:21,440 --> 00:10:27,260 always looking for a simple change. 165 00:10:31,060 --> 00:10:34,680 And the idea of introducing a shift 166 00:10:34,680 --> 00:10:42,140 was tried and turned out to work extremely well. 167 00:10:42,140 --> 00:10:44,960 So that's the improvement. 168 00:10:44,960 --> 00:10:47,230 So we take A1-- 169 00:10:47,230 --> 00:10:47,730 no. 170 00:10:47,730 --> 00:10:49,500 So how does it-- 171 00:10:49,500 --> 00:10:51,910 how does the shift work? 172 00:10:51,910 --> 00:10:54,890 So instead of A1-- 173 00:10:54,890 --> 00:10:57,480 have I got space here to do this? 174 00:10:57,480 --> 00:11:02,430 Instead of A0, I take A1 minus a shift. 175 00:11:02,430 --> 00:11:06,360 So a shift is some multiple of the identity. 176 00:11:06,360 --> 00:11:10,040 If I just move a matrix by a multiple of the identity, 177 00:11:10,040 --> 00:11:12,050 what happens to its eigenvectors? 178 00:11:12,050 --> 00:11:14,720 What happens to its eigenvalues? 179 00:11:14,720 --> 00:11:16,970 Something pretty simple, right, because I'm just 180 00:11:16,970 --> 00:11:19,610 shifting by sI. 181 00:11:19,610 --> 00:11:21,282 What happens to its eigenvectors? 182 00:11:21,282 --> 00:11:22,490 AUDIENCE: They stay the same. 183 00:11:22,490 --> 00:11:24,820 GILBERT STRANG: They're the same. 184 00:11:24,820 --> 00:11:26,605 And what happens to the eigenvalues? 185 00:11:26,605 --> 00:11:27,730 AUDIENCE: They change by s. 186 00:11:27,730 --> 00:11:30,460 GILBERT STRANG: They change by s. 187 00:11:30,460 --> 00:11:37,150 If A0 V equaled lambda v, then when I multiply this by V, 188 00:11:37,150 --> 00:11:39,010 there will be an extra-- 189 00:11:39,010 --> 00:11:44,910 it'll be lambda minus s because the identity times the V 190 00:11:44,910 --> 00:11:50,310 is just V. So it just shifts all the eigenvalues by s. 191 00:11:50,310 --> 00:11:52,010 And you try to shift-- 192 00:11:52,010 --> 00:11:54,570 you look for a shift. 193 00:11:54,570 --> 00:11:55,620 This would be great. 194 00:11:55,620 --> 00:11:59,460 If you knew lambda N, a shift there, 195 00:11:59,460 --> 00:12:03,450 you would be looking for 0 then if you shifted them 196 00:12:03,450 --> 00:12:05,250 all by that lambda. 197 00:12:05,250 --> 00:12:07,650 And it turns out that would speed things up. 198 00:12:07,650 --> 00:12:12,780 So it will work instead with this matrix as now, 199 00:12:12,780 --> 00:12:17,190 again, I'm factoring it in a Q0 R0. 200 00:12:17,190 --> 00:12:23,250 So that's the work of the method is in doing Gram-Schmidt 201 00:12:23,250 --> 00:12:25,020 at every step. 202 00:12:25,020 --> 00:12:28,740 And then there's a little work in reversing the order. 203 00:12:33,150 --> 00:12:37,010 And then I want to undo the shift, 204 00:12:37,010 --> 00:12:39,470 so I do that factorization. 205 00:12:39,470 --> 00:12:40,410 Now, let's see. 206 00:12:40,410 --> 00:12:45,240 You may have to help me to remember what I should do here. 207 00:12:45,240 --> 00:12:47,880 So I factor those. 208 00:12:47,880 --> 00:12:50,250 I reverse those. 209 00:12:50,250 --> 00:12:57,460 And then I think I add back the shift, and that's my A1. 210 00:13:00,700 --> 00:13:01,800 So I took A0. 211 00:13:01,800 --> 00:13:03,450 I shifted it. 212 00:13:03,450 --> 00:13:04,590 I worked with it. 213 00:13:04,590 --> 00:13:05,610 QR. 214 00:13:05,610 --> 00:13:06,750 Reversed the order. 215 00:13:06,750 --> 00:13:07,800 RQ. 216 00:13:07,800 --> 00:13:11,150 Added back the shift to get a matrix. 217 00:13:11,150 --> 00:13:16,980 And what am I, of course, hoping about the matrix A1 and A0? 218 00:13:16,980 --> 00:13:19,960 I'm hoping they're still similar. 219 00:13:19,960 --> 00:13:23,030 So I did a shift and I undid a shift. 220 00:13:23,030 --> 00:13:28,780 But of course, after doing our QR, I have to check, 221 00:13:28,780 --> 00:13:31,060 are these really still similar? 222 00:13:31,060 --> 00:13:34,880 So let me just try to check that one again. 223 00:13:34,880 --> 00:13:39,940 Maybe I'll just-- it's sitting right there, 224 00:13:39,940 --> 00:13:42,550 so let me do it again. 225 00:13:42,550 --> 00:13:49,400 I'm hoping something-- where did we-- 226 00:13:49,400 --> 00:13:50,570 oh yeah, here. 227 00:13:50,570 --> 00:13:54,730 We show that A1 was similar to A0, 228 00:13:54,730 --> 00:13:58,860 and I'm hoping that's probably still true even-- 229 00:13:58,860 --> 00:14:02,380 the shift didn't mess that up. 230 00:14:02,380 --> 00:14:03,620 Let's just try. 231 00:14:03,620 --> 00:14:08,825 So A0-- that's R0, Q0 plus sI. 232 00:14:12,120 --> 00:14:15,240 And now what am I going to do? 233 00:14:15,240 --> 00:14:19,170 I'm going to-- what did I do before? 234 00:14:19,170 --> 00:14:24,060 I figured out what Q0 was from this. 235 00:14:26,770 --> 00:14:27,570 You remember? 236 00:14:27,570 --> 00:14:28,850 So this R0. 237 00:14:31,670 --> 00:14:33,320 Now I have to put in Q0. 238 00:14:33,320 --> 00:14:36,860 But Q0 is this thing inverse times this. 239 00:14:36,860 --> 00:14:38,750 Is this going to work? 240 00:14:38,750 --> 00:14:42,920 I'm hoping, but I don't think I wanted to get that. 241 00:14:47,190 --> 00:14:50,610 No, it's not Q0 there. 242 00:14:50,610 --> 00:14:52,470 What do I put it here? 243 00:14:55,070 --> 00:15:01,666 And if it doesn't work, we'll leave it as an exercise. 244 00:15:01,666 --> 00:15:02,870 AUDIENCE: [INAUDIBLE] 245 00:15:03,370 --> 00:15:08,125 GILBERT STRANG: Yeah, because I didn't start right somehow. 246 00:15:11,320 --> 00:15:14,530 But let me just push along to see what happens. 247 00:15:14,530 --> 00:15:17,620 So I'm plugging in for Q0 here. 248 00:15:17,620 --> 00:15:20,970 I'm plugging in this matrix, so it's shouldn't have 249 00:15:20,970 --> 00:15:25,600 inverted it, times R0 inverse. 250 00:15:25,600 --> 00:15:26,170 Who knows? 251 00:15:26,170 --> 00:15:28,390 It might work. 252 00:15:28,390 --> 00:15:29,840 Who knows? 253 00:15:29,840 --> 00:15:31,990 So that's the R0, Q0. 254 00:15:31,990 --> 00:15:33,160 Right? 255 00:15:33,160 --> 00:15:34,120 Is everybody with me? 256 00:15:34,120 --> 00:15:37,090 Sorry about the-- 257 00:15:37,090 --> 00:15:39,820 R0 inverse. 258 00:15:39,820 --> 00:15:41,275 And then I have to add sI. 259 00:15:45,860 --> 00:15:47,210 So what have I done? 260 00:15:47,210 --> 00:15:51,840 I've just pushed on, believing that this 261 00:15:51,840 --> 00:15:59,340 would work because it's the method that is constantly used. 262 00:15:59,340 --> 00:16:01,555 Now, do I have-- 263 00:16:01,555 --> 00:16:02,430 what do I have there? 264 00:16:05,020 --> 00:16:07,760 Is it working? 265 00:16:07,760 --> 00:16:11,700 This is R0 A. That was, of course, A0. 266 00:16:11,700 --> 00:16:14,390 R0, A0, R0 inverse. 267 00:16:14,390 --> 00:16:19,040 R0, A0, R0 inverse. 268 00:16:19,040 --> 00:16:21,750 Good. 269 00:16:21,750 --> 00:16:23,270 Minus s. 270 00:16:23,270 --> 00:16:29,940 What have I got there from the R0 minus sI R0? 271 00:16:29,940 --> 00:16:31,332 What is that? 272 00:16:31,332 --> 00:16:32,337 AUDIENCE: Minus sI. 273 00:16:32,337 --> 00:16:33,670 GILBERT STRANG: That's minus sI. 274 00:16:33,670 --> 00:16:35,450 Ha, look. 275 00:16:35,450 --> 00:16:36,540 Success. 276 00:16:36,540 --> 00:16:38,850 The R0 cancels the R0 inverse. 277 00:16:38,850 --> 00:16:44,580 So that term from that, that, that is minus sI 278 00:16:44,580 --> 00:16:45,770 cancels plus sI. 279 00:16:45,770 --> 00:16:46,980 I'm finished. 280 00:16:46,980 --> 00:16:51,930 And lo and behold, we have the same similarity. 281 00:16:51,930 --> 00:16:56,520 So we messed around by a multiple of the identity, 282 00:16:56,520 --> 00:16:58,440 and it didn't-- 283 00:16:58,440 --> 00:17:04,839 it actually makes the thing converge faster 284 00:17:04,839 --> 00:17:08,950 if we choose the shifts well. 285 00:17:08,950 --> 00:17:12,640 But basically, the same idea is still working. 286 00:17:12,640 --> 00:17:16,230 So that's the QR method. 287 00:17:16,230 --> 00:17:17,819 Well, that's the method. 288 00:17:17,819 --> 00:17:22,260 We haven't shown and won't show that-- 289 00:17:22,260 --> 00:17:28,970 except for this half-completed example, 290 00:17:28,970 --> 00:17:36,310 I don't plan to prove that the lower triangular part begins 291 00:17:36,310 --> 00:17:40,130 to disappear, gets smaller and smaller and smaller, 292 00:17:40,130 --> 00:17:44,000 and then the eigenvalues pop up on the diagonal. 293 00:17:44,000 --> 00:17:44,755 It's amazing. 294 00:17:47,700 --> 00:17:50,480 Amazing. 295 00:17:50,480 --> 00:17:55,430 Now, is there any other improvement we can make? 296 00:17:55,430 --> 00:17:57,820 So that's the method. 297 00:17:57,820 --> 00:18:00,760 And where is the work in using that method? 298 00:18:00,760 --> 00:18:03,160 Because that's what we always focus on. 299 00:18:03,160 --> 00:18:06,430 Where are we spending computer time? 300 00:18:06,430 --> 00:18:08,230 Well, we're spending computer time 301 00:18:08,230 --> 00:18:13,880 in doing the factorization. 302 00:18:13,880 --> 00:18:17,380 So it didn't cost anything to shift by the identity, 303 00:18:17,380 --> 00:18:21,270 but then we had to factor that into Q0, R0. 304 00:18:21,270 --> 00:18:24,070 Then it didn't cost much to multiply them 305 00:18:24,070 --> 00:18:25,580 in the opposite order. 306 00:18:25,580 --> 00:18:29,050 So the work was in QR. 307 00:18:29,050 --> 00:18:36,650 So could we think of anything to improve that aspect? 308 00:18:36,650 --> 00:18:38,760 Can we think of anything there? 309 00:18:38,760 --> 00:18:43,910 And then we've got a really first class method. 310 00:18:43,910 --> 00:18:48,790 Well, the matrix A-- 311 00:18:48,790 --> 00:18:52,730 A0, the matrix we started with-- 312 00:18:52,730 --> 00:19:02,690 had some zeros that allowed us to skip steps 313 00:19:02,690 --> 00:19:05,370 in doing the QR factorization. 314 00:19:05,370 --> 00:19:07,610 So what am I going to say? 315 00:19:07,610 --> 00:19:14,570 I'm going to say if A or A0, our original matrix, 316 00:19:14,570 --> 00:19:16,100 has a bunch of zeros-- 317 00:19:16,100 --> 00:19:18,410 let's say it's got a whole lot of zeros there. 318 00:19:18,410 --> 00:19:18,950 Maybe it's-- 319 00:19:25,530 --> 00:19:26,940 Well, OK. 320 00:19:26,940 --> 00:19:29,460 I overdid it here. 321 00:19:29,460 --> 00:19:33,470 I know the eigenvalues right off. 322 00:19:33,470 --> 00:19:37,130 But so the truth is I can't-- 323 00:19:37,130 --> 00:19:41,060 saying that if that is not going to happen. 324 00:19:41,060 --> 00:19:48,470 But we can get zeros with one extra diagonal. 325 00:19:48,470 --> 00:19:51,050 That turns out-- so here is the main diagonal. 326 00:19:51,050 --> 00:19:55,250 Everybody's got his eye on the main diagonal. 327 00:19:55,250 --> 00:20:01,140 And one diagonal-- 328 00:20:01,140 --> 00:20:06,830 I can get a lot of zeros, but I can't by simple computations. 329 00:20:06,830 --> 00:20:10,740 And I'll show you how to get one. 330 00:20:10,740 --> 00:20:12,680 But I can't get all those to be 0, 331 00:20:12,680 --> 00:20:18,300 because then I would have the eigenvalues right there. 332 00:20:18,300 --> 00:20:20,700 Well, how do I know that I can't? 333 00:20:20,700 --> 00:20:24,600 In elimination, ordinary solving AX equal B, 334 00:20:24,600 --> 00:20:29,520 you really do get to an upper triangular form. 335 00:20:29,520 --> 00:20:35,670 Some operator, some elimination steps you plug away, 336 00:20:35,670 --> 00:20:42,030 and your matrix becomes upper triangular U and you're golden. 337 00:20:42,030 --> 00:20:48,000 But that's too much to expect here. 338 00:20:48,000 --> 00:20:52,460 In fact, we know we can't do it by simple steps 339 00:20:52,460 --> 00:20:55,500 because if we could do it, if we could get to a U 340 00:20:55,500 --> 00:20:58,860 with a whole lower triangular part 0, 341 00:20:58,860 --> 00:21:02,230 we would have found the eigenvalues. 342 00:21:02,230 --> 00:21:06,040 And we know that the eigenvalues solve a system-- 343 00:21:06,040 --> 00:21:09,420 solve an equation of nth degree. 344 00:21:09,420 --> 00:21:16,440 And we know-- somebody proved a century or more ago-- 345 00:21:16,440 --> 00:21:18,900 that you can't solve an nth degree equation 346 00:21:18,900 --> 00:21:22,100 by simple little steps. 347 00:21:22,100 --> 00:21:24,510 Do you know who that was and that you 348 00:21:24,510 --> 00:21:30,450 know that fact and what degree does it apply to? 349 00:21:30,450 --> 00:21:36,540 So that's an important fact that you pick up in math. 350 00:21:36,540 --> 00:21:37,040 Yeah? 351 00:21:37,040 --> 00:21:38,300 AUDIENCE: The 5th [INAUDIBLE]. 352 00:21:38,300 --> 00:21:39,675 GILBERT STRANG: 5th degree, yeah. 353 00:21:39,675 --> 00:21:44,280 So 5 by 5 and up would be-- 354 00:21:44,280 --> 00:21:45,870 this is impossible. 355 00:21:45,870 --> 00:21:48,830 Impossible. 356 00:21:48,830 --> 00:22:02,700 There is no formula to find a simple formula for the lambdas. 357 00:22:02,700 --> 00:22:06,200 And similarly for the sigmas for singular values. 358 00:22:06,200 --> 00:22:10,680 So the eigenvalues is definitely a level 359 00:22:10,680 --> 00:22:15,330 of difficulty beyond AX equal B, the inverse matrix, 360 00:22:15,330 --> 00:22:17,220 or something, that pivots. 361 00:22:17,220 --> 00:22:20,910 All that you can do exactly if you're doing exact arithmetic. 362 00:22:20,910 --> 00:22:27,970 We cannot find the lambdas exactly, 363 00:22:27,970 --> 00:22:35,840 but we can get as close as we like by continuing with the QR 364 00:22:35,840 --> 00:22:36,340 method. 365 00:22:36,340 --> 00:22:37,570 So yeah. 366 00:22:37,570 --> 00:22:39,040 In other words, we can't-- 367 00:22:39,040 --> 00:22:41,350 we have to settle for-- 368 00:22:41,350 --> 00:22:44,620 if we want to, like at the beginning, 369 00:22:44,620 --> 00:22:49,980 improve our matrix before we start doing that stuff, 370 00:22:49,980 --> 00:22:52,860 we can get it with one extra diagonal. 371 00:22:52,860 --> 00:22:56,330 And do you know what kind of a matrix, whose name-- 372 00:22:56,330 --> 00:22:57,675 I don't know why. 373 00:22:57,675 --> 00:22:58,890 AUDIENCE: Upper Hessenberg. 374 00:22:58,890 --> 00:22:59,190 GILBERT STRANG: Yeah. 375 00:22:59,190 --> 00:22:59,745 Say it again. 376 00:22:59,745 --> 00:23:00,870 AUDIENCE: Upper Hessenberg. 377 00:23:00,870 --> 00:23:02,245 GILBERT STRANG: Upper Hessenberg. 378 00:23:02,245 --> 00:23:04,260 So upper is just like upper triangular. 379 00:23:04,260 --> 00:23:05,520 It's up there. 380 00:23:05,520 --> 00:23:10,980 But key person's name is Hessenberg. 381 00:23:10,980 --> 00:23:17,120 As I say, that's a Hessenberg matrix. 382 00:23:17,120 --> 00:23:21,810 So Hessenberg matrix is a matrix with one triangular 383 00:23:21,810 --> 00:23:26,680 plus one more diagonal, but lots of zeros. 384 00:23:26,680 --> 00:23:28,450 Order of N squared. 385 00:23:28,450 --> 00:23:31,210 Something like almost like half N squared-- 386 00:23:31,210 --> 00:23:34,640 not quite, but close-- 387 00:23:34,640 --> 00:23:35,920 zeros. 388 00:23:35,920 --> 00:23:42,280 And you could show those zeros stay zeros in QR. 389 00:23:42,280 --> 00:23:44,810 So that really pays off. 390 00:23:44,810 --> 00:23:49,340 It cuts the work down significantly. 391 00:23:49,340 --> 00:23:55,750 So that's the-- full QR method is Step 1-- 392 00:23:55,750 --> 00:24:08,110 reduce A to Hessenberg form with these zeros. 393 00:24:08,110 --> 00:24:11,410 And when I say reduce, I mean find a similarity 394 00:24:11,410 --> 00:24:13,540 transformation, of course, because I 395 00:24:13,540 --> 00:24:19,030 want the eigenvalues of this to end up the same as the Hessen. 396 00:24:19,030 --> 00:24:21,820 I want to keep the same eigenvalues as I go. 397 00:24:21,820 --> 00:24:28,495 And then, Step 2 is QR on this Hessenberg matrix with shifts. 398 00:24:33,250 --> 00:24:37,970 So that's the code that would be programmed in eig(A). 399 00:24:41,060 --> 00:24:42,990 That's what Matlab and-- 400 00:24:42,990 --> 00:24:46,110 well really, Matlab is appealing-- 401 00:24:46,110 --> 00:24:49,800 like other matrix systems-- 402 00:24:49,800 --> 00:24:54,360 is appealing to LAPACK and LINPACK. 403 00:24:54,360 --> 00:24:59,160 A team of professional numerical analysts 404 00:24:59,160 --> 00:25:04,200 really spent a lot of effort and time. 405 00:25:04,200 --> 00:25:14,220 The book LAPACK has 10 authors, and you 406 00:25:14,220 --> 00:25:18,870 can download any of these codes, like the eigenvalue code. 407 00:25:18,870 --> 00:25:23,760 So that's where Matlab, naturally-- 408 00:25:23,760 --> 00:25:29,580 that's the Bible for code in linear algebra. 409 00:25:32,640 --> 00:25:35,780 I think it's interesting to know. 410 00:25:35,780 --> 00:25:42,030 And there's one more good thing to tell you about this method. 411 00:25:42,030 --> 00:25:45,910 And it applies if the matrix is symmetric. 412 00:25:45,910 --> 00:25:49,370 If the matrix is symmetric, then if we check all this, 413 00:25:49,370 --> 00:25:55,450 we could find that the matrices stayed symmetric. 414 00:25:55,450 --> 00:25:58,210 If A0 is symmetric, I can check-- 415 00:25:58,210 --> 00:26:00,430 you could easily check through this 416 00:26:00,430 --> 00:26:03,130 and you would discover that A1 is also symmetric. 417 00:26:08,630 --> 00:26:12,140 It turns out you could rewrite this 418 00:26:12,140 --> 00:26:17,850 with a Q0 in a Q0 inverse on the other side, 419 00:26:17,850 --> 00:26:21,360 but that Q0 inverse is the same as Q0 transposed 420 00:26:21,360 --> 00:26:23,220 because it's an orthogonal matrix, 421 00:26:23,220 --> 00:26:26,010 and symmetry would fall out. 422 00:26:26,010 --> 00:26:31,630 So if it's symmetric, and it's in his Hessenberg form 423 00:26:31,630 --> 00:26:34,980 and it stays symmetric at every step, 424 00:26:34,980 --> 00:26:39,840 what can you tell me about a symmetric Heisenberg matrix? 425 00:26:39,840 --> 00:26:41,058 AUDIENCE: [INAUDIBLE] 426 00:26:41,058 --> 00:26:42,350 GILBERT STRANG: It's only got-- 427 00:26:42,350 --> 00:26:42,980 yeah. 428 00:26:42,980 --> 00:26:45,020 You just erase all these. 429 00:26:45,020 --> 00:26:48,920 If they are zeros there and if the matrix is symmetric, 430 00:26:48,920 --> 00:26:54,990 then we can safely predict that it will only 431 00:26:54,990 --> 00:26:57,940 have one diagonal above, one non-zero 432 00:26:57,940 --> 00:27:00,120 diagonal above the main diagonal. 433 00:27:00,120 --> 00:27:04,200 In fact, it will say symmetric. 434 00:27:04,200 --> 00:27:09,540 So now I should write "symmetric Hessenberg matrix," 435 00:27:09,540 --> 00:27:13,380 and equals tridiagonal matrix. 436 00:27:19,900 --> 00:27:23,380 Three diagonals. 437 00:27:23,380 --> 00:27:25,540 So now you really have reduced the time 438 00:27:25,540 --> 00:27:32,000 to do QR because you've got a tridiagonal matrix. 439 00:27:32,000 --> 00:27:35,360 It'll stay tridiagonal in all these steps. 440 00:27:35,360 --> 00:27:38,520 So you're working with just three N numbers. 441 00:27:38,520 --> 00:27:42,890 Well actually, two N because the diagonal above and the diagonal 442 00:27:42,890 --> 00:27:44,760 below are the same. 443 00:27:44,760 --> 00:27:47,540 You're working with just two N numbers instead of order N 444 00:27:47,540 --> 00:27:50,600 squared, and it just goes like a bomb. 445 00:27:50,600 --> 00:27:55,460 So that's eig for symmetric matrices. 446 00:27:55,460 --> 00:27:57,900 And you see that it was all based-- 447 00:27:57,900 --> 00:28:08,480 that really the heart of the algorithm was QR. 448 00:28:08,480 --> 00:28:12,230 So that's my-- that took half the class 449 00:28:12,230 --> 00:28:20,350 to report on the favorite way, the eig way, 450 00:28:20,350 --> 00:28:23,650 to find eigenvalues. 451 00:28:23,650 --> 00:28:29,450 Oh, I should say something about singular values. 452 00:28:29,450 --> 00:28:31,760 So singular values. 453 00:28:31,760 --> 00:28:35,910 Of course, the singular values of the matrix 454 00:28:35,910 --> 00:28:39,870 are the eigenvalues of A transpose A-- 455 00:28:39,870 --> 00:28:43,140 square root of those eigenvalues. 456 00:28:43,140 --> 00:28:44,550 But you wouldn't do it that way. 457 00:28:44,550 --> 00:28:46,470 You would never form A transpose A. 458 00:28:46,470 --> 00:28:49,830 Oh, I didn't mention the other thing you would never, 459 00:28:49,830 --> 00:28:57,000 ever, ever do, so let me just put it here like in disgrace. 460 00:28:57,000 --> 00:29:03,870 To solve that equation is like, OK, back to first grade, 461 00:29:03,870 --> 00:29:06,870 because that's not-- 462 00:29:06,870 --> 00:29:09,140 that's very bad. 463 00:29:09,140 --> 00:29:14,280 A determinant-- first of all, it's extremely slow-- extremely 464 00:29:14,280 --> 00:29:15,090 slow. 465 00:29:15,090 --> 00:29:21,870 And the determinant is packing all this N squared pieces 466 00:29:21,870 --> 00:29:24,850 of information into N coefficient, 467 00:29:24,850 --> 00:29:28,110 and it's hopelessly ill conditioned. 468 00:29:28,110 --> 00:29:28,610 Yeah. 469 00:29:28,610 --> 00:29:31,320 You lose information all the time. 470 00:29:31,320 --> 00:29:36,100 So really, if this is going on camera, 471 00:29:36,100 --> 00:29:40,240 it better go on camera with an x, because you don't do it. 472 00:29:42,820 --> 00:29:44,410 Yeah. 473 00:29:44,410 --> 00:29:47,810 So where was I? 474 00:29:47,810 --> 00:29:49,940 Singular values. 475 00:29:49,940 --> 00:29:56,670 So A transpose A. So again, let's 476 00:29:56,670 --> 00:29:59,940 think about what you could do at the beginning 477 00:29:59,940 --> 00:30:05,670 before starting on QR for A transpose A 478 00:30:05,670 --> 00:30:08,880 or for the matrix A. What could you 479 00:30:08,880 --> 00:30:13,540 do with orthogonal matrices? 480 00:30:13,540 --> 00:30:17,610 So I guess-- what did we say about symmetric matrices? 481 00:30:17,610 --> 00:30:20,250 So here's what I said about symmetric matrices. 482 00:30:20,250 --> 00:30:22,510 If you'll give me a symmetric matrix, 483 00:30:22,510 --> 00:30:26,820 I can in just a simple number of simple steps 484 00:30:26,820 --> 00:30:28,320 make it tridiagonal. 485 00:30:31,230 --> 00:30:33,090 I can't make it diagonal because then I'd 486 00:30:33,090 --> 00:30:39,300 be finding the eigenvalues and Abel, who was the first person 487 00:30:39,300 --> 00:30:42,360 to see that that was impossible, forbids it. 488 00:30:45,290 --> 00:30:48,450 So let me let me write down what I'm saying here. 489 00:30:48,450 --> 00:30:51,750 If I have a symmetric matrix S, I 490 00:30:51,750 --> 00:30:56,720 can find a bunch of Qs and Q transposes, 491 00:30:56,720 --> 00:31:00,510 and I can put them all together into one big Q, 492 00:31:00,510 --> 00:31:02,490 and it's transposed. 493 00:31:02,490 --> 00:31:05,205 And what do I know about the eigenvalues of that matrix? 494 00:31:09,780 --> 00:31:12,000 Q is orthogonal always. 495 00:31:12,000 --> 00:31:13,860 So what can you tell me about the-- 496 00:31:13,860 --> 00:31:20,400 this is the same as QSQ inverse, and therefore the eigenvalues 497 00:31:20,400 --> 00:31:23,520 are the same. 498 00:31:23,520 --> 00:31:35,630 It's similar to S. And it becomes tridiagonal After? 499 00:31:35,630 --> 00:31:43,255 I find a good Q. Therefore, same lambdas. 500 00:31:47,007 --> 00:31:50,240 It's tridiagonal with the same lambdas. 501 00:31:50,240 --> 00:31:52,820 Now, what am I thinking about here? 502 00:31:52,820 --> 00:32:00,390 I'm thinking about-- tell me the corresponding possibility 503 00:32:00,390 --> 00:32:01,770 about singular values. 504 00:32:01,770 --> 00:32:04,470 I wanted to do something to my matrix. 505 00:32:04,470 --> 00:32:11,126 Now, I'm always taking a general matrix A. 506 00:32:11,126 --> 00:32:14,070 And I'm looking for its singular values. 507 00:32:14,070 --> 00:32:17,590 And I'm looking to simplify it. 508 00:32:17,590 --> 00:32:21,240 And what am I allowed to do? 509 00:32:21,240 --> 00:32:24,070 Yeah, I guess my question is-- 510 00:32:24,070 --> 00:32:28,210 similarity transformations left the eigenvalues alone. 511 00:32:28,210 --> 00:32:32,270 What can I do that leaves the singular values alone? 512 00:32:32,270 --> 00:32:35,590 That's a fundamental question because it was so fundamental 513 00:32:35,590 --> 00:32:37,240 for eigenvalues. 514 00:32:37,240 --> 00:32:41,110 By doing this, a matrix and its inverse, 515 00:32:41,110 --> 00:32:44,440 I got something similar, and I checked even 516 00:32:44,440 --> 00:32:49,370 in this class that the eigenvalues, same lambdas. 517 00:32:49,370 --> 00:32:56,200 Now I want a whole line that ends up with the same sigmas. 518 00:32:56,200 --> 00:33:01,930 And I want you to tell me what I'm allowed to do to the matrix 519 00:33:01,930 --> 00:33:03,880 without changing the sigmas. 520 00:33:03,880 --> 00:33:05,500 So this is a-- 521 00:33:05,500 --> 00:33:07,950 maybe don't shout it out immediately. 522 00:33:07,950 --> 00:33:10,030 Let everybody think. 523 00:33:10,030 --> 00:33:13,330 What am I allowed to do to a matrix? 524 00:33:13,330 --> 00:33:16,790 Every matrix has got these singular values, 525 00:33:16,790 --> 00:33:21,250 and now I want to make it a better matrix 526 00:33:21,250 --> 00:33:25,020 with more zeros or something. 527 00:33:25,020 --> 00:33:32,530 If I do that to it, does that change the sigmas? 528 00:33:32,530 --> 00:33:35,080 Can I do more than that to it? 529 00:33:35,080 --> 00:33:36,630 What can I do? 530 00:33:36,630 --> 00:33:42,970 What group of matrices will have the same sigmas 531 00:33:42,970 --> 00:33:45,710 as my starting matrix A? 532 00:33:45,710 --> 00:33:50,500 So that's a basic, basic question about singular values 533 00:33:50,500 --> 00:33:52,300 and the SVD. 534 00:33:52,300 --> 00:33:57,390 So let's think of the answer together. 535 00:33:57,390 --> 00:34:02,490 So it's connected to the SVD, so let me remember the SVD. 536 00:34:02,490 --> 00:34:06,250 The SVD-- I have some orthogonal matrix. 537 00:34:06,250 --> 00:34:08,310 Then the singular value of the matrix-- 538 00:34:08,310 --> 00:34:10,650 SV for singular values. 539 00:34:10,650 --> 00:34:12,930 And then another orthogonal matrix. 540 00:34:16,670 --> 00:34:20,560 What could I do to that equation that would not touch this guy? 541 00:34:25,330 --> 00:34:28,870 So I'm asking, what invariants? 542 00:34:28,870 --> 00:34:32,909 Because not touching it means leaving it not varying. 543 00:34:32,909 --> 00:34:39,060 So I'm looking for under what operations are the singular 544 00:34:39,060 --> 00:34:41,040 values invariant? 545 00:34:41,040 --> 00:34:43,570 When I was looking at eigenvalues, 546 00:34:43,570 --> 00:34:45,040 this was the operation. 547 00:34:45,040 --> 00:34:47,940 Well, it didn't have to be orthogonal-- 548 00:34:47,940 --> 00:34:50,159 something, then its inverse. 549 00:34:50,159 --> 00:34:52,420 But now, what is it up there? 550 00:34:52,420 --> 00:34:54,780 What could I do to that matrix A? 551 00:34:57,850 --> 00:34:59,900 Could I multiply it by Q? 552 00:34:59,900 --> 00:35:04,690 Could I throw in a Q maybe not even on the other end? 553 00:35:04,690 --> 00:35:10,760 If I throw in an orthogonal Q, do I change the singular values 554 00:35:10,760 --> 00:35:12,470 or do I not change them? 555 00:35:12,470 --> 00:35:15,170 Fundamental question. 556 00:35:15,170 --> 00:35:18,770 The answer is no, I don't change them. 557 00:35:18,770 --> 00:35:19,670 I'm allowed to do. 558 00:35:19,670 --> 00:35:23,540 That because here's an orthogonal matrix, 559 00:35:23,540 --> 00:35:26,870 a Q times U. If both of those are orthogonal, 560 00:35:26,870 --> 00:35:29,060 then the product is. 561 00:35:29,060 --> 00:35:33,020 Everybody knows that a product of two orthogonal matrix 562 00:35:33,020 --> 00:35:35,020 is still orthogonal. 563 00:35:35,020 --> 00:35:36,650 Better know that. 564 00:35:36,650 --> 00:35:37,400 Better know that. 565 00:35:39,970 --> 00:35:42,670 So if I have an orthogonal matrix Q 566 00:35:42,670 --> 00:35:45,370 and an orthogonal matrix U, I claim 567 00:35:45,370 --> 00:35:51,650 that this is still orthogonal. 568 00:35:51,650 --> 00:35:53,330 And how do I check it? 569 00:35:53,330 --> 00:35:56,450 Well, I use some test for orthogonality. 570 00:35:56,450 --> 00:35:57,790 What would be the-- 571 00:35:57,790 --> 00:36:01,180 what test do you like to use? 572 00:36:01,180 --> 00:36:03,115 The inverse is the same as that transpose, 573 00:36:03,115 --> 00:36:05,800 do you like that test? 574 00:36:05,800 --> 00:36:08,010 So I'll invert it. 575 00:36:08,010 --> 00:36:10,660 QU inverse. 576 00:36:10,660 --> 00:36:13,705 Of course, for any matrix that's U inverse, Q inverse. 577 00:36:16,220 --> 00:36:18,470 But these were separately orthogonal, 578 00:36:18,470 --> 00:36:20,990 so that's U transpose Q transpose. 579 00:36:23,660 --> 00:36:28,550 And that is the same as QU transpose. 580 00:36:31,950 --> 00:36:36,920 So I used the orthogonality of U and the orthogonality of Q 581 00:36:36,920 --> 00:36:40,770 to conclude that the inverse is the transpose. 582 00:36:40,770 --> 00:36:44,060 So the answer is yes, I could do that. 583 00:36:44,060 --> 00:36:45,620 Now, with singular value-- 584 00:36:45,620 --> 00:36:50,420 with eigenvalues, I had to multiply on the other side 585 00:36:50,420 --> 00:36:53,960 by Q inverse or Q transpose. 586 00:36:53,960 --> 00:36:55,160 Do I have to do that now? 587 00:36:55,160 --> 00:36:56,960 No. 588 00:36:56,960 --> 00:36:58,680 What can I do on the right-hand side? 589 00:37:01,600 --> 00:37:03,240 I can multiply by-- 590 00:37:03,240 --> 00:37:06,050 I can leave it alone. 591 00:37:06,050 --> 00:37:10,040 Then it has the same singular values because it's 592 00:37:10,040 --> 00:37:11,580 the same sigma in there. 593 00:37:11,580 --> 00:37:14,910 If I have a orthogonal matrix times a diagonal 594 00:37:14,910 --> 00:37:18,090 times an orthogonal, that diagonal is-- 595 00:37:18,090 --> 00:37:22,290 positive diagonal is going to be sigma. 596 00:37:22,290 --> 00:37:26,620 So what can I do on this side? 597 00:37:26,620 --> 00:37:32,380 I can multiply by any orthogonal matrix on that side too. 598 00:37:32,380 --> 00:37:37,330 So let's call this guy Q1 and this guy Q2. 599 00:37:37,330 --> 00:37:40,090 I still have an orthogonal matrix there, 600 00:37:40,090 --> 00:37:43,450 orthogonal matrix there, and the same sigma 601 00:37:43,450 --> 00:37:45,070 popped in the middle. 602 00:37:45,070 --> 00:37:48,610 So that's what you're allowed to do. 603 00:37:48,610 --> 00:37:51,250 That gives us more freedom. 604 00:37:51,250 --> 00:37:55,660 Before we got-- when we had to do similarity transformations 605 00:37:55,660 --> 00:38:00,640 with the same guy, we got it to be tridiagonal. 606 00:38:00,640 --> 00:38:04,960 But now, we're allowed to do more stuff. 607 00:38:04,960 --> 00:38:09,880 We're allowed to use different orthogonal matrices on the left 608 00:38:09,880 --> 00:38:11,290 and right. 609 00:38:11,290 --> 00:38:13,420 And we can reduce it even further 610 00:38:13,420 --> 00:38:17,110 from tridiagonal to bidiagonal. 611 00:38:17,110 --> 00:38:21,250 So the first step is getting zero. 612 00:38:21,250 --> 00:38:25,510 The step of getting zeros reduces it all the way 613 00:38:25,510 --> 00:38:28,480 to that, with all zeros there. 614 00:38:31,290 --> 00:38:37,690 So it's easier. 615 00:38:37,690 --> 00:38:40,570 Then I work on this. 616 00:38:40,570 --> 00:38:44,230 This is the matrix I work on using a QR type 617 00:38:44,230 --> 00:38:46,810 idea, some method like that. 618 00:38:46,810 --> 00:38:53,360 So everybody's seeing that our algorithm has got two stages. 619 00:38:53,360 --> 00:38:56,960 One is get a lot of zeros and get them 620 00:38:56,960 --> 00:39:02,450 in places that will stay zero as Part 2 of the algorithm 621 00:39:02,450 --> 00:39:03,860 gets going. 622 00:39:03,860 --> 00:39:06,640 And then, run Part 2 of the algorithm. 623 00:39:06,640 --> 00:39:10,910 You staying with-- each step is very fast now 624 00:39:10,910 --> 00:39:17,570 because doing a QR factorization is fast. 625 00:39:17,570 --> 00:39:18,750 Was there a question? 626 00:39:18,750 --> 00:39:20,420 Yeah. 627 00:39:20,420 --> 00:39:23,710 So I would call this bidiagonal, of course. 628 00:39:26,760 --> 00:39:31,250 And everybody recognizes that if I have a bidiagonal matrix-- 629 00:39:31,250 --> 00:39:35,530 call it A or A0 or whatever-- 630 00:39:35,530 --> 00:39:39,820 then what do you think about A transpose A? 631 00:39:39,820 --> 00:39:43,970 What would A transpose A-- if that was A, 632 00:39:43,970 --> 00:39:46,460 what could you tell me about A transpose A? 633 00:39:50,710 --> 00:39:57,570 Could you multiply matrices knowing where the non-zeros are 634 00:39:57,570 --> 00:40:00,210 in your head and get an idea of where-- 635 00:40:00,210 --> 00:40:06,000 and so if I have a bidiagonal matrix A, then implicitly 636 00:40:06,000 --> 00:40:10,590 in the SVD, I'm looking at A transpose A. 637 00:40:10,590 --> 00:40:15,950 And what would be true about A transpose A? 638 00:40:15,950 --> 00:40:18,750 It would be tridiagonal. 639 00:40:18,750 --> 00:40:25,410 So what I've done here and what I've done there just match up. 640 00:40:25,410 --> 00:40:29,700 You can operate-- if you don't want to change singular values, 641 00:40:29,700 --> 00:40:32,220 you can get all the way to here. 642 00:40:32,220 --> 00:40:35,100 But then, to find those singular values, that 643 00:40:35,100 --> 00:40:37,590 would involve A transpose A. It would 644 00:40:37,590 --> 00:40:44,010 be symmetric and tridiagonal, and then you'd be in that game. 645 00:40:44,010 --> 00:40:50,950 So those are the basic facts of eig and SVD 646 00:40:50,950 --> 00:40:56,720 for matrices of order up to 1,000, say. 647 00:40:56,720 --> 00:40:59,265 I'm not enough of an expert to know where-- 648 00:41:03,130 --> 00:41:09,670 Maybe higher, because in perfect math, 649 00:41:09,670 --> 00:41:12,930 it's going to take infinitely many steps 650 00:41:12,930 --> 00:41:16,050 or Abel would be very surprised. 651 00:41:16,050 --> 00:41:19,670 He would see you solving for eigenvalues an nth degree 652 00:41:19,670 --> 00:41:21,960 equation by a whole lot of little steps 653 00:41:21,960 --> 00:41:23,310 and getting them exactly right. 654 00:41:23,310 --> 00:41:24,480 That won't happen. 655 00:41:24,480 --> 00:41:29,160 But you get them within epsilon in a number 656 00:41:29,160 --> 00:41:32,170 of steps that's like N cubed. 657 00:41:32,170 --> 00:41:35,040 So that's pretty impressive. 658 00:41:35,040 --> 00:41:38,340 The eigenvalue problem is being "solved," 659 00:41:38,340 --> 00:41:47,020 in quotes, by a fast method that gets 660 00:41:47,020 --> 00:41:52,360 you a good answer within a tolerance in N cube steps. 661 00:41:55,290 --> 00:41:58,810 So that's great as long as N isn't too big. 662 00:41:58,810 --> 00:42:03,400 And then, when N is too big-- which, of course, happens-- 663 00:42:03,400 --> 00:42:04,910 you have to think again. 664 00:42:04,910 --> 00:42:12,940 So this method is a giant success up to large matrices, 665 00:42:12,940 --> 00:42:15,780 but then you have to think again. 666 00:42:15,780 --> 00:42:19,270 And what is involved in thinking again? 667 00:42:19,270 --> 00:42:24,030 Well, I guess more thinking. 668 00:42:24,030 --> 00:42:26,490 So what do you do if the matrix is bigger? 669 00:42:29,710 --> 00:42:35,960 I guess that Krylov would say, use my method. 670 00:42:35,960 --> 00:42:39,550 So Krylov would say-- especially if your matrix was sparse. 671 00:42:39,550 --> 00:42:43,190 Can we just remember what Krylov was? 672 00:42:43,190 --> 00:42:47,150 Krylov started with a vector b, multiplied it 673 00:42:47,150 --> 00:42:53,210 by A, multiplied that by A, and got up to, 674 00:42:53,210 --> 00:42:55,182 let's say, A to the 999b. 675 00:42:59,870 --> 00:43:01,460 So now he's got-- 676 00:43:01,460 --> 00:43:05,300 Krylov has got 1,000-dimensional space. 677 00:43:05,300 --> 00:43:06,950 He's got a basis for it-- 678 00:43:06,950 --> 00:43:13,320 1,000 vectors that span 1,000-dimensional space. 679 00:43:13,320 --> 00:43:17,580 And he'll look at the matrix A only on that space. 680 00:43:17,580 --> 00:43:20,120 In other words-- 681 00:43:20,120 --> 00:43:22,850 I won't go into detail about that. 682 00:43:22,850 --> 00:43:28,130 He restricts the matrix to this 1,000-dimensional space, 683 00:43:28,130 --> 00:43:29,990 and he hopes that it's captured. 684 00:43:29,990 --> 00:43:33,510 We hope that the eigenvector is almost-- 685 00:43:33,510 --> 00:43:35,720 is virtually in that space. 686 00:43:35,720 --> 00:43:38,450 And actually, I wouldn't go to-- 687 00:43:38,450 --> 00:43:39,910 let me take a 9 out of that. 688 00:43:42,510 --> 00:43:46,170 100-dimensional would probably catch the eigenvector. 689 00:43:46,170 --> 00:43:50,290 So if the eigenvector is virtually in this space, 690 00:43:50,290 --> 00:43:53,590 then we can look at a matrix of order 100. 691 00:43:53,590 --> 00:43:59,700 We can bring A down to just see its action on that space. 692 00:43:59,700 --> 00:44:02,350 And any piece of-- 693 00:44:02,350 --> 00:44:06,190 so here is a-- so I look at vectors 694 00:44:06,190 --> 00:44:14,980 v, which are some combination-- c1 b plus c2 Ab plus c3 A 695 00:44:14,980 --> 00:44:20,710 squared b, and c100 A to the 99th b. 696 00:44:25,540 --> 00:44:26,410 Plus an error. 697 00:44:29,710 --> 00:44:36,280 And I'm going to ignore that error because I've 698 00:44:36,280 --> 00:44:38,640 gone up to dimension 100. 699 00:44:38,640 --> 00:44:42,010 I'd probably say it's pretty safe to ignore that error. 700 00:44:42,010 --> 00:44:47,050 And then, in this space, just looking at the matrix 701 00:44:47,050 --> 00:44:51,460 A-- so wherever A to the 100th comes in, forget it. 702 00:44:51,460 --> 00:44:56,650 Just think about the matrix A as multiplying vectors 703 00:44:56,650 --> 00:44:58,810 of this kind in this space. 704 00:44:58,810 --> 00:45:03,940 Then I have 100 by 100 eigenvalue problem. 705 00:45:03,940 --> 00:45:12,370 And so the big matrix A is reduced to a matrix of size 100 706 00:45:12,370 --> 00:45:13,060 by-- 707 00:45:13,060 --> 00:45:15,130 do you see what I'm saying even? 708 00:45:15,130 --> 00:45:17,260 So I'm not giving the details. 709 00:45:20,010 --> 00:45:25,050 Think of a matrix A of size a million. 710 00:45:27,810 --> 00:45:30,760 And you apply it to Krylov vectors-- 711 00:45:30,760 --> 00:45:33,750 so I call them little k for a Krylov vector-- 712 00:45:36,540 --> 00:45:38,190 in this 100-dimensional space. 713 00:45:41,310 --> 00:45:47,740 So they have a million minus 100 0-components, 714 00:45:47,740 --> 00:45:49,330 you could say, this k. 715 00:45:49,330 --> 00:45:51,350 This is in the Krylov space. 716 00:45:51,350 --> 00:45:53,770 This is A, a million-- 717 00:45:53,770 --> 00:45:58,720 k, 100. 718 00:45:58,720 --> 00:46:02,030 It's a full-- so it's got a million components, 719 00:46:02,030 --> 00:46:04,650 but it's out of just 100-dimensional space. 720 00:46:04,650 --> 00:46:10,880 So when I multiply by A, it'll be mostly in-- 721 00:46:10,880 --> 00:46:13,910 partly in the Krylov space-- 722 00:46:13,910 --> 00:46:21,880 k100-- and a piece out of k100. 723 00:46:21,880 --> 00:46:25,740 And I just ignore that part of the matrix. 724 00:46:25,740 --> 00:46:27,770 So I have 100 by 100 problem, and I 725 00:46:27,770 --> 00:46:29,570 solve to find the eigenvalues. 726 00:46:29,570 --> 00:46:31,970 And they're a pretty good approximation 727 00:46:31,970 --> 00:46:35,840 to the eigenvalues, to the, hopefully, 728 00:46:35,840 --> 00:46:38,210 like the lowest 100 eigenvalues. 729 00:46:38,210 --> 00:46:40,940 I'd like to know that, but I might not 730 00:46:40,940 --> 00:46:43,910 be sure that this idea would give me 731 00:46:43,910 --> 00:46:49,060 the lowest 100-- the first 100 eigenvalues of the million, 732 00:46:49,060 --> 00:46:51,140 of the matrix of size a million. 733 00:46:51,140 --> 00:46:58,780 I'm just taking a few minutes here to wave hands 734 00:46:58,780 --> 00:47:03,730 about what Krylov idea would do. 735 00:47:03,730 --> 00:47:07,550 And I probably won't mention Krylov again in this semester. 736 00:47:07,550 --> 00:47:14,630 So what it can do is look at this particular type of space 737 00:47:14,630 --> 00:47:17,660 because we can get a basis for it quickly. 738 00:47:17,660 --> 00:47:21,320 Just multiply again and again by A. Then we 739 00:47:21,320 --> 00:47:24,700 can orthogonalize that basis. 740 00:47:24,700 --> 00:47:27,160 That's Gram-Schmidt in some form. 741 00:47:27,160 --> 00:47:29,620 We're always going back to Gram-Schmidt. 742 00:47:29,620 --> 00:47:31,570 Then I have 100 by 100-- 743 00:47:31,570 --> 00:47:34,900 I have a subspace of size 100. 744 00:47:34,900 --> 00:47:38,440 I look at what the matrix does in that space, 745 00:47:38,440 --> 00:47:41,800 and that I could look for-- 746 00:47:41,800 --> 00:47:45,190 I could find eigenvalues restricted to that space. 747 00:47:45,190 --> 00:47:47,620 They wouldn't be the perfect eigenvalues, 748 00:47:47,620 --> 00:47:50,820 but they would be accurate. 749 00:47:50,820 --> 00:47:54,630 So I didn't know it would take one class time to talk 750 00:47:54,630 --> 00:47:57,330 about finding eigenvalues and singular values, 751 00:47:57,330 --> 00:48:00,330 but we did some important things. 752 00:48:00,330 --> 00:48:05,250 We remembered that similarity is the thing to check, 753 00:48:05,250 --> 00:48:09,360 thing to preserve, because it doesn't change the eigenvalues. 754 00:48:09,360 --> 00:48:14,760 And then we-- for singular values, what was the thing? 755 00:48:14,760 --> 00:48:16,740 You could multiply left and right 756 00:48:16,740 --> 00:48:20,100 by different orthogonal matrices. 757 00:48:20,100 --> 00:48:24,150 And somehow, maybe that doesn't have an established name, 758 00:48:24,150 --> 00:48:31,070 multiplying left and right by a Q1 and a Q2 transpose. 759 00:48:31,070 --> 00:48:34,520 But the idea is clear, and that doesn't 760 00:48:34,520 --> 00:48:37,680 change the singular values. 761 00:48:37,680 --> 00:48:42,630 We're ready to move now into-- 762 00:48:42,630 --> 00:48:47,550 maybe our next step, which we don't spend a long time on, 763 00:48:47,550 --> 00:48:49,890 will be random sampling. 764 00:48:49,890 --> 00:48:53,700 What if your matrix is just way too big? 765 00:48:53,700 --> 00:49:00,000 So that's a very new idea, very different idea 766 00:49:00,000 --> 00:49:05,160 in numerical linear algebra, is just to sample the matrix. 767 00:49:05,160 --> 00:49:06,780 Could you believe that the answer 768 00:49:06,780 --> 00:49:10,300 is going to come out right just for a random sample? 769 00:49:10,300 --> 00:49:13,110 Well, the odds are in your favor. 770 00:49:13,110 --> 00:49:15,810 So that will be Wednesday, and then we 771 00:49:15,810 --> 00:49:23,190 have lots of new-- we'll move onward after that. 772 00:49:23,190 --> 00:49:24,360 See you Wednesday. 773 00:49:24,360 --> 00:49:25,910 Thanks.