1 00:00:01,550 --> 00:00:03,920 The following content is provided under a Creative 2 00:00:03,920 --> 00:00:05,310 Commons license. 3 00:00:05,310 --> 00:00:07,520 Your support will help MIT OpenCourseWare 4 00:00:07,520 --> 00:00:11,610 continue to offer high-quality educational resources for free. 5 00:00:11,610 --> 00:00:14,180 To make a donation or to view additional materials 6 00:00:14,180 --> 00:00:18,140 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:18,140 --> 00:00:19,026 at ocw.mit.edu. 8 00:00:22,720 --> 00:00:27,910 GILBERT STRANG: So just to orient where we are, 9 00:00:27,910 --> 00:00:34,840 today starts a new chapter about low-rank matrices. 10 00:00:34,840 --> 00:00:38,830 So that's an important bunch of matrices. 11 00:00:38,830 --> 00:00:44,620 They can be truly low-rank, like uv transpose. 12 00:00:44,620 --> 00:00:46,720 That's a rank 1. 13 00:00:46,720 --> 00:00:51,560 And we have some questions about those. 14 00:00:51,560 --> 00:00:53,230 And then later in the chapter, we'll 15 00:00:53,230 --> 00:00:59,096 meet matrices that are approximately low-rank, where 16 00:00:59,096 --> 00:01:02,850 the singular values drop off like crazy. 17 00:01:02,850 --> 00:01:08,340 And those are quite remarkable matrices. 18 00:01:08,340 --> 00:01:11,910 So this is my topic at the beginning-- 19 00:01:11,910 --> 00:01:16,970 and if it doesn't take the whole hour, 20 00:01:16,970 --> 00:01:24,210 I want to go back to a topic in chapter 2-- 21 00:01:24,210 --> 00:01:25,200 that should be 2.4-- 22 00:01:27,750 --> 00:01:30,520 where we were last time. 23 00:01:30,520 --> 00:01:31,020 Good. 24 00:01:35,340 --> 00:01:38,640 So that's for later in the hour. 25 00:01:38,640 --> 00:01:42,080 This is for now. 26 00:01:42,080 --> 00:01:44,580 Let's focus on this. 27 00:01:44,580 --> 00:01:46,080 So what's the question there? 28 00:01:46,080 --> 00:01:48,870 I start with the identity matrix. 29 00:01:48,870 --> 00:01:54,750 I perturb it by a matrix of rank 1. 30 00:01:54,750 --> 00:01:57,720 And I ask what the inverse is. 31 00:01:57,720 --> 00:02:01,830 So I'm making a small change in the matrix. 32 00:02:01,830 --> 00:02:04,080 When I say small change, I don't mean 33 00:02:04,080 --> 00:02:06,760 that the numbers are small. 34 00:02:06,760 --> 00:02:13,150 In fact, uv transpose could be the all 1's matrix, 35 00:02:13,150 --> 00:02:15,940 or the all millions matrix, even. 36 00:02:15,940 --> 00:02:19,330 But its rank is small. 37 00:02:19,330 --> 00:02:23,110 That's the idea of small that's important here. 38 00:02:23,110 --> 00:02:26,100 And I would like to know what the inverse is. 39 00:02:28,810 --> 00:02:32,410 So that's the question here in 4.1. 40 00:02:32,410 --> 00:02:36,220 And there's a famous formula that has at least three names, 41 00:02:36,220 --> 00:02:41,140 and probably more, and it's also called the matrix inversion 42 00:02:41,140 --> 00:02:48,350 formula in signal processing. 43 00:02:54,250 --> 00:02:56,630 And let me write down the example. 44 00:02:56,630 --> 00:03:00,200 So I start with the identity matrix. 45 00:03:00,200 --> 00:03:04,760 I do a rank 1 perturbation. 46 00:03:04,760 --> 00:03:07,610 And I want to know, what's the inverse? 47 00:03:07,610 --> 00:03:12,800 And I'll write down the answer and check that it's correct. 48 00:03:12,800 --> 00:03:15,680 So I'm perturbing the identity matrix 49 00:03:15,680 --> 00:03:20,480 in this case, whose inverse is also the identity matrix. 50 00:03:20,480 --> 00:03:26,980 So I'm going to write the answer as a perturbation of I. 51 00:03:26,980 --> 00:03:30,220 And the question is, what is that perturbation? 52 00:03:30,220 --> 00:03:33,040 And it's a famous formula. 53 00:03:33,040 --> 00:03:36,870 There is a uv transpose that comes in, 54 00:03:36,870 --> 00:03:43,870 copied from there, divided by 1 minus v transpose u. 55 00:03:46,390 --> 00:03:49,270 Now, let's first of all just see that this 56 00:03:49,270 --> 00:03:52,930 is a reasonable formula. 57 00:03:52,930 --> 00:03:58,020 So u a column-- u and v are column vectors, as always. 58 00:03:58,020 --> 00:04:02,910 So uv transpose is a matrix, column times row. 59 00:04:02,910 --> 00:04:07,900 And it's being subtracted from the identity. 60 00:04:07,900 --> 00:04:12,450 And over here on this side, I have uv transpose-- again, 61 00:04:12,450 --> 00:04:14,740 a rank 1 matrix-- 62 00:04:14,740 --> 00:04:16,220 divided by a number. 63 00:04:16,220 --> 00:04:17,109 So that's the point. 64 00:04:17,109 --> 00:04:17,859 This is a number-- 65 00:04:23,040 --> 00:04:25,650 a 1 by 1 matrix, you could say. 66 00:04:28,470 --> 00:04:33,690 So the point of this formula is to find the inverse of an n 67 00:04:33,690 --> 00:04:37,320 by n matrix in terms of the inverse of a 1 68 00:04:37,320 --> 00:04:42,780 by 1 matrix, which is a lot simpler and easier to do. 69 00:04:42,780 --> 00:04:46,650 I mean, that right-hand side is clearly easy. 70 00:04:46,650 --> 00:04:51,160 And let's see what other things it tells us. 71 00:04:51,160 --> 00:04:56,380 So this was a rank 1 perturbation in the identity. 72 00:04:56,380 --> 00:04:59,200 Then when I invert, I get-- 73 00:04:59,200 --> 00:04:59,860 look at this. 74 00:04:59,860 --> 00:05:01,600 This is also a rank 1. 75 00:05:01,600 --> 00:05:04,150 That's a number. 76 00:05:04,150 --> 00:05:06,160 In fact, it's the very same rank 1 77 00:05:06,160 --> 00:05:10,990 that we had there in this nice case. 78 00:05:10,990 --> 00:05:20,320 So conclusion-- if I change a matrix by rank 1, 79 00:05:20,320 --> 00:05:23,290 I change its inverse by rank 1. 80 00:05:23,290 --> 00:05:26,860 That doesn't seem completely obvious 81 00:05:26,860 --> 00:05:31,210 until you go to figure it out. 82 00:05:31,210 --> 00:05:34,240 When you figure it out, you get a formula that tells you. 83 00:05:34,240 --> 00:05:38,830 Well, so far what I'm perturbing is the identity matrix. 84 00:05:38,830 --> 00:05:41,985 When I get here, I'm going to perturb any matrix. 85 00:05:41,985 --> 00:05:44,620 And I'm going to reach the same conclusion. 86 00:05:44,620 --> 00:05:52,150 If this is rank 1, then the change in the inverse 87 00:05:52,150 --> 00:05:53,800 is also rank 1. 88 00:05:53,800 --> 00:05:57,380 But let's see it first for a equal to the identity. 89 00:05:57,380 --> 00:06:01,480 So I guess I have two or three questions. 90 00:06:01,480 --> 00:06:04,405 One would be, how do you check that this is correct? 91 00:06:08,120 --> 00:06:12,520 Again, remember what it's doing. 92 00:06:12,520 --> 00:06:14,775 It's pretty neat. 93 00:06:14,775 --> 00:06:21,630 It's computing an n by n inverse in terms of a 1 by 1 inverse. 94 00:06:21,630 --> 00:06:22,350 It's a number. 95 00:06:24,970 --> 00:06:27,490 And that's certainly a favorable exchange. 96 00:06:27,490 --> 00:06:31,060 So it's a formula that you want and a formula 97 00:06:31,060 --> 00:06:31,900 that's quite useful. 98 00:06:35,410 --> 00:06:38,350 Let me just check the formula before I 99 00:06:38,350 --> 00:06:40,930 talk about how it's useful. 100 00:06:40,930 --> 00:06:42,230 So how shall I check it? 101 00:06:42,230 --> 00:06:45,190 I guess, if this is claimed to be the inverse, 102 00:06:45,190 --> 00:06:47,170 then let's just che-- 103 00:06:47,170 --> 00:06:49,660 so this will be the check. 104 00:06:49,660 --> 00:07:00,930 I'll multiply by that, I plus uv transpose over 1 105 00:07:00,930 --> 00:07:05,400 minus v transpose u, the claimed inverse. 106 00:07:05,400 --> 00:07:09,610 And what am I hoping to get from this multiplication? 107 00:07:09,610 --> 00:07:10,110 AUDIENCE: I. 108 00:07:10,110 --> 00:07:14,360 GILBERT STRANG: I. I'm hoping that, that result is 109 00:07:14,360 --> 00:07:15,990 the identity matrix. 110 00:07:15,990 --> 00:07:18,950 So I'm just going to do it out, and we'll 111 00:07:18,950 --> 00:07:21,020 see the identity appear. 112 00:07:21,020 --> 00:07:24,910 So that times I-- so let me just write that part first, 113 00:07:24,910 --> 00:07:28,700 I minus uv transpose. 114 00:07:28,700 --> 00:07:32,030 And now I minus uv transpose times this-- 115 00:07:32,030 --> 00:07:38,720 so that's I minus uv transpose times 116 00:07:38,720 --> 00:07:46,040 uv transpose over 1 minus v transpose u. 117 00:07:46,040 --> 00:07:51,440 It's just multiplying it out and seeing 118 00:07:51,440 --> 00:07:54,530 that we get the identity. 119 00:07:54,530 --> 00:07:56,630 I see the identity there. 120 00:07:56,630 --> 00:08:03,230 So I guess, I hope that all this part reduces to-- 121 00:08:03,230 --> 00:08:04,490 what does it reduce to? 122 00:08:07,610 --> 00:08:09,980 Let me do that numerator there. 123 00:08:09,980 --> 00:08:17,490 So that's uv transpose minus u-- 124 00:08:17,490 --> 00:08:22,740 ooh-- minus u times v transpose u-- 125 00:08:22,740 --> 00:08:25,410 do you see what's happening here-- 126 00:08:25,410 --> 00:08:26,310 v transpose. 127 00:08:30,390 --> 00:08:33,600 The key is that when I multiplied that times 128 00:08:33,600 --> 00:08:37,900 that, I've got four things in a row. 129 00:08:37,900 --> 00:08:40,570 And I do the middle pair first. 130 00:08:40,570 --> 00:08:43,270 Because that's a number, v transpose u. 131 00:08:43,270 --> 00:08:49,300 So I had uv transpose from that and uv transpose times 132 00:08:49,300 --> 00:08:50,590 that number minus-- 133 00:08:50,590 --> 00:08:53,170 do you see-- what have I got here? 134 00:08:56,540 --> 00:08:57,170 I've got-- 135 00:08:57,170 --> 00:08:58,245 AUDIENCE: [INAUDIBLE] 136 00:08:58,245 --> 00:08:59,120 GILBERT STRANG: Yeah. 137 00:08:59,120 --> 00:09:01,230 I've got uv transpose. 138 00:09:01,230 --> 00:09:04,830 Now I factor out a 1 minus v transpose u, 139 00:09:04,830 --> 00:09:08,390 which is just what I want. 140 00:09:08,390 --> 00:09:11,540 I factor the 1 minus v transpose u out of here. 141 00:09:11,540 --> 00:09:12,930 It's there. 142 00:09:12,930 --> 00:09:14,280 Cancel those. 143 00:09:14,280 --> 00:09:18,330 And then I'm left with a uv transpose, which cancels 144 00:09:18,330 --> 00:09:22,200 that and leaves the identity. 145 00:09:22,200 --> 00:09:24,900 It's kind of magic. 146 00:09:24,900 --> 00:09:30,130 Of course, somebody had to figure out that formula 147 00:09:30,130 --> 00:09:33,160 in the first place. 148 00:09:33,160 --> 00:09:43,510 I could do the next one, if you like, since I'm on a roll. 149 00:09:43,510 --> 00:09:47,800 Suppose I take this second one, now what's the difference in-- 150 00:09:47,800 --> 00:09:52,120 it's uv transpose, but those are larger letters, the u 151 00:09:52,120 --> 00:09:55,980 and the v. So what am I meaning by those? 152 00:09:55,980 --> 00:09:57,650 Those are matrices. 153 00:09:57,650 --> 00:10:00,200 This is a bigger rank. 154 00:10:00,200 --> 00:10:03,230 u has k columns. 155 00:10:03,230 --> 00:10:06,380 v transpose has k rows. 156 00:10:06,380 --> 00:10:11,960 That product uv transpose is k by k. 157 00:10:11,960 --> 00:10:15,830 So let me see what I think the formula would be. 158 00:10:15,830 --> 00:10:21,260 So now, I minus uv transpose inverse-- 159 00:10:21,260 --> 00:10:28,810 so this is n by k times k by n. 160 00:10:28,810 --> 00:10:31,000 So it's n by n matrix. 161 00:10:31,000 --> 00:10:32,320 And this is the ide-- 162 00:10:32,320 --> 00:10:33,730 identity of this. 163 00:10:33,730 --> 00:10:36,670 But its rank is-- 164 00:10:36,670 --> 00:10:38,763 what is the rank of that now? 165 00:10:38,763 --> 00:10:39,430 AUDIENCE: k. 166 00:10:39,430 --> 00:10:40,180 GILBERT STRANG: k. 167 00:10:40,180 --> 00:10:41,170 Right. 168 00:10:41,170 --> 00:10:47,200 So I have here, the whole thing is the inverse 169 00:10:47,200 --> 00:10:49,520 of an n by n matrix. 170 00:10:49,520 --> 00:10:51,130 So I have an n by n matrix. 171 00:10:51,130 --> 00:10:56,290 Let me put that up here, n by n matrix to invert. 172 00:11:02,300 --> 00:11:03,890 There it is. 173 00:11:03,890 --> 00:11:05,660 I'm going to write down the formula. 174 00:11:05,660 --> 00:11:07,535 And you're going to be able to write it down, 175 00:11:07,535 --> 00:11:10,880 because it's just copied from that one. 176 00:11:10,880 --> 00:11:16,470 And you'll see that it involves a k by k inverse. 177 00:11:16,470 --> 00:11:18,910 So I have an n by n matrix to invert, 178 00:11:18,910 --> 00:11:21,000 but I don't have to do that. 179 00:11:21,000 --> 00:11:23,870 I can switch it to a k by k matrix. 180 00:11:23,870 --> 00:11:27,300 That's pretty nice. 181 00:11:27,300 --> 00:11:29,370 So let's do it. 182 00:11:29,370 --> 00:11:37,090 So I'm basically gonna copy that I plus u-- 183 00:11:37,090 --> 00:11:40,070 now, I've matrices. 184 00:11:40,070 --> 00:11:42,890 So that's an inverse, but I can't leave it 185 00:11:42,890 --> 00:11:43,980 as a denominator. 186 00:11:43,980 --> 00:11:46,835 Because through here, we're talking about a k by k matrix. 187 00:11:51,130 --> 00:12:00,070 So I have to put it like any matrix inverse, Ik minus v 188 00:12:00,070 --> 00:12:06,040 transpose u inverse. 189 00:12:06,040 --> 00:12:08,170 So I'm just copying this formula, 190 00:12:08,170 --> 00:12:12,220 wisely taking the u first, then this part, 191 00:12:12,220 --> 00:12:13,800 then finally, the v transpose. 192 00:12:18,080 --> 00:12:20,230 I put this one up. 193 00:12:20,230 --> 00:12:23,620 This one now has made that one obsolete. 194 00:12:23,620 --> 00:12:25,900 This was the case k equal to 1 over here. 195 00:12:28,430 --> 00:12:30,990 But I think it helps to see it. 196 00:12:30,990 --> 00:12:33,640 It certainly helps me to see the formula first 197 00:12:33,640 --> 00:12:41,860 for a rank 1 perturbation, which is a very likely possibility, 198 00:12:41,860 --> 00:12:45,610 and then see that we have a totally analogous formula. 199 00:12:45,610 --> 00:12:48,940 It's sort of fun to check it. 200 00:12:48,940 --> 00:12:51,710 So I plan to do the same check. 201 00:12:51,710 --> 00:12:55,690 But I'll just notice here that I have a k by k inverse. 202 00:13:02,490 --> 00:13:06,390 So I'm exchanging something that's a perturbation of n 203 00:13:06,390 --> 00:13:11,880 by n identity, an n by n matrix, to have 204 00:13:11,880 --> 00:13:16,260 to invert something that's a k by k matrix, 205 00:13:16,260 --> 00:13:21,840 perturbing the k by k identity, much smaller. 206 00:13:21,840 --> 00:13:25,200 In that case, k was 1. 207 00:13:25,200 --> 00:13:28,620 So are you good for checking it now? 208 00:13:28,620 --> 00:13:32,010 So I want to multiply that by this 209 00:13:32,010 --> 00:13:35,830 and hopefully get In, the identity. 210 00:13:35,830 --> 00:13:38,155 This is the n by n identity. 211 00:13:43,390 --> 00:13:47,480 I haven't given credit to the two or three or four, 212 00:13:47,480 --> 00:13:50,560 or possibly 11 people, who found this formula. 213 00:13:53,300 --> 00:13:57,200 And I'm not see-- oh, yeah, here are their names. 214 00:13:57,200 --> 00:14:01,720 Is it OK to make them famous by putting their names up here? 215 00:14:01,720 --> 00:14:03,430 Yes. 216 00:14:03,430 --> 00:14:10,440 Sherman-- and I couldn't tell you who did what. 217 00:14:12,960 --> 00:14:22,840 Morrison, Woodbury, and then no doubt others-- 218 00:14:22,840 --> 00:14:26,470 maybe one of them did this rank 1 case, 219 00:14:26,470 --> 00:14:28,450 and then another generalized it. 220 00:14:28,450 --> 00:14:34,690 And then you could see from the outline there that eventually 221 00:14:34,690 --> 00:14:37,460 we'll-- we can perturb a matrix A. Of course, 222 00:14:37,460 --> 00:14:41,770 that's not going to be much harder than the identity. 223 00:14:41,770 --> 00:14:44,560 Let's do just one more check, and then 224 00:14:44,560 --> 00:14:45,865 show how it could be used. 225 00:14:49,630 --> 00:14:51,260 I multiply that by that. 226 00:14:51,260 --> 00:14:54,080 Can we do that-- 227 00:14:54,080 --> 00:14:57,680 this thing times this, and hope to get the identity? 228 00:14:57,680 --> 00:15:01,130 So first, I'll write down this thing. 229 00:15:01,130 --> 00:15:07,630 So I won't put equal, because I'm multiplying that by that. 230 00:15:07,630 --> 00:15:11,680 So I get In minus uv transpose. 231 00:15:14,620 --> 00:15:18,840 On the left side, of course, I'm getting the identity, 232 00:15:18,840 --> 00:15:21,860 and hoping I'm getting the identity on the right. 233 00:15:21,860 --> 00:15:23,780 So I'm multiplying by this. 234 00:15:23,780 --> 00:15:25,010 I did it there. 235 00:15:25,010 --> 00:15:30,440 Now I have to put I minus uv transposed. 236 00:15:30,440 --> 00:15:39,110 It takes faith here. u I minus v transpose u inverse v 237 00:15:39,110 --> 00:15:39,920 transpose-- 238 00:15:46,100 --> 00:15:46,600 gulp. 239 00:15:49,210 --> 00:15:52,380 Shall I just leave it there? 240 00:15:52,380 --> 00:15:56,860 Or you're-- had lunch, you're strong. 241 00:15:56,860 --> 00:15:58,930 Let's see what we can do. 242 00:15:58,930 --> 00:16:01,540 This part-- fine. 243 00:16:01,540 --> 00:16:05,470 This part-- oh, boy. 244 00:16:05,470 --> 00:16:08,010 What do I do here? 245 00:16:08,010 --> 00:16:17,120 That trick is going to be that this dopey-looking thing there 246 00:16:17,120 --> 00:16:20,460 can be written differently. 247 00:16:20,460 --> 00:16:22,590 Well, just tell me what it equals. 248 00:16:22,590 --> 00:16:29,520 It equals u minus uv transpose u. 249 00:16:29,520 --> 00:16:32,740 Everybody sees that. 250 00:16:32,740 --> 00:16:36,050 But now what am I going to do? 251 00:16:36,050 --> 00:16:43,640 I'm going to put the parentheses differently. 252 00:16:43,640 --> 00:16:46,670 I'm bringing u, the u that was on the right up there-- 253 00:16:46,670 --> 00:16:49,020 I'm going to put it on the left. 254 00:16:49,020 --> 00:16:50,610 You see that? 255 00:16:50,610 --> 00:16:55,330 So this is obviously u minus uv transpose u. 256 00:16:55,330 --> 00:16:58,410 And now I'm going to write it another time. 257 00:16:58,410 --> 00:17:03,490 It's u times I minus v transpose u. 258 00:17:03,490 --> 00:17:06,300 I'm going to factor the u out on the left side 259 00:17:06,300 --> 00:17:11,780 instead of where it originally came on the right side of this. 260 00:17:11,780 --> 00:17:14,640 You see the great news there? 261 00:17:14,640 --> 00:17:19,619 This was actually well-organized by your professor-- 262 00:17:19,619 --> 00:17:21,540 by accident. 263 00:17:21,540 --> 00:17:24,390 So what do I do now? 264 00:17:24,390 --> 00:17:25,920 This thing-- look. 265 00:17:25,920 --> 00:17:30,900 I've got exactly here what I've got inverted there. 266 00:17:30,900 --> 00:17:36,600 So altogether, I have I minus uv transpose plus this u, 267 00:17:36,600 --> 00:17:39,070 this cancelled, times v transpose. 268 00:17:39,070 --> 00:17:45,650 So I get I. So it was just this little sleight of hand, 269 00:17:45,650 --> 00:17:49,150 where u came in on the right and came out on the left side. 270 00:17:52,880 --> 00:17:58,020 And there'd be a similar formula with A in there. 271 00:17:58,020 --> 00:18:01,170 So now, it's a fair question, how 272 00:18:01,170 --> 00:18:03,770 did anybody come up with these formulas? 273 00:18:03,770 --> 00:18:08,190 We're proving them correct just by checking that they 274 00:18:08,190 --> 00:18:11,820 give the identity matrix. 275 00:18:11,820 --> 00:18:15,780 We should really think how to find 276 00:18:15,780 --> 00:18:18,750 the formula in the first place. 277 00:18:18,750 --> 00:18:22,070 I could go back to this one. 278 00:18:22,070 --> 00:18:24,768 How could you find that formula in the first place? 279 00:18:24,768 --> 00:18:26,060 Well, here is one way to do it. 280 00:18:31,890 --> 00:18:36,780 This is then about "to discover the formula." 281 00:18:43,530 --> 00:18:56,130 And then, over here, let me put over here "to use the formula," 282 00:18:56,130 --> 00:18:57,860 while I think of it. 283 00:18:57,860 --> 00:19:02,020 And actually, I have two uses in mind. 284 00:19:02,020 --> 00:19:04,995 Use one would be to solve-- 285 00:19:12,110 --> 00:19:15,410 I'm just doing I. Suppose I want to-- 286 00:19:15,410 --> 00:19:20,750 so this is my new matrix uv transpose x 287 00:19:20,750 --> 00:19:25,120 equals some right-hand side b. 288 00:19:25,120 --> 00:19:31,150 Solve a linear system that has that coefficient matrix. 289 00:19:31,150 --> 00:19:44,790 The use two would be, in least squares, a new measurement 290 00:19:44,790 --> 00:19:48,760 or observation or data point in these squares. 291 00:19:56,460 --> 00:19:59,250 So what do I mean by this? 292 00:19:59,250 --> 00:20:05,250 The old problem was Ax equal b. 293 00:20:09,690 --> 00:20:13,150 And I'm thinking, when I'm talking about least squares 294 00:20:13,150 --> 00:20:17,830 here, I'm imagining that A is rectangular. 295 00:20:17,830 --> 00:20:19,620 Too many equations-- 296 00:20:19,620 --> 00:20:22,330 A is a tall, thin matrix, just the kind 297 00:20:22,330 --> 00:20:23,890 we've been talking about. 298 00:20:23,890 --> 00:20:27,190 So that equation becomes-- 299 00:20:27,190 --> 00:20:32,720 of course, we know we go to the normal equations-- 300 00:20:32,720 --> 00:20:39,200 A transpose Ax, the good x, is A transpose b. 301 00:20:42,840 --> 00:20:44,928 Now we get a new measurement. 302 00:20:48,280 --> 00:20:50,710 So a new measurement, how does a new measurement 303 00:20:50,710 --> 00:20:52,150 change the problem? 304 00:20:52,150 --> 00:20:55,210 So this is the old problem before the measurement 305 00:20:55,210 --> 00:20:56,470 comes in. 306 00:20:56,470 --> 00:20:58,060 Now a new measurement arrives. 307 00:20:58,060 --> 00:21:02,610 So that's another b, a b M plus 1. 308 00:21:02,610 --> 00:21:05,800 And we get another equation. 309 00:21:05,800 --> 00:21:07,360 We get another equation. 310 00:21:07,360 --> 00:21:10,150 That's the new measurement, new point 311 00:21:10,150 --> 00:21:14,360 to get this straight line it's trying to stay close to. 312 00:21:14,360 --> 00:21:17,500 So we'll call this equation-- 313 00:21:17,500 --> 00:21:19,000 I don't know what we should call it. 314 00:21:19,000 --> 00:21:23,440 Maybe it'll be one more row. 315 00:21:23,440 --> 00:21:29,510 So now, old now is becoming new. 316 00:21:29,510 --> 00:21:33,440 I'm sort of more excited about this than proving the formula. 317 00:21:33,440 --> 00:21:35,840 So I'll just keep going. 318 00:21:35,840 --> 00:21:42,470 So there is our M measurements, M being large, 319 00:21:42,470 --> 00:21:44,927 A being a tall, thin matrix. 320 00:21:44,927 --> 00:21:46,760 And now we're going to give it one more row. 321 00:21:46,760 --> 00:21:48,560 We're going to make it even taller, 322 00:21:48,560 --> 00:21:57,320 say maybe, v transpose times this same x is some b 323 00:21:57,320 --> 00:22:00,980 new, or something like that. 324 00:22:00,980 --> 00:22:04,970 So there's one more row, one more measurement. 325 00:22:04,970 --> 00:22:09,192 So what happens to the normal equation? 326 00:22:09,192 --> 00:22:11,640 This makes it even more likely. 327 00:22:11,640 --> 00:22:14,630 There's another point, hoping that a straight line will 328 00:22:14,630 --> 00:22:15,230 go through. 329 00:22:15,230 --> 00:22:16,960 But if we give it one more point, 330 00:22:16,960 --> 00:22:21,500 there is even less chance of a straight line going exactly 331 00:22:21,500 --> 00:22:22,310 through. 332 00:22:22,310 --> 00:22:24,680 But still, so what do I-- 333 00:22:24,680 --> 00:22:26,420 this is my new matrix. 334 00:22:26,420 --> 00:22:29,080 So what's my new normal equation? 335 00:22:29,080 --> 00:22:41,110 The new normal equation, the A, or transpose-- 336 00:22:41,110 --> 00:22:44,420 the A has got a new row. 337 00:22:44,420 --> 00:22:46,280 A has got a new row. 338 00:22:46,280 --> 00:22:48,590 So a transpose will have a new column. 339 00:22:48,590 --> 00:22:51,020 I'm just copying the normal equation, 340 00:22:51,020 --> 00:22:54,030 but I'm giving it its new thing. 341 00:22:54,030 --> 00:22:56,000 It's got a new column. 342 00:22:56,000 --> 00:23:01,520 That is my A with a new row. 343 00:23:01,520 --> 00:23:09,320 This is my x hat, my least squares answer, new. 344 00:23:09,320 --> 00:23:13,400 And A transpose is this. 345 00:23:13,400 --> 00:23:22,280 A transpose with a new, now, column times b, so b and b new. 346 00:23:26,660 --> 00:23:28,930 Pretty OK with this. 347 00:23:28,930 --> 00:23:31,580 Do you see that this is the new normal equation? 348 00:23:31,580 --> 00:23:35,210 I'm using the new matrix and the new right-hand side. 349 00:23:38,030 --> 00:23:42,146 And just one more data point has come 350 00:23:42,146 --> 00:23:47,650 into the system from the sensor. 351 00:23:47,650 --> 00:23:49,640 And what's the key point here? 352 00:23:49,640 --> 00:23:54,880 The key point is I don't want to recompute. 353 00:23:54,880 --> 00:23:59,170 I don't want to multiply that matrix again. 354 00:23:59,170 --> 00:24:02,080 I don't want to start over. 355 00:24:02,080 --> 00:24:08,650 I don't want to compute this A transpose times that A. I want 356 00:24:08,650 --> 00:24:09,910 to use what I've already done. 357 00:24:13,330 --> 00:24:15,400 If I multiply those two together, 358 00:24:15,400 --> 00:24:16,510 what do I actually get? 359 00:24:19,280 --> 00:24:26,900 So this is, I'm adding a new column here and a new row here. 360 00:24:26,900 --> 00:24:28,850 Tell me what you think the answer is, and then 361 00:24:28,850 --> 00:24:30,290 let's just see why. 362 00:24:30,290 --> 00:24:32,650 I'm asking you what that matrix is. 363 00:24:35,947 --> 00:24:39,250 What do you think? 364 00:24:39,250 --> 00:24:42,940 Start me out, anyway. 365 00:24:42,940 --> 00:24:46,450 I'm just asking for ordinary matrix multiplication. 366 00:24:46,450 --> 00:24:49,880 Well, I guess I'm asking you to do it 367 00:24:49,880 --> 00:24:57,220 columns times rows since that's what 18 and 6.5 specializes in. 368 00:24:57,220 --> 00:25:04,120 So I think I have that times that, A transpose A, 369 00:25:04,120 --> 00:25:07,540 plus one new column times one new row. 370 00:25:07,540 --> 00:25:15,520 vv transpose is multiplying this x hat new. 371 00:25:15,520 --> 00:25:18,190 And over on the right side, I get whatever I had, 372 00:25:18,190 --> 00:25:25,280 the A transpose b old and the v b new. 373 00:25:25,280 --> 00:25:29,320 But let me come back to that, because that really shows you 374 00:25:29,320 --> 00:25:33,250 why you must understand matrix multiplication, 375 00:25:33,250 --> 00:25:36,930 both the usual row times column-- 376 00:25:36,930 --> 00:25:41,200 but that would not be so attractive here-- 377 00:25:41,200 --> 00:25:47,080 and also the new way as columns times rows. 378 00:25:47,080 --> 00:25:50,200 Because when I see it as columns times rows, 379 00:25:50,200 --> 00:25:54,160 I see that I have the same columns and same rows there. 380 00:25:54,160 --> 00:25:56,740 So that's just what I already knew. 381 00:25:56,740 --> 00:25:59,470 And then I have one new column times one new row. 382 00:25:59,470 --> 00:26:03,850 Of course, that's column n plus 1, and that's a row n plus 1. 383 00:26:03,850 --> 00:26:06,580 And they give that rank 1. 384 00:26:06,580 --> 00:26:10,890 It's a rank 1 change in A transpose A. It's a rank 1 385 00:26:10,890 --> 00:26:13,280 change in A transpose A. 386 00:26:13,280 --> 00:26:18,140 So this is like part of least squares. 387 00:26:18,140 --> 00:26:24,050 I mean, you can see the relevance in a real problem. 388 00:26:24,050 --> 00:26:27,940 You maybe have a missile flying along. 389 00:26:27,940 --> 00:26:31,210 You've sent up a satellite, GPS satellite. 390 00:26:31,210 --> 00:26:32,690 You're tracking it. 391 00:26:32,690 --> 00:26:34,790 More data comes in. 392 00:26:34,790 --> 00:26:39,170 The data is just one more position. 393 00:26:39,170 --> 00:26:41,490 The tracker isn't perfect. 394 00:26:41,490 --> 00:26:42,590 So we're going to fit-- 395 00:26:42,590 --> 00:26:47,770 well, here I'm fitting a straight line, maybe. 396 00:26:47,770 --> 00:26:51,410 But we're fitting to the data. 397 00:26:51,410 --> 00:26:57,160 And the only change in the left-hand, the big problem, 398 00:26:57,160 --> 00:27:01,390 the big part of the computation is the A transpose A part. 399 00:27:01,390 --> 00:27:06,740 And it's just changed by rank 1, or by rank k, 400 00:27:06,740 --> 00:27:10,190 if we had k new data points. 401 00:27:10,190 --> 00:27:13,010 If we had k new data points, then this 402 00:27:13,010 --> 00:27:16,040 would be a rank k matrix. 403 00:27:16,040 --> 00:27:19,840 So you see, then I go back here-- 404 00:27:19,840 --> 00:27:20,990 well, OK. 405 00:27:20,990 --> 00:27:23,150 Now I'm perturbing A transpose A. 406 00:27:23,150 --> 00:27:28,370 And I haven't given a formula for that one yet. 407 00:27:28,370 --> 00:27:30,950 I've only perturbed up to, now, the identity. 408 00:27:30,950 --> 00:27:34,370 But you can believe, since all these formulas are working, 409 00:27:34,370 --> 00:27:37,520 that Sherman or Morrison or Woodbury 410 00:27:37,520 --> 00:27:44,130 came up with the correct perturbation for A transpose A. 411 00:27:44,130 --> 00:27:51,560 So I'm sort of happy that, that application, so natural, 412 00:27:51,560 --> 00:27:55,565 came out so simply. 413 00:27:58,440 --> 00:28:02,090 While I'm on that sort of subject, 414 00:28:02,090 --> 00:28:05,700 have you ever heard of the Kalman filter? 415 00:28:05,700 --> 00:28:11,010 So that Kalman filter, what's that about? 416 00:28:11,010 --> 00:28:13,430 It's about exactly this. 417 00:28:13,430 --> 00:28:15,590 It's about dynamic least squares. 418 00:28:15,590 --> 00:28:17,270 It's about least squares problems 419 00:28:17,270 --> 00:28:20,130 in which new data is coming in. 420 00:28:20,130 --> 00:28:22,900 That's what the Kalman filter-- or in other words, 421 00:28:22,900 --> 00:28:26,170 let me just write a couple of words up here. 422 00:28:26,170 --> 00:28:31,210 This is really recursive least squares. 423 00:28:31,210 --> 00:28:36,700 Recursive least squares-- what I mean by recursive 424 00:28:36,700 --> 00:28:39,600 is new data comes in. 425 00:28:39,600 --> 00:28:45,650 It changes the answer, but it doesn't change our method. 426 00:28:45,650 --> 00:28:56,120 And then the Kalman filter is a very, very big deal 427 00:28:56,120 --> 00:28:57,585 in guidance. 428 00:28:57,585 --> 00:29:05,230 If you're sending up a missile, a satellite, you track it. 429 00:29:05,230 --> 00:29:09,880 You do just what I've been discussing here. 430 00:29:09,880 --> 00:29:12,860 But the Kalman filter is-- 431 00:29:12,860 --> 00:29:16,630 it's got more possibilities built in than this one. 432 00:29:16,630 --> 00:29:20,980 This is the simplest update possible. 433 00:29:20,980 --> 00:29:25,800 And it would go in this category. 434 00:29:25,800 --> 00:29:31,810 Kalman went beyond a standard update. 435 00:29:31,810 --> 00:29:33,620 How did he go beyond? 436 00:29:33,620 --> 00:29:36,030 Let's see. 437 00:29:36,030 --> 00:29:38,500 If I've used the words Kalman filter, 438 00:29:38,500 --> 00:29:42,730 I should tell you what it is that Kalman does 439 00:29:42,730 --> 00:29:45,460 and what is that least squares problem. 440 00:29:45,460 --> 00:29:49,150 It's just part of general knowledge, it seems to me. 441 00:29:52,340 --> 00:29:54,560 So what is it, more gen-- 442 00:29:54,560 --> 00:29:56,610 what are the additional pieces? 443 00:29:56,610 --> 00:30:03,010 You've seen the main idea here of getting a simple recursive 444 00:30:03,010 --> 00:30:06,130 step that doesn't require recomputing all 445 00:30:06,130 --> 00:30:08,110 that you did before. 446 00:30:08,110 --> 00:30:13,780 And of course, to this inverse, I'm 447 00:30:13,780 --> 00:30:16,190 going to apply the Sherman-Morrison-Woodbury 448 00:30:16,190 --> 00:30:17,140 formula. 449 00:30:17,140 --> 00:30:20,730 So I'll use the inverse that I had before. 450 00:30:20,730 --> 00:30:23,340 And this will be a rank 1-- 451 00:30:23,340 --> 00:30:27,270 this is a rank 1 perturbation of that. 452 00:30:27,270 --> 00:30:30,270 I'm looking here at A transpose A. Over there, 453 00:30:30,270 --> 00:30:33,330 I was looking at, I just called it A. 454 00:30:33,330 --> 00:30:35,850 Or I even called it the identity matrix. 455 00:30:35,850 --> 00:30:38,490 But it's whatever matrix you have here 456 00:30:38,490 --> 00:30:40,620 with a rank 1 perturbation. 457 00:30:40,620 --> 00:30:44,220 And the whole thing has to be inverted. 458 00:30:44,220 --> 00:30:48,480 And so I was going to say what is additional, 459 00:30:48,480 --> 00:30:51,970 just so you know about Kalman filters. 460 00:30:51,970 --> 00:30:53,880 So two things are additional. 461 00:30:53,880 --> 00:30:59,320 The point is Kalman filters are for dynamic least squares. 462 00:30:59,320 --> 00:31:02,080 I would say dynamic squares. 463 00:31:02,080 --> 00:31:07,430 And so there are two ingredients that you haven't seen here. 464 00:31:07,430 --> 00:31:13,770 One ingredient is the idea of using 465 00:31:13,770 --> 00:31:16,980 the covariance matrix, which tells you 466 00:31:16,980 --> 00:31:20,010 how errors are correlated. 467 00:31:20,010 --> 00:31:23,010 So that would be weighted least squares, 468 00:31:23,010 --> 00:31:25,485 or correlated least squares. 469 00:31:29,040 --> 00:31:36,210 So these squares-- let me just remind you, 470 00:31:36,210 --> 00:31:37,930 if I can write it here. 471 00:31:37,930 --> 00:31:42,325 So least squares-- standard. 472 00:31:48,120 --> 00:31:53,970 Standard meaning that data is not correlated. 473 00:31:53,970 --> 00:31:57,900 It all has the same variance. 474 00:31:57,900 --> 00:32:00,750 These are the statistics words that I'm 475 00:32:00,750 --> 00:32:08,770 using last time and this, but that I'll talk about properly. 476 00:32:08,770 --> 00:32:12,370 So the standard one is the covari-- 477 00:32:12,370 --> 00:32:20,530 the standard means covariance equal the identity matrix. 478 00:32:23,890 --> 00:32:28,090 You're doing Gaussian normal probability 479 00:32:28,090 --> 00:32:35,300 but with just standard Gaussians, standard Gaussians. 480 00:32:35,300 --> 00:32:42,370 So that's one aspect, which in the work of the Draper lab, 481 00:32:42,370 --> 00:32:47,320 let's say, they have to think, OK, 482 00:32:47,320 --> 00:32:50,500 they're getting sensors, different kinds of sensors, 483 00:32:50,500 --> 00:32:53,680 with different accuracies, different reliability. 484 00:32:53,680 --> 00:32:57,980 So they have to take account of the covariance matrix. 485 00:32:57,980 --> 00:33:00,920 Then the other point is-- 486 00:33:00,920 --> 00:33:04,130 and also-- so that was point one. 487 00:33:04,130 --> 00:33:07,055 Point two is the dynamic part. 488 00:33:11,050 --> 00:33:12,530 There is a state equation. 489 00:33:15,920 --> 00:33:19,290 So I'm really into the edge of control theory. 490 00:33:19,290 --> 00:33:22,970 So let me just use some words here that you've seen, 491 00:33:22,970 --> 00:33:23,960 if you've-- 492 00:33:23,960 --> 00:33:28,550 so control theory has state equation. 493 00:33:28,550 --> 00:33:29,960 What's a state equation? 494 00:33:29,960 --> 00:33:31,610 What is the state? 495 00:33:31,610 --> 00:33:35,900 This is the position, in my example, 496 00:33:35,900 --> 00:33:38,526 is the position of the satellite. 497 00:33:43,880 --> 00:33:50,580 So are we looking for a fixed satellite? 498 00:33:50,580 --> 00:33:51,450 Certainly not. 499 00:33:51,450 --> 00:33:53,340 The satellite is moving. 500 00:33:53,340 --> 00:33:59,700 So this state equation tells me how much the satellite-- 501 00:33:59,700 --> 00:34:04,080 it's Newton's law-- tells me where the satellite should be. 502 00:34:04,080 --> 00:34:06,090 Where am I looking? 503 00:34:06,090 --> 00:34:10,469 And then the least squares tells me, it says, 504 00:34:10,469 --> 00:34:15,929 look around that sort of median position 505 00:34:15,929 --> 00:34:21,239 for the actual position that the data is giving. 506 00:34:21,239 --> 00:34:24,750 So just to say-- 507 00:34:24,750 --> 00:34:28,230 let me-- let me summarize this. 508 00:34:28,230 --> 00:34:34,040 The Kalman filter is a significantly improved version 509 00:34:34,040 --> 00:34:35,929 of recursive least squares. 510 00:34:35,929 --> 00:34:38,270 That's recursive least squares. 511 00:34:38,270 --> 00:34:43,679 New measurement comes in, changes things 512 00:34:43,679 --> 00:34:48,270 but leaves a big part unchanged. 513 00:34:48,270 --> 00:34:54,510 And you find that new x hat. 514 00:34:54,510 --> 00:35:02,520 With a Kalman filter, there's an covariance matrix in the middle 515 00:35:02,520 --> 00:35:03,360 here. 516 00:35:03,360 --> 00:35:05,980 That's where covariance matrices go. 517 00:35:05,980 --> 00:35:11,520 That's matrix covariance, or inverse covariance, times that. 518 00:35:11,520 --> 00:35:19,060 And oh, why don't we see the covariances at the right time? 519 00:35:19,060 --> 00:35:22,650 So maybe those minutes that I've occupied 520 00:35:22,650 --> 00:35:32,500 were just really to get you to hear that name for the simplest 521 00:35:32,500 --> 00:35:33,580 update. 522 00:35:33,580 --> 00:35:38,510 And Kalman's name for a more general update. 523 00:35:38,510 --> 00:35:40,680 Done. 524 00:35:40,680 --> 00:35:43,530 Oh, this is also to be done. 525 00:35:43,530 --> 00:35:45,810 Where is-- yeah. 526 00:35:45,810 --> 00:35:46,930 No-- up at the top. 527 00:35:49,750 --> 00:35:53,800 I seem to be into the applications here. 528 00:35:53,800 --> 00:35:58,270 I promised how to discover the formula. 529 00:35:58,270 --> 00:35:59,320 Maybe I'll never know. 530 00:36:03,070 --> 00:36:05,530 Because I'm really more interested in, 531 00:36:05,530 --> 00:36:06,650 what's it good for? 532 00:36:11,200 --> 00:36:16,400 Use number one, now, I'm backing up to the easy, easy question. 533 00:36:16,400 --> 00:36:23,645 Suppose I had-- and let me even change the matrix to A here. 534 00:36:23,645 --> 00:36:27,680 It makes it more realistic. 535 00:36:27,680 --> 00:36:30,350 I'm going to copy that here, fully 536 00:36:30,350 --> 00:36:34,040 on discovering the formula. 537 00:36:34,040 --> 00:36:37,880 For the moment, let's just use it. 538 00:36:37,880 --> 00:36:56,360 So I suppose that Au is b is solved for u. 539 00:36:56,360 --> 00:37:04,075 Now, now solve A plus-- 540 00:37:06,710 --> 00:37:14,470 or minus a rank-- let me do the rank 1, A minus uv transpose b. 541 00:37:18,230 --> 00:37:20,990 What's-- x. 542 00:37:20,990 --> 00:37:24,080 This is the problem that I really would like to solve. 543 00:37:26,590 --> 00:37:28,360 Let me just be sure I'm doing this right. 544 00:37:46,900 --> 00:37:51,190 It's similar to what we had in the Kalman situation. 545 00:37:51,190 --> 00:37:54,490 Suppose I've solved one problem. 546 00:37:54,490 --> 00:37:59,500 But now I perturb the matrix by rank 1. 547 00:37:59,500 --> 00:38:02,760 So I have a new problem with a new answer. 548 00:38:02,760 --> 00:38:04,550 And I want to get that answer quickly. 549 00:38:04,550 --> 00:38:05,050 Yeah? 550 00:38:05,050 --> 00:38:07,300 AUDIENCE: Are the u's related in those two lines? 551 00:38:07,300 --> 00:38:08,650 GILBERT STRANG: Oh, no. 552 00:38:08,650 --> 00:38:10,000 Thank you. 553 00:38:10,000 --> 00:38:11,560 Thank you very much. 554 00:38:11,560 --> 00:38:18,175 Let's call this guy z, or w, maybe w. 555 00:38:22,040 --> 00:38:23,180 So thank you. 556 00:38:23,180 --> 00:38:24,620 That's great. 557 00:38:24,620 --> 00:38:27,500 So that's what I've solved for w. 558 00:38:30,280 --> 00:38:33,520 In other words, I have found A inverse b. 559 00:38:33,520 --> 00:38:38,380 And I want to find the answer to that new question. 560 00:38:38,380 --> 00:38:41,310 So I've perturbed the matrix. 561 00:38:41,310 --> 00:38:44,290 It's the coefficient matrix in a linear system. 562 00:38:44,290 --> 00:38:48,030 And I just want to solve that linear system. 563 00:38:48,030 --> 00:38:52,080 Now, so if I didn't know anything about the formulas, 564 00:38:52,080 --> 00:38:55,540 I would have a new matrix here. 565 00:38:55,540 --> 00:39:05,190 It would take n cube steps to do elimination and get 566 00:39:05,190 --> 00:39:07,920 the new answer. 567 00:39:07,920 --> 00:39:12,820 But the point is to use the old answer. 568 00:39:12,820 --> 00:39:15,090 The point is to use the old answer. 569 00:39:15,090 --> 00:39:17,680 And now let me just say what this is. 570 00:39:17,680 --> 00:39:22,030 And it's problem three in this section. 571 00:39:30,040 --> 00:39:40,730 So in other words, we know about A. We've solved that one. 572 00:39:40,730 --> 00:39:44,607 So what I'm going to do, instead of solving a whole new problem, 573 00:39:44,607 --> 00:39:45,190 I'm going to-- 574 00:39:47,780 --> 00:39:49,510 so quickly is the idea. 575 00:39:52,950 --> 00:39:55,720 And here's the idea. 576 00:39:55,720 --> 00:39:56,650 I've solved that one. 577 00:39:56,650 --> 00:39:59,560 And I'm going to solve a second problem, 578 00:39:59,560 --> 00:40:06,190 also solve, A with the same matrix A-- 579 00:40:06,190 --> 00:40:10,210 oops-- A times what shall I call the unknown? 580 00:40:10,210 --> 00:40:12,940 There's z equal u. 581 00:40:17,060 --> 00:40:18,880 So this is my problem. 582 00:40:18,880 --> 00:40:21,190 I know the u and v transpose. 583 00:40:21,190 --> 00:40:25,360 But I don't really want to find the inverse of this matrix 584 00:40:25,360 --> 00:40:27,130 from scratch. 585 00:40:27,130 --> 00:40:29,920 So the idea is that if I suitably 586 00:40:29,920 --> 00:40:33,790 combine the solutions to the original problem, 587 00:40:33,790 --> 00:40:37,120 the original solution w, and the solution 588 00:40:37,120 --> 00:40:43,240 to this problem with the new guy u there, that somehow, 589 00:40:43,240 --> 00:40:49,200 by combining the w and the z, I'm going to get this answer x. 590 00:40:49,200 --> 00:40:51,130 That's where I'm headed. 591 00:40:51,130 --> 00:40:56,410 That's where I'm headed, that by figuring out w and z-- so 592 00:40:56,410 --> 00:40:59,030 do you see what I've done? 593 00:40:59,030 --> 00:41:01,355 With the matrix A, I've solved two problems. 594 00:41:04,820 --> 00:41:07,210 Does that take twice as long? 595 00:41:07,210 --> 00:41:13,695 If I have the same matrix A but different right-hand sides, b 596 00:41:13,695 --> 00:41:18,170 and u, if I factor A into Lu, all the hard work 597 00:41:18,170 --> 00:41:21,570 is done there on the left side. 598 00:41:21,570 --> 00:41:24,200 All the work is done in finding Lu. 599 00:41:24,200 --> 00:41:29,440 And then I just back substitute to find the second solution. 600 00:41:29,440 --> 00:41:33,315 This is so fundamental that I'm emphasizing it, 601 00:41:33,315 --> 00:41:36,800 that if you have multiple right-hand sides, 602 00:41:36,800 --> 00:41:40,580 you don't every time go back and work on the left side. 603 00:41:40,580 --> 00:41:43,670 The left side with the same matrix A, 604 00:41:43,670 --> 00:41:45,560 just would do the same stuff. 605 00:41:45,560 --> 00:41:49,070 The same rows and pivots and all that stuff 606 00:41:49,070 --> 00:41:53,620 will happen because it's the same A. You just attach-- 607 00:41:53,620 --> 00:41:57,270 you really just stick a second column-- 608 00:41:57,270 --> 00:42:03,980 really, the fast way to do it is A wz. 609 00:42:03,980 --> 00:42:10,280 The two solutions come from the b and the u. 610 00:42:10,280 --> 00:42:12,920 I've just put the two problems together 611 00:42:12,920 --> 00:42:18,290 to emphasize that the matrix A is the same. 612 00:42:18,290 --> 00:42:20,450 This is where most of the work goes. 613 00:42:20,450 --> 00:42:22,630 But it only goes there once. 614 00:42:22,630 --> 00:42:24,260 That's the point. 615 00:42:24,260 --> 00:42:28,060 Then finally, I'm supposed to-- 616 00:42:28,060 --> 00:42:30,160 the notes are supposed to tell me 617 00:42:30,160 --> 00:42:37,040 how to combine the two answers w and z to get the answer x. 618 00:42:37,040 --> 00:42:39,160 So let me write that down. 619 00:42:39,160 --> 00:42:47,930 And that will be use number one of 620 00:42:47,930 --> 00:42:51,140 the Sherman-Morrison-Woodbury formula. 621 00:42:51,140 --> 00:42:53,570 So I'm planning to just write it down. 622 00:42:53,570 --> 00:43:00,620 According to this, step two is the answer we want, 623 00:43:00,620 --> 00:43:02,120 here written x-- 624 00:43:02,120 --> 00:43:05,600 so I haven't used the same letters as in the notes-- 625 00:43:05,600 --> 00:43:08,660 is the original w-- 626 00:43:08,660 --> 00:43:12,590 good-- and then a change. 627 00:43:12,590 --> 00:43:16,490 Because I've changed the problem. 628 00:43:16,490 --> 00:43:18,770 So I'm going to change the answer. 629 00:43:18,770 --> 00:43:23,240 But the thing is, to get this answer, 630 00:43:23,240 --> 00:43:29,060 I have to divide by that determinant wherever it was, 631 00:43:29,060 --> 00:43:30,050 the-- 632 00:43:30,050 --> 00:43:34,826 oh, yeah, the-- 633 00:43:34,826 --> 00:43:35,840 I'm sorry. 634 00:43:35,840 --> 00:43:40,040 I'm using an A here, and I haven't given you the formula 635 00:43:40,040 --> 00:43:46,360 for A. So that'll have to wait. 636 00:43:49,250 --> 00:43:52,240 So I'll put determinant here. 637 00:43:52,240 --> 00:43:56,150 And Sherman, Morrison, Woodbury figured that out. 638 00:43:56,150 --> 00:43:58,150 And then-- oh, I'm sorry. 639 00:44:03,380 --> 00:44:05,210 We know what that is. 640 00:44:05,210 --> 00:44:10,360 We want 1 minus v transpose z. 641 00:44:10,360 --> 00:44:15,130 So it's the v from here, 1 minus v transpose. 642 00:44:15,130 --> 00:44:17,290 And it's the z there. 643 00:44:17,290 --> 00:44:18,610 Oh, yeah. 644 00:44:18,610 --> 00:44:20,740 This is good. 645 00:44:20,740 --> 00:44:25,270 So I'm going to get a formula that changes the solution 646 00:44:25,270 --> 00:44:30,880 w by a term that we recognize is coming from Sherman, Morrison, 647 00:44:30,880 --> 00:44:31,510 Woodbury. 648 00:44:31,510 --> 00:44:36,480 And that term in the numerator is a multiple of zx. 649 00:44:44,090 --> 00:44:47,790 So I'm using w-- 650 00:44:47,790 --> 00:44:50,746 yes-- w times-- 651 00:44:54,950 --> 00:45:01,830 sorry-- times x, which was-- 652 00:45:01,830 --> 00:45:02,820 I'm sorry. 653 00:45:02,820 --> 00:45:05,870 w is xv. 654 00:45:05,870 --> 00:45:08,620 So the v here, I'm just using-- 655 00:45:08,620 --> 00:45:15,930 the v is the same v. So v transpose z, 656 00:45:15,930 --> 00:45:18,030 I think that's got it. 657 00:45:18,030 --> 00:45:19,390 I think that's got it. 658 00:45:19,390 --> 00:45:23,990 So again, the point was to-- 659 00:45:23,990 --> 00:45:27,380 x doesn't appear here, because that's what I'm after-- 660 00:45:27,380 --> 00:45:31,820 is to use the w and the z, the w and the z. 661 00:45:31,820 --> 00:45:36,740 And I have to use, of course, the v. And I have to use the u. 662 00:45:36,740 --> 00:45:40,020 And that got used here. 663 00:45:40,020 --> 00:45:43,070 So everything that had to be used got used. 664 00:45:43,070 --> 00:45:44,840 And that was the answer. 665 00:45:47,420 --> 00:45:51,860 So that's the basic use, is, if I perturb 666 00:45:51,860 --> 00:45:57,500 the problem, the left-hand side, what's 667 00:45:57,500 --> 00:45:59,750 the change in the solution? 668 00:45:59,750 --> 00:46:03,290 Over here with least squares, it was the same. 669 00:46:03,290 --> 00:46:05,460 I perturbed the left-hand side. 670 00:46:05,460 --> 00:46:07,070 But because it was least squares, 671 00:46:07,070 --> 00:46:10,200 there was an A transpose A to deal with. 672 00:46:10,200 --> 00:46:13,130 So this was one level more difficult, 673 00:46:13,130 --> 00:46:15,500 because it involved A transpose A's. 674 00:46:15,500 --> 00:46:21,220 And here, it was just straightforward A. So, yes? 675 00:46:21,220 --> 00:46:21,720 Thanks. 676 00:46:21,720 --> 00:46:23,387 AUDIENCE: Would it be also a good idea-- 677 00:46:23,387 --> 00:46:29,040 so you have A minus uv transpose x plus b-- to write A minus Az 678 00:46:29,040 --> 00:46:30,992 b transpose x equals b? 679 00:46:30,992 --> 00:46:32,470 I'm backtracking on the left. 680 00:46:32,470 --> 00:46:34,250 GILBERT STRANG: I could probably do this other ways. 681 00:46:34,250 --> 00:46:34,650 Yeah. 682 00:46:34,650 --> 00:46:35,080 Yeah. 683 00:46:35,080 --> 00:46:35,580 Yeah. 684 00:46:35,580 --> 00:46:40,090 I hope you follow up on that and write it down. 685 00:46:42,610 --> 00:46:48,640 So maybe what I've done is what my goal was for today, first, 686 00:46:48,640 --> 00:46:53,790 to show you the formula for the inverse; second, 687 00:46:53,790 --> 00:46:58,470 to recognize that it has interesting importance 688 00:46:58,470 --> 00:47:01,260 that a rank k change in the matrix 689 00:47:01,260 --> 00:47:05,910 brings a rank k change in the inverse, and that we can-- 690 00:47:05,910 --> 00:47:09,210 the formula says to invert an n by n matrix, 691 00:47:09,210 --> 00:47:12,480 you can switch an inverted k by k matrix. 692 00:47:12,480 --> 00:47:15,610 And then I spoke about a couple of the applications. 693 00:47:15,610 --> 00:47:17,220 So the only thing I have not done 694 00:47:17,220 --> 00:47:21,150 is to give you the full Sherman-Morrison-Woodbury 695 00:47:21,150 --> 00:47:24,900 formula, the one with an A, the one that's up on the right. 696 00:47:24,900 --> 00:47:27,600 Can I do that finally? 697 00:47:27,600 --> 00:47:30,210 You can probably guess what it's going to be. 698 00:47:32,850 --> 00:47:37,710 Where is-- there's got to be one more blackboard. 699 00:47:37,710 --> 00:47:43,500 So here Sherman-Morrison-Woodbury. 700 00:47:43,500 --> 00:47:45,180 And what's the complete thing? 701 00:47:45,180 --> 00:47:49,050 It's A minus uv transpose inverse. 702 00:47:49,050 --> 00:47:52,390 So this is n by n. 703 00:47:52,390 --> 00:47:55,210 And it's an A now instead of an identity matrix. 704 00:47:55,210 --> 00:47:57,350 That's the difference. 705 00:47:57,350 --> 00:48:01,270 And this is n by k, k by n. 706 00:48:01,270 --> 00:48:04,150 So it's rank k perturbation. 707 00:48:04,150 --> 00:48:10,880 And start me out on the formula. 708 00:48:10,880 --> 00:48:13,260 What's the first thing I have to write? 709 00:48:13,260 --> 00:48:14,520 AUDIENCE: A inverse. 710 00:48:14,520 --> 00:48:16,450 GILBERT STRANG: A inverse. 711 00:48:16,450 --> 00:48:18,680 So I start with A inverse. 712 00:48:18,680 --> 00:48:20,770 And then I subtract something. 713 00:48:20,770 --> 00:48:26,380 And it's going to be a copy of this, except-- 714 00:48:26,380 --> 00:48:29,860 well, maybe it's just a perfect copy. 715 00:48:29,860 --> 00:48:33,280 Maybe I just need the u. 716 00:48:33,280 --> 00:48:35,170 I'm going from that left board. 717 00:48:35,170 --> 00:48:39,230 Now, Ik, what do I put it in there? 718 00:48:42,040 --> 00:48:44,980 I am going to have to look. 719 00:48:44,980 --> 00:48:51,300 So we're writing down now the formula, the full scale-- 720 00:48:51,300 --> 00:48:52,090 OK. 721 00:48:52,090 --> 00:48:52,840 Oh, all right. 722 00:48:58,300 --> 00:48:59,840 Here it is. 723 00:48:59,840 --> 00:49:03,800 There is an A inverse u. 724 00:49:03,800 --> 00:49:06,800 Now comes that whole thing inverted 725 00:49:06,800 --> 00:49:12,770 that you'll expect, I minus v transpose A inverse u, 726 00:49:12,770 --> 00:49:14,780 all inverse. 727 00:49:14,780 --> 00:49:19,560 And then there's another two pieces, v transpose A inverse. 728 00:49:22,140 --> 00:49:24,600 So I didn't look ahead to get it at all on one line. 729 00:49:24,600 --> 00:49:26,310 But do you see what-- 730 00:49:26,310 --> 00:49:30,020 this is the-- oh, that's probably-- is that-- 731 00:49:30,020 --> 00:49:30,520 yeah. 732 00:49:30,520 --> 00:49:32,460 That's the identity. 733 00:49:32,460 --> 00:49:35,220 Because we've got enough A inverses. 734 00:49:35,220 --> 00:49:39,900 I believe that's the final, ultimate formula of life 735 00:49:39,900 --> 00:49:45,850 here, that we've changed it by rank k. 736 00:49:45,850 --> 00:49:47,910 Here's the original inverse. 737 00:49:47,910 --> 00:49:51,325 And presumably, this is a rank k change. 738 00:49:54,140 --> 00:49:57,250 That will be a rank k change. 739 00:49:57,250 --> 00:50:03,100 And it only requires us to compute that inverse, where 740 00:50:03,100 --> 00:50:06,280 that's a k by k matrix. 741 00:50:06,280 --> 00:50:13,990 Thank you for allowing this, our 50 minutes of formulas. 742 00:50:13,990 --> 00:50:17,650 So that's really what that comes to. 743 00:50:17,650 --> 00:50:23,710 So that's section 4.1 of the notes with some applications. 744 00:50:23,710 --> 00:50:29,440 And we will move on to other circumstances of low rank. 745 00:50:29,440 --> 00:50:29,950 Good. 746 00:50:29,950 --> 00:50:31,820 Thank you.