1 00:00:01,550 --> 00:00:03,920 The following content is provided under a Creative 2 00:00:03,920 --> 00:00:05,310 Commons license. 3 00:00:05,310 --> 00:00:07,520 Your support will help MIT OpenCourseWare 4 00:00:07,520 --> 00:00:11,610 continue to offer high quality educational resources for free. 5 00:00:11,610 --> 00:00:14,180 To make a donation or to view additional materials 6 00:00:14,180 --> 00:00:18,140 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:18,140 --> 00:00:19,026 at ocw.mit.edu. 8 00:00:23,490 --> 00:00:26,130 GILBERT STRANG: Shall we start? 9 00:00:26,130 --> 00:00:30,780 Let me just say, this is a great adventure for me 10 00:00:30,780 --> 00:00:33,900 to be here all on my own, teaching a course that 11 00:00:33,900 --> 00:00:35,400 involves learning from data. 12 00:00:35,400 --> 00:00:39,450 So it's an exciting subject, and a lot of linear algebra 13 00:00:39,450 --> 00:00:40,290 goes into it. 14 00:00:40,290 --> 00:00:43,350 So it's a second course on linear algebra. 15 00:00:45,960 --> 00:00:52,900 Can I just-- so there is a Stellar site established, 16 00:00:52,900 --> 00:00:56,770 and that will be the basic thing that we use. 17 00:00:56,770 --> 00:00:59,470 This is a public site-- 18 00:00:59,470 --> 00:01:05,310 math.mit.edu/learningfromdata. 19 00:01:05,310 --> 00:01:11,630 So a book is coming pretty quickly as we speak, 20 00:01:11,630 --> 00:01:20,410 or after we speak, and that site has the table of contents 21 00:01:20,410 --> 00:01:23,140 of the book, which would give you an idea of what 22 00:01:23,140 --> 00:01:24,580 could be in the course. 23 00:01:24,580 --> 00:01:28,480 And I printed out a copy for everybody just of that one 24 00:01:28,480 --> 00:01:28,980 page. 25 00:01:28,980 --> 00:01:30,340 This is probably the final-- 26 00:01:30,340 --> 00:01:32,820 first and last handout-- 27 00:01:32,820 --> 00:01:35,780 maybe-- with a table of contents, 28 00:01:35,780 --> 00:01:37,930 which you'll see there. 29 00:01:37,930 --> 00:01:41,050 And also, you'll see there the first two sections 30 00:01:41,050 --> 00:01:44,820 of the book, which is what I'll talk about today 31 00:01:44,820 --> 00:01:47,830 and a little bit into Friday. 32 00:01:47,830 --> 00:01:51,400 So that's linear algebra, of course, because the course 33 00:01:51,400 --> 00:01:54,820 begins with linear algebra-- 34 00:01:54,820 --> 00:01:57,640 actually, things that you would know from 18.06, 35 00:01:57,640 --> 00:02:04,310 but this is a way to say this is really important stuff. 36 00:02:04,310 --> 00:02:06,820 So that's what I'll do today. 37 00:02:06,820 --> 00:02:11,410 I'd like to start on the linear algebra today. 38 00:02:13,960 --> 00:02:18,205 Here's a great fact about the course. 39 00:02:21,310 --> 00:02:23,830 So we taught it last year, several of us 40 00:02:23,830 --> 00:02:28,840 together, and we knew there wouldn't be a final exam, 41 00:02:28,840 --> 00:02:33,580 but we imagined there might be quizzes along the way. 42 00:02:33,580 --> 00:02:36,610 But then we couldn't think of anything to put on the quizzes. 43 00:02:36,610 --> 00:02:41,590 So we canceled those. 44 00:02:41,590 --> 00:02:43,930 But you do we learn a lot, nevertheless. 45 00:02:43,930 --> 00:02:48,410 And so I guess we base the grades on the homeworks. 46 00:02:48,410 --> 00:02:53,140 So the homeworks will be partly linear algebra questions 47 00:02:53,140 --> 00:02:58,990 and partly online, like recognizing handwriting, 48 00:02:58,990 --> 00:03:03,880 stitching images together, many other things. 49 00:03:03,880 --> 00:03:07,330 And I'll talk about those as we go. 50 00:03:07,330 --> 00:03:08,980 Good. 51 00:03:08,980 --> 00:03:11,710 So that's the general picture. 52 00:03:11,710 --> 00:03:15,310 And I'll say more about it today. 53 00:03:15,310 --> 00:03:17,140 And I could answer any question about it. 54 00:03:17,140 --> 00:03:20,740 So we're getting videotaped. 55 00:03:20,740 --> 00:03:25,990 So if anybody's bashful, sit in the far back. 56 00:03:25,990 --> 00:03:29,860 But it'll be fun. 57 00:03:29,860 --> 00:03:33,430 You may know the videos for 18.06. 58 00:03:33,430 --> 00:03:37,270 So this is the next step, 18.065. 59 00:03:37,270 --> 00:03:41,170 It's pretty exciting for me. 60 00:03:41,170 --> 00:03:46,210 So any question, or shall I do a little math? 61 00:03:46,210 --> 00:03:46,990 Why not? 62 00:03:46,990 --> 00:03:53,620 And then I'll say a little more about the course, 63 00:03:53,620 --> 00:03:57,610 just so you have an idea of what's ahead. 64 00:03:57,610 --> 00:04:00,910 And it looks like this room is exactly the right size to me, 65 00:04:00,910 --> 00:04:02,140 so I'm pleased. 66 00:04:08,500 --> 00:04:10,210 So what's the deal in linear algebra? 67 00:04:13,330 --> 00:04:15,970 Forgive me if I start with what I 68 00:04:15,970 --> 00:04:18,730 do the very first day in 18.06, which is 69 00:04:18,730 --> 00:04:22,770 multiply a matrix by a vector. 70 00:04:22,770 --> 00:04:25,290 And then I'll graduate to multiplying a matrix 71 00:04:25,290 --> 00:04:27,410 by a matrix. 72 00:04:27,410 --> 00:04:29,640 And you will say, I know that stuff. 73 00:04:29,640 --> 00:04:31,500 But do you know it the right way? 74 00:04:31,500 --> 00:04:33,660 Do you think of the multiplication the right way? 75 00:04:33,660 --> 00:04:37,320 So let me tell you what I believe to be the right way. 76 00:04:37,320 --> 00:04:56,190 So let me take a matrix, say, 2, 3, 5, 1, 1, 7, and 3, 4, 12. 77 00:04:56,190 --> 00:05:02,470 And I'll always call matrices A. 78 00:05:02,470 --> 00:05:07,750 So first step is just A times x, A times a vector. 79 00:05:07,750 --> 00:05:12,640 So I multiply A by, let's say, x1, x2, x3. 80 00:05:12,640 --> 00:05:17,310 And how do I look at that answer? 81 00:05:17,310 --> 00:05:22,020 So the choices-- think of the rows of the matrix 82 00:05:22,020 --> 00:05:24,440 or think of the columns. 83 00:05:24,440 --> 00:05:27,110 And if you think of the rows, which 84 00:05:27,110 --> 00:05:32,690 is the standard way to multiply, you would take the dot product. 85 00:05:32,690 --> 00:05:43,280 So the first way is dot products of row dotted with x. 86 00:05:46,190 --> 00:05:49,200 2x1 plus x2 plus 3x3. 87 00:05:49,200 --> 00:05:54,330 It gives you the answer a component at a time. 88 00:05:54,330 --> 00:05:56,630 That's the low level way. 89 00:05:56,630 --> 00:06:00,140 The good way to see it is vector-wise. 90 00:06:00,140 --> 00:06:08,690 See this as x1 multiplies that first column, x2 times 91 00:06:08,690 --> 00:06:14,360 the second column, 1, 1, 7, and x3 times 92 00:06:14,360 --> 00:06:17,530 the third column, 3, 4, 12. 93 00:06:21,994 --> 00:06:24,480 Good. 94 00:06:24,480 --> 00:06:27,750 So it's a combination of vectors, 95 00:06:27,750 --> 00:06:30,740 and of course, it produces a vector. 96 00:06:30,740 --> 00:06:33,800 And here, we have a 3 by 3 matrix on our vectors 97 00:06:33,800 --> 00:06:35,960 are in R3. 98 00:06:35,960 --> 00:06:40,100 And most vectors will be in R3 or Rn for this course. 99 00:06:42,770 --> 00:06:44,400 So that's the right answer. 100 00:06:44,400 --> 00:06:46,430 And of course, the first component 101 00:06:46,430 --> 00:06:50,950 is 2x1 and 1x2 and 3x3. 102 00:06:50,950 --> 00:06:54,510 The same 2, 1, 3-- the same dot product, it comes out right. 103 00:06:54,510 --> 00:06:57,920 But you see it all at once instead of piecemeal. 104 00:07:00,720 --> 00:07:01,860 Part of the course-- 105 00:07:01,860 --> 00:07:04,740 I guess part of what I hope to get across 106 00:07:04,740 --> 00:07:11,590 is thinking of a matrix as a whole thing, 107 00:07:11,590 --> 00:07:15,270 not just a bunch of nine or m times n numbers. 108 00:07:15,270 --> 00:07:19,280 But thinking of it as a thing. 109 00:07:19,280 --> 00:07:25,020 A matrix multiplies a vector to give another vector. 110 00:07:25,020 --> 00:07:32,040 So when I say Ax, you immediately think that-- 111 00:07:32,040 --> 00:07:35,580 you immediately think, OK, Ax has a clear meaning, 112 00:07:35,580 --> 00:07:38,310 it's a combination of the columns of A. 113 00:07:38,310 --> 00:07:44,530 So now let me take that next step. 114 00:07:44,530 --> 00:07:49,540 And the next step is think about all combinations 115 00:07:49,540 --> 00:07:52,645 of the columns of A. We take the matrix A, 116 00:07:52,645 --> 00:07:57,670 and we take all x's and we imagine all the outputs. 117 00:07:57,670 --> 00:08:01,330 And what I want to ask you is, what does that look like? 118 00:08:01,330 --> 00:08:06,070 If I just take 1 vector x, I get a vector output. 119 00:08:06,070 --> 00:08:08,410 It takes a vector to a vector. 120 00:08:08,410 --> 00:08:10,990 But now I take all x's-- 121 00:08:10,990 --> 00:08:17,060 all of vectors x in 3D and I get all these answers, 122 00:08:17,060 --> 00:08:18,435 and I think of them all together. 123 00:08:21,020 --> 00:08:23,390 So I've got a bunch of vectors-- 124 00:08:23,390 --> 00:08:25,940 infinitely many vectors, actually. 125 00:08:25,940 --> 00:08:30,290 And the question is, if I plot those infinitely many vectors, 126 00:08:30,290 --> 00:08:32,179 what do I have? 127 00:08:32,179 --> 00:08:35,480 And the beauty of linear algebra is that questions like this-- 128 00:08:35,480 --> 00:08:40,429 you can answer them and you intuitively see it. 129 00:08:40,429 --> 00:08:45,680 You certainly see it in 3D and you have the right idea 130 00:08:45,680 --> 00:08:48,620 in 10 dimensions, even. 131 00:08:48,620 --> 00:08:52,290 Most of us don't see too well in 10d-- 132 00:08:52,290 --> 00:08:53,090 in R10. 133 00:08:53,090 --> 00:08:54,740 But here we got three. 134 00:08:57,250 --> 00:08:58,850 So do you see what I'm saying? 135 00:08:58,850 --> 00:09:00,950 I'm taking all x's. 136 00:09:00,950 --> 00:09:09,800 So all Ax gives us a big bunch of vectors. 137 00:09:09,800 --> 00:09:13,400 And that collection of vectors is 138 00:09:13,400 --> 00:09:20,460 called the column space of A. It's a space, in other words. 139 00:09:20,460 --> 00:09:23,412 That's the key word there, the column space of A. 140 00:09:23,412 --> 00:09:29,900 And I'll just write it as C of A when I need letters for it. 141 00:09:29,900 --> 00:09:32,540 So I'm going to ask you, what does this column space 142 00:09:32,540 --> 00:09:34,190 look like? 143 00:09:34,190 --> 00:09:36,650 And that depends on the matrix. 144 00:09:36,650 --> 00:09:41,230 Sometimes that column space would be the whole of R3. 145 00:09:41,230 --> 00:09:47,070 Sometimes it's a smaller set in R3, a space. 146 00:09:47,070 --> 00:09:48,700 Do you know what it is in this case? 147 00:09:48,700 --> 00:09:53,110 Do I get all of 3D out of these guys 148 00:09:53,110 --> 00:09:57,380 by choosing all x1 and x2 and x3? 149 00:09:57,380 --> 00:10:01,090 It seems like if it was a random matrix, 150 00:10:01,090 --> 00:10:04,010 the answer would be, surely, yes. 151 00:10:04,010 --> 00:10:07,720 If a random three by three matrix is-- 152 00:10:07,720 --> 00:10:10,390 it's column space is going to be all of our three, 153 00:10:10,390 --> 00:10:13,050 its columns are going to be independent, 154 00:10:13,050 --> 00:10:14,990 its rows are going to be independent, 155 00:10:14,990 --> 00:10:17,890 it's going to be invertible, it's going to be great. 156 00:10:17,890 --> 00:10:19,420 But is this matrix-- 157 00:10:19,420 --> 00:10:21,930 what's up with that matrix? 158 00:10:21,930 --> 00:10:24,210 No, it's not. 159 00:10:24,210 --> 00:10:28,560 So what do I get instead of all of R3? 160 00:10:28,560 --> 00:10:30,390 I get a plane, yeah. 161 00:10:30,390 --> 00:10:32,910 If you get that insight. 162 00:10:32,910 --> 00:10:35,280 So by taking x's-- 163 00:10:35,280 --> 00:10:38,170 everybody's with me-- all x's here. 164 00:10:38,170 --> 00:10:42,460 So that means that it fills in whatever. 165 00:10:42,460 --> 00:10:45,250 And, well, because it's linear algebra, 166 00:10:45,250 --> 00:10:52,960 it's going to be likely all of R3 or a plane, or even a line. 167 00:10:52,960 --> 00:10:56,390 Let me just bat over here for a moment. 168 00:10:56,390 --> 00:11:03,650 Give me a matrix whose column space would only be a line. 169 00:11:03,650 --> 00:11:04,780 All 1s? 170 00:11:04,780 --> 00:11:05,300 OK. 171 00:11:05,300 --> 00:11:09,500 Wow, that's a-- let me liven it up a little. 172 00:11:09,500 --> 00:11:14,180 3, 3, 3, 8, 8, 8. 173 00:11:18,350 --> 00:11:21,410 So I think that the combinations of those columns 174 00:11:21,410 --> 00:11:23,870 are all on the same line. 175 00:11:23,870 --> 00:11:27,800 That says that the column space is just a line. 176 00:11:27,800 --> 00:11:30,220 And I would say then that the matrix-- 177 00:11:30,220 --> 00:11:34,520 so column space of this A is a line. 178 00:11:34,520 --> 00:11:38,930 Another way I could say this is that the rank of the matrix 179 00:11:38,930 --> 00:11:41,450 is 1. 180 00:11:41,450 --> 00:11:45,470 The rank is sort of the dimension of the column space. 181 00:11:45,470 --> 00:11:47,180 Well, not sort, that's what it is. 182 00:11:47,180 --> 00:11:49,430 The rank is the dimension of the column-- 183 00:11:49,430 --> 00:11:51,530 everybody sees that you get a line? 184 00:11:51,530 --> 00:11:54,410 Because any combination x1 of that plus x2 185 00:11:54,410 --> 00:12:00,150 of that plus x3 of that is going to go along that line. 186 00:12:00,150 --> 00:12:03,050 Here's the first column, here's the second column. 187 00:12:03,050 --> 00:12:04,580 They're all on a line. 188 00:12:04,580 --> 00:12:07,070 So I'll never get off that line. 189 00:12:07,070 --> 00:12:10,470 If I allow all x's, I'll get the whole line. 190 00:12:10,470 --> 00:12:14,070 Now here, you said not a line. 191 00:12:14,070 --> 00:12:18,410 What was the column space for this guy? 192 00:12:18,410 --> 00:12:19,730 Plane. 193 00:12:19,730 --> 00:12:23,130 Now, why isn't it all of R3? 194 00:12:23,130 --> 00:12:25,860 What do you see that's special there-- 195 00:12:25,860 --> 00:12:28,050 because it is special-- 196 00:12:28,050 --> 00:12:34,350 that's making the column space special, a plane instead 197 00:12:34,350 --> 00:12:35,820 of the whole thing? 198 00:12:35,820 --> 00:12:38,430 Yeah, what's up with these three columns? 199 00:12:38,430 --> 00:12:40,820 AUDIENCE: The third column is the sum of it. 200 00:12:40,820 --> 00:12:46,160 GILBERT STRANG: The third column is the sum of these two. 201 00:12:46,160 --> 00:12:49,660 So the first column is fine, 2, 3, 5. 202 00:12:49,660 --> 00:12:52,210 Second column is in a different direction. 203 00:12:52,210 --> 00:12:55,190 And when I take combinations of the first two columns, 204 00:12:55,190 --> 00:12:57,200 what will I get? 205 00:12:57,200 --> 00:12:59,900 Sorry to keep asking you questions, but that's-- 206 00:13:03,600 --> 00:13:06,960 anyway, that's what I always do. 207 00:13:06,960 --> 00:13:10,470 So the combinations of those first two columns are--? 208 00:13:10,470 --> 00:13:11,420 AUDIENCE: A plane. 209 00:13:11,420 --> 00:13:13,170 GILBERT STRANG: A plane because they're 210 00:13:13,170 --> 00:13:15,790 in different-- everybody sees that picture. 211 00:13:15,790 --> 00:13:18,960 We've got column one going that way and column two 212 00:13:18,960 --> 00:13:19,630 going that way. 213 00:13:19,630 --> 00:13:22,500 And then if I take any multiple of column one, 214 00:13:22,500 --> 00:13:25,890 I've got a whole line, any multiple of column two. 215 00:13:25,890 --> 00:13:27,840 And then when I put the two together, 216 00:13:27,840 --> 00:13:32,940 it fills in the plane. 217 00:13:32,940 --> 00:13:33,440 Yeah. 218 00:13:33,440 --> 00:13:35,910 Your intuition says that's right. 219 00:13:35,910 --> 00:13:38,640 So this is a matrix of rank. 220 00:13:38,640 --> 00:13:40,350 What's the rank of this matrix? 221 00:13:40,350 --> 00:13:41,240 AUDIENCE: Two. 222 00:13:41,240 --> 00:13:43,030 GILBERT STRANG: Two. 223 00:13:43,030 --> 00:13:45,550 Because it's got two independent columns, 224 00:13:45,550 --> 00:13:47,800 but the third column is dependent. 225 00:13:47,800 --> 00:13:50,510 The third column is a combination of the others. 226 00:13:50,510 --> 00:13:56,620 So matrices like this are really the building blocks 227 00:13:56,620 --> 00:13:59,290 of linear algebra, they're the building blocks 228 00:13:59,290 --> 00:14:00,930 of data science. 229 00:14:00,930 --> 00:14:03,100 They're rank one matrices. 230 00:14:03,100 --> 00:14:07,390 And let me show you a special way to write those rank one 231 00:14:07,390 --> 00:14:10,010 matrices. 232 00:14:10,010 --> 00:14:15,100 I think of this matrix as the column vector 1, 1, 233 00:14:15,100 --> 00:14:19,720 1 times the row vector 1, 3, 8. 234 00:14:19,720 --> 00:14:23,050 So it's a column times a row. 235 00:14:25,860 --> 00:14:28,640 That's a rank one matrix. 236 00:14:28,640 --> 00:14:31,850 Do you see-- that's a true multiplication there. 237 00:14:31,850 --> 00:14:35,960 It looks a little weird, but it's a 3 by 1 matrix times a 1 238 00:14:35,960 --> 00:14:37,580 by 3 matrix. 239 00:14:37,580 --> 00:14:39,350 These numbers have to be the same, 240 00:14:39,350 --> 00:14:42,110 and then the output is 3 by 3. 241 00:14:42,110 --> 00:14:43,020 And it's that. 242 00:14:45,720 --> 00:14:48,780 And do you see that it factors? 243 00:14:48,780 --> 00:14:51,820 So I'm going to move on to that idea. 244 00:14:51,820 --> 00:15:01,600 The next idea coming up will be that we can see-- 245 00:15:01,600 --> 00:15:06,570 well, that's coming, that we're going to see matrices 246 00:15:06,570 --> 00:15:08,950 with two factors. 247 00:15:08,950 --> 00:15:13,630 Let me move to that, but back for this original matrix. 248 00:15:17,000 --> 00:15:19,380 So what's with this? 249 00:15:19,380 --> 00:15:21,860 The column space is a plane. 250 00:15:26,090 --> 00:15:30,110 Think about now the key idea of independent columns. 251 00:15:30,110 --> 00:15:33,777 How many independent columns have I got here? 252 00:15:33,777 --> 00:15:34,360 AUDIENCE: Two. 253 00:15:34,360 --> 00:15:35,740 GILBERT STRANG: Two, correct. 254 00:15:35,740 --> 00:15:36,970 Two. 255 00:15:36,970 --> 00:15:41,290 The third column, if I want to especially pick on that one-- 256 00:15:41,290 --> 00:15:42,850 I'll often go left to right. 257 00:15:42,850 --> 00:15:46,230 So I'll say the first guy is good, second guy is good, 258 00:15:46,230 --> 00:15:51,080 the third guy is not independent of the others. 259 00:15:51,080 --> 00:15:54,290 So I just have two independent columns. 260 00:15:54,290 --> 00:16:01,080 And those two columns would be a basis for the column space. 261 00:16:01,080 --> 00:16:04,280 So that's the critical idea of linear algebra. 262 00:16:04,280 --> 00:16:07,160 That's what you compute when you find a basis, 263 00:16:07,160 --> 00:16:09,910 and everything in the column space 264 00:16:09,910 --> 00:16:14,780 is a combination of these, including that is also-- 265 00:16:14,780 --> 00:16:16,430 that's already a combination of those. 266 00:16:16,430 --> 00:16:18,920 But everything else in the column space 267 00:16:18,920 --> 00:16:22,010 is a combination of those two. 268 00:16:22,010 --> 00:16:23,600 So they're a basis for it. 269 00:16:26,200 --> 00:16:31,060 So you have the idea of A times x. 270 00:16:31,060 --> 00:16:34,090 You have the idea of column space of A, 271 00:16:34,090 --> 00:16:36,470 which allows all x's. 272 00:16:36,470 --> 00:16:43,990 Then now, we're moving into the idea of independent columns, 273 00:16:43,990 --> 00:16:48,230 and the number of independent columns is the rank. 274 00:16:48,230 --> 00:16:51,640 So the rank-- shall I write that somewhere? 275 00:16:51,640 --> 00:16:52,630 Maybe here. 276 00:16:52,630 --> 00:16:55,765 The rank is the number of independent columns. 277 00:17:01,690 --> 00:17:04,880 And right now, what do I mean by independent columns? 278 00:17:04,880 --> 00:17:10,730 Well, let's just see what that means by using it. 279 00:17:10,730 --> 00:17:11,720 Are we good? 280 00:17:11,720 --> 00:17:13,520 I know I'm reviewing here. 281 00:17:13,520 --> 00:17:19,690 But allow me for this first class and part of next time 282 00:17:19,690 --> 00:17:20,589 also to review. 283 00:17:20,589 --> 00:17:21,970 But you'll see something new. 284 00:17:21,970 --> 00:17:24,220 In fact, why don't we see something new right away. 285 00:17:28,020 --> 00:17:32,520 Let me follow up on the idea of independence 286 00:17:32,520 --> 00:17:34,900 in a systematic way. 287 00:17:34,900 --> 00:17:41,370 So here's my matrix A. Can I write that again? 288 00:17:41,370 --> 00:17:50,430 2, 3, 5, 1, 1, 7, and this guy was the sum of those. 289 00:17:50,430 --> 00:17:53,970 So that's my matrix A. 290 00:17:53,970 --> 00:17:59,520 So let's start from scratch and find a basis for the column 291 00:17:59,520 --> 00:18:02,670 space in the most natural way. 292 00:18:02,670 --> 00:18:05,760 So I'm going to take a basis-- 293 00:18:05,760 --> 00:18:07,500 what's a basis? 294 00:18:07,500 --> 00:18:13,270 A basis is independent columns. 295 00:18:13,270 --> 00:18:16,840 So all three together would not be a basis. 296 00:18:16,840 --> 00:18:20,170 But they have to be not just independent, 297 00:18:20,170 --> 00:18:23,290 but they have to fill the space-- their combinations have 298 00:18:23,290 --> 00:18:24,760 to fill the space. 299 00:18:24,760 --> 00:18:27,580 So 2, 3, 5-- 300 00:18:27,580 --> 00:18:30,860 let's say I want to create a basis. 301 00:18:30,860 --> 00:18:35,080 I'll call the matrix C, a basis for the column space. 302 00:18:35,080 --> 00:18:37,460 So here's a natural way to do it. 303 00:18:37,460 --> 00:18:38,620 I look at the first column. 304 00:18:41,170 --> 00:18:42,370 It's not 0s. 305 00:18:42,370 --> 00:18:45,925 If it was all 0s, I wouldn't want that in a basis. 306 00:18:48,530 --> 00:18:49,760 But it's not. 307 00:18:49,760 --> 00:18:51,530 So I put it in. 308 00:18:51,530 --> 00:18:55,930 So that's the first vector in my basis. 309 00:18:55,930 --> 00:18:59,130 Then I go on to the second column. 310 00:18:59,130 --> 00:19:04,780 If that column was 4, 6, 10, what would I do? 311 00:19:04,780 --> 00:19:07,040 If that second column was 4, 6, 10, 312 00:19:07,040 --> 00:19:09,084 would I put it in the basis? 313 00:19:09,084 --> 00:19:11,060 No. 314 00:19:11,060 --> 00:19:13,100 But 1, 1, 7, is OK, right? 315 00:19:13,100 --> 00:19:15,390 1, 1, 7 is in a different direction. 316 00:19:15,390 --> 00:19:18,380 It's not a combination of what we've already got. 317 00:19:18,380 --> 00:19:20,980 So I say, OK, that adds something new. 318 00:19:20,980 --> 00:19:23,500 Put it in. 319 00:19:23,500 --> 00:19:26,590 Then I move on to the third column. 320 00:19:26,590 --> 00:19:29,730 Do I put that into a basis? 321 00:19:29,730 --> 00:19:31,110 You know the answer by now. 322 00:19:31,110 --> 00:19:32,130 No. 323 00:19:32,130 --> 00:19:37,140 Because I'm looking to see, is it a combination of these guys? 324 00:19:37,140 --> 00:19:38,640 And it is. 325 00:19:38,640 --> 00:19:40,650 It's one of that plus one of that. 326 00:19:40,650 --> 00:19:42,120 So it's not independent. 327 00:19:42,120 --> 00:19:45,730 So I've finished now. 328 00:19:45,730 --> 00:19:51,974 I've got a matrix C, which was taken directly from A, 329 00:19:51,974 --> 00:19:55,100 and I kept only independent columns 330 00:19:55,100 --> 00:19:58,360 and I worked from left to right. 331 00:19:58,360 --> 00:20:02,230 And I can see that right away now that the rank is two-- 332 00:20:02,230 --> 00:20:06,350 the column rank, I should say-- the column rank. 333 00:20:06,350 --> 00:20:08,490 The number of independent columns is two. 334 00:20:12,720 --> 00:20:13,220 Good? 335 00:20:16,060 --> 00:20:19,830 Now comes the key step. 336 00:20:19,830 --> 00:20:26,790 I'm going to produce a third matrix, R, 337 00:20:26,790 --> 00:20:32,090 which is going to tell me how to get these columns 338 00:20:32,090 --> 00:20:35,160 from these columns. 339 00:20:35,160 --> 00:20:37,290 And its shape is going to be-- 340 00:20:37,290 --> 00:20:40,140 well, its shape-- I don't have any choice. 341 00:20:40,140 --> 00:20:45,960 This is three by two, so Rx would be two by something. 342 00:20:45,960 --> 00:20:48,840 So I'm like so. 343 00:20:48,840 --> 00:20:51,360 I guess it has to be two by three because I want to come 344 00:20:51,360 --> 00:20:55,230 out this way, two by three. 345 00:20:55,230 --> 00:20:57,670 What am I going to do here? 346 00:20:57,670 --> 00:20:59,310 I'm just going to put it in the numbers 347 00:20:59,310 --> 00:21:00,555 R that make this correct. 348 00:21:04,150 --> 00:21:07,600 So this is a first matrix factorization. 349 00:21:07,600 --> 00:21:10,510 It's not-- well, it is a famous one, actually. 350 00:21:10,510 --> 00:21:14,930 When we see it, we'll recognize what it is. 351 00:21:14,930 --> 00:21:17,850 It's famous in teaching linear algebra, 352 00:21:17,850 --> 00:21:22,210 but now, actually, C times R, columns times rows 353 00:21:22,210 --> 00:21:28,120 has become very, very important in large scale 354 00:21:28,120 --> 00:21:31,490 numerical linear algebra. 355 00:21:31,490 --> 00:21:38,610 So let's figure out what goes into R. 356 00:21:38,610 --> 00:21:40,560 What am I thinking here? 357 00:21:40,560 --> 00:21:44,940 I'm putting in R. So every one of these columns 358 00:21:44,940 --> 00:21:46,350 is a combination of these. 359 00:21:46,350 --> 00:21:48,300 That's the whole point. 360 00:21:48,300 --> 00:21:52,080 And I'm just going to put in the numbers that you need. 361 00:21:52,080 --> 00:21:54,960 So what goes in the first column of R? 362 00:21:54,960 --> 00:21:57,300 What goes in the first column of R? 363 00:21:57,300 --> 00:21:59,460 So I want to look, what combination 364 00:21:59,460 --> 00:22:03,450 of that column and that column gives me this one? 365 00:22:03,450 --> 00:22:03,950 Yeah? 366 00:22:03,950 --> 00:22:04,795 AUDIENCE: 1, 0. 367 00:22:04,795 --> 00:22:05,670 GILBERT STRANG: 1, 0. 368 00:22:11,210 --> 00:22:16,750 Because you remember how we multiply a matrix by a vector? 369 00:22:16,750 --> 00:22:18,860 When I multiply that matrix by that vector, 370 00:22:18,860 --> 00:22:21,410 I take one of this plus zero of that. 371 00:22:21,410 --> 00:22:25,580 I see it vector-wise, and of course I get it right. 372 00:22:25,580 --> 00:22:28,460 And what about the second column of R? 373 00:22:28,460 --> 00:22:30,980 So the second column should be the combination 374 00:22:30,980 --> 00:22:34,820 that produces the second column of A correctly. 375 00:22:34,820 --> 00:22:35,720 What will that be? 376 00:22:35,720 --> 00:22:36,415 AUDIENCE: 0, 1. 377 00:22:36,415 --> 00:22:37,290 GILBERT STRANG: 0, 1. 378 00:22:37,290 --> 00:22:39,570 Thanks. 379 00:22:39,570 --> 00:22:41,736 And finally the third column? 380 00:22:41,736 --> 00:22:42,650 AUDIENCE: 1, 1. 381 00:22:42,650 --> 00:22:44,040 GILBERT STRANG: 1, 1. 382 00:22:44,040 --> 00:22:45,570 Yes, right. 383 00:22:45,570 --> 00:22:50,690 Because one of this plus one of this produces the third column. 384 00:22:50,690 --> 00:22:54,000 So all I did was put in the right numbers there, really. 385 00:23:03,220 --> 00:23:07,070 And this is correct now. 386 00:23:07,070 --> 00:23:12,380 A is C times R. And so this is-- 387 00:23:12,380 --> 00:23:15,590 what I've done here is the first two pages 388 00:23:15,590 --> 00:23:20,395 of section 1.1 in these notes. 389 00:23:20,395 --> 00:23:25,900 So 1.1. 390 00:23:25,900 --> 00:23:28,600 And actually, I'll tell you literally 391 00:23:28,600 --> 00:23:32,980 what happened earlier this year. 392 00:23:32,980 --> 00:23:34,510 I had finished this. 393 00:23:34,510 --> 00:23:38,830 I wrote this down with a different example. 394 00:23:38,830 --> 00:23:44,290 And then I realized something, that sitting here 395 00:23:44,290 --> 00:23:48,850 in front of me was the first great theorem 396 00:23:48,850 --> 00:23:55,030 in linear algebra, the fact that the column rank equals the row 397 00:23:55,030 --> 00:23:57,700 rank. 398 00:23:57,700 --> 00:24:03,980 The fact that if I have a matrix where 399 00:24:03,980 --> 00:24:06,750 that column plus that column gives that one. 400 00:24:06,750 --> 00:24:07,250 Oh. 401 00:24:10,360 --> 00:24:12,120 So what am I going to say here? 402 00:24:12,120 --> 00:24:14,920 I'm getting nervous about it. 403 00:24:14,920 --> 00:24:20,070 I believe that a combination of the rows gives 0. 404 00:24:20,070 --> 00:24:21,570 Do you believe that? 405 00:24:21,570 --> 00:24:22,830 You've got to believe it. 406 00:24:22,830 --> 00:24:24,120 This is linear algebra. 407 00:24:24,120 --> 00:24:28,410 The matrix is not invertible, it's square. 408 00:24:28,410 --> 00:24:32,020 But the columns are dependent. 409 00:24:32,020 --> 00:24:34,800 So the rows have to be dependent. 410 00:24:34,800 --> 00:24:38,230 And I don't exactly see-- 411 00:24:38,230 --> 00:24:41,280 there's some combination of that row and that row 412 00:24:41,280 --> 00:24:43,080 that gives that one. 413 00:24:43,080 --> 00:24:45,390 And of course, when I looked at the first column, 414 00:24:45,390 --> 00:24:49,980 I thought, OK, it's going to be too easy. 415 00:24:49,980 --> 00:24:51,810 One of that and one of that gives that. 416 00:24:51,810 --> 00:24:55,320 But then my eye went over to the second column, 417 00:24:55,320 --> 00:24:57,850 and I realized it's not easy at all. 418 00:24:57,850 --> 00:25:03,810 So you're entitled to pull out your phone and figure this out. 419 00:25:03,810 --> 00:25:06,894 But there is some damn combination of those-- 420 00:25:06,894 --> 00:25:08,136 [LAUGHTER] 421 00:25:08,136 --> 00:25:11,220 --of those two rows that gives the third row. 422 00:25:11,220 --> 00:25:13,900 Otherwise, the course is over, we stop. 423 00:25:17,720 --> 00:25:20,580 Well, and maybe we're going to find somehow. 424 00:25:24,200 --> 00:25:26,050 So this is the theorem. 425 00:25:26,050 --> 00:25:30,880 So I have to tell you, I was really pleased. 426 00:25:30,880 --> 00:25:34,450 So the first two pages got two more pages 427 00:25:34,450 --> 00:25:36,910 to follow up on that idea, that here, we 428 00:25:36,910 --> 00:25:38,710 were seeing something that-- 429 00:25:38,710 --> 00:25:45,400 I proved in 18.06, but not in the first lecture, that's 430 00:25:45,400 --> 00:25:49,330 for sure, and not maybe so clearly. 431 00:25:49,330 --> 00:25:54,310 But now I can try to prove it. 432 00:25:54,310 --> 00:25:56,890 There won't be a lot of proofs in this class, 433 00:25:56,890 --> 00:25:59,650 but this is such an important fact-- 434 00:25:59,650 --> 00:26:02,590 A equals CR is an important factorization. 435 00:26:02,590 --> 00:26:06,730 And out of it, we can connect them. 436 00:26:06,730 --> 00:26:08,650 So what am I saying? 437 00:26:08,650 --> 00:26:12,445 I am saying that all-- 438 00:26:16,630 --> 00:26:18,550 so what's the row rank? 439 00:26:18,550 --> 00:26:20,110 I have to back up here. 440 00:26:20,110 --> 00:26:22,690 What's the row rank? 441 00:26:22,690 --> 00:26:24,160 What's the row space? 442 00:26:24,160 --> 00:26:29,740 So the row rank is going to be the dimension of that space. 443 00:26:29,740 --> 00:26:33,040 So I look at my matrix A. What's the row space of A? 444 00:26:36,000 --> 00:26:38,590 I'm going to look at its rows. 445 00:26:38,590 --> 00:26:42,760 Now, maybe just so we don't get whole new letters 446 00:26:42,760 --> 00:26:43,750 for the row space-- 447 00:26:46,960 --> 00:26:52,540 for me, the row space of A of a matrix-- 448 00:26:52,540 --> 00:26:56,180 so first of all, tell me in words what it is. 449 00:26:56,180 --> 00:26:59,072 What's the row space of the matrix? 450 00:26:59,072 --> 00:27:00,280 AUDIENCE: [INAUDIBLE] 451 00:27:00,280 --> 00:27:04,960 GILBERT STRANG: All combinations of the rows. 452 00:27:04,960 --> 00:27:09,520 All combinations of the rows, that's the space. 453 00:27:09,520 --> 00:27:13,960 So I would take all combinations of those rows. 454 00:27:13,960 --> 00:27:16,930 To get combinations of the rows-- 455 00:27:16,930 --> 00:27:19,630 well, two ways I can get combinations of the rows. 456 00:27:19,630 --> 00:27:25,250 And the way I'll do it is, I'll just transpose the matrix. 457 00:27:25,250 --> 00:27:27,730 So those rows become columns, and then I'm 458 00:27:27,730 --> 00:27:29,770 back to what I did. 459 00:27:29,770 --> 00:27:36,850 So the row space of A is the column space of A transpose. 460 00:27:36,850 --> 00:27:38,920 And this has the advantage that we 461 00:27:38,920 --> 00:27:41,260 don't introduce a new letter. 462 00:27:41,260 --> 00:27:45,560 So it'll be the column space of A transpose. 463 00:27:48,140 --> 00:27:49,780 So we don't introduce a new letter. 464 00:27:49,780 --> 00:27:52,600 We keep the convention that vectors 465 00:27:52,600 --> 00:27:57,990 are column vectors, which would be the MATLAB and Julia 466 00:27:57,990 --> 00:28:03,730 and Python convention. 467 00:28:03,730 --> 00:28:05,290 So is that OK? 468 00:28:05,290 --> 00:28:08,020 The row space is the combination of these rows. 469 00:28:08,020 --> 00:28:10,840 But I'll flip-- I'll make them stand up-- 470 00:28:10,840 --> 00:28:15,470 2, 1, 3-- to be column vectors. 471 00:28:15,470 --> 00:28:17,630 So it's a totally different space. 472 00:28:17,630 --> 00:28:22,520 And actually, I happened to take a three by three example, 473 00:28:22,520 --> 00:28:27,320 so that the column space is part of R3 474 00:28:27,320 --> 00:28:29,780 and the row space is also part of R3. 475 00:28:29,780 --> 00:28:32,270 Because my matrix is three by three. 476 00:28:32,270 --> 00:28:36,980 A better example-- and the whole point of data-- data 477 00:28:36,980 --> 00:28:39,770 doesn't come in square matrices. 478 00:28:39,770 --> 00:28:44,420 Fortunately for us, data very, very often comes in matrices. 479 00:28:44,420 --> 00:28:48,125 But the two-- the columns might be sample, 480 00:28:48,125 --> 00:28:51,050 it might be patients, and the rows 481 00:28:51,050 --> 00:28:54,530 might be diseases or something. 482 00:28:54,530 --> 00:28:56,540 They're different spaces. 483 00:28:56,540 --> 00:29:03,820 So matrices are not likely to be square. 484 00:29:03,820 --> 00:29:07,120 But anyway, we're good here. 485 00:29:07,120 --> 00:29:09,640 So the row space. 486 00:29:09,640 --> 00:29:11,310 Now can I come back to the proof? 487 00:29:11,310 --> 00:29:14,170 Because what I want to say is that the proof 488 00:29:14,170 --> 00:29:17,800 of this fundamental fact is staring at us, 489 00:29:17,800 --> 00:29:19,860 but we don't quite see it yet. 490 00:29:23,150 --> 00:29:25,620 And I want to see it. 491 00:29:25,620 --> 00:29:37,745 So I claim that these rows are a basis for the row space. 492 00:29:40,690 --> 00:29:45,450 And we already saw that these columns are 493 00:29:45,450 --> 00:29:48,690 a basis for the column space. 494 00:29:53,450 --> 00:29:55,940 And two equals two, right? 495 00:29:55,940 --> 00:29:59,740 Two vectors here were a basis for the column space. 496 00:29:59,740 --> 00:30:03,820 Now if I can see why it shows me that these two vectors are 497 00:30:03,820 --> 00:30:08,110 a basis for the row space, then my example 498 00:30:08,110 --> 00:30:11,830 is right, that both of these will give two. 499 00:30:11,830 --> 00:30:14,320 The column rank is two, two columns, 500 00:30:14,320 --> 00:30:16,210 the row rank is two, two rows. 501 00:30:21,200 --> 00:30:27,050 I have to explain why I believe that these two rows are 502 00:30:27,050 --> 00:30:31,040 a basis for the row space. 503 00:30:31,040 --> 00:30:31,822 Are you with me? 504 00:30:31,822 --> 00:30:32,530 I have to prove-- 505 00:30:32,530 --> 00:30:34,210 I have to see why. 506 00:30:34,210 --> 00:30:39,240 First, so when I say basis, what do I have to check? 507 00:30:39,240 --> 00:30:45,660 Basis is the critical idea. 508 00:30:45,660 --> 00:30:49,060 I have to check that they're independent, so I 509 00:30:49,060 --> 00:30:50,770 haven't got too many vectors-- 510 00:30:50,770 --> 00:30:53,290 I haven't got any extra vectors in there. 511 00:30:53,290 --> 00:30:57,760 And I also have to check that their combinations produce--? 512 00:31:01,080 --> 00:31:01,830 All the rows. 513 00:31:04,570 --> 00:31:05,670 Should I say that again? 514 00:31:05,670 --> 00:31:07,350 Because that's what I'm going to check. 515 00:31:07,350 --> 00:31:09,475 I'm going to check that those guys are independent. 516 00:31:09,475 --> 00:31:11,458 Well, you can see that they're independent. 517 00:31:14,330 --> 00:31:19,070 And I'm going to check that their combinations produce 518 00:31:19,070 --> 00:31:22,370 all three of these rows. 519 00:31:22,370 --> 00:31:27,200 We didn't create those numbers for this purpose, 520 00:31:27,200 --> 00:31:31,150 but what I'm saying is they work. 521 00:31:31,150 --> 00:31:35,860 So I claim that this is a basis, because what combination 522 00:31:35,860 --> 00:31:40,330 of those two rows would produce this first row? 523 00:31:40,330 --> 00:31:44,110 Yeah, let me just ask you that. 524 00:31:44,110 --> 00:31:47,440 What numbers should I multiply these two rows by 525 00:31:47,440 --> 00:31:49,410 to get the first row of A? 526 00:31:49,410 --> 00:31:50,410 AUDIENCE: 2, 1. 527 00:31:50,410 --> 00:31:52,630 GILBERT STRANG: 2 and 1. 528 00:31:52,630 --> 00:31:56,510 And where do you find 2 and 1? 529 00:31:56,510 --> 00:32:04,550 It's sitting there in C. Will it work again? 530 00:32:04,550 --> 00:32:10,720 Does three of this plus one of that give 3, 1, 4? 531 00:32:10,720 --> 00:32:12,190 Yes. 532 00:32:12,190 --> 00:32:13,930 So far, so good. 533 00:32:13,930 --> 00:32:19,660 Does five of this and seven of that-- see, I'm multiplying-- 534 00:32:19,660 --> 00:32:23,560 I guess I'm doing matrix multiplication a backward way 535 00:32:23,560 --> 00:32:25,120 or a different way. 536 00:32:25,120 --> 00:32:31,300 I'm taking combinations of the rows of the second guy. 537 00:32:31,300 --> 00:32:33,430 The wonderful thing about matrix multiplication 538 00:32:33,430 --> 00:32:36,220 is you can do it a lot of ways, it comes out 539 00:32:36,220 --> 00:32:40,590 the same every way, and each way tells you something. 540 00:32:40,590 --> 00:32:44,950 So five of that row plus seven of that row, sure enough, 541 00:32:44,950 --> 00:32:45,450 is here. 542 00:32:48,240 --> 00:32:50,595 Do you see that that is not accident? 543 00:32:54,040 --> 00:32:57,100 The proof is really to look at this multiplication, 544 00:32:57,100 --> 00:32:59,680 C times R, two ways. 545 00:32:59,680 --> 00:33:02,800 Look at it first as combinations of columns of C 546 00:33:02,800 --> 00:33:04,380 to give the columns. 547 00:33:04,380 --> 00:33:09,100 Look at it second to get as combinations of the rows of R, 548 00:33:09,100 --> 00:33:11,750 and that produces the rows. 549 00:33:11,750 --> 00:33:19,910 So that factorization A equals CR was the key idea. 550 00:33:25,020 --> 00:33:32,690 And actually, this R that we've come up with has a name. 551 00:33:32,690 --> 00:33:36,770 Anybody remember enough 18.06? 552 00:33:36,770 --> 00:33:39,690 Have you all taken 18.06? 553 00:33:39,690 --> 00:33:40,310 No. 554 00:33:40,310 --> 00:33:43,660 I see-- how many have? 555 00:33:43,660 --> 00:33:44,240 Yes. 556 00:33:44,240 --> 00:33:44,940 OK. 557 00:33:44,940 --> 00:33:45,440 Good. 558 00:33:48,100 --> 00:33:52,540 For a while, 18.06 was taught in a very abstract way. 559 00:33:52,540 --> 00:33:53,980 I said, what's going on? 560 00:33:53,980 --> 00:33:58,120 But anyway, so if you took it that semester, 561 00:33:58,120 --> 00:34:01,210 you maybe never heard of the column space. 562 00:34:01,210 --> 00:34:02,900 I'm not sure. 563 00:34:02,900 --> 00:34:04,630 Or by a different name. 564 00:34:04,630 --> 00:34:06,070 It has another name. 565 00:34:06,070 --> 00:34:07,840 What's its other name? 566 00:34:07,840 --> 00:34:09,579 The column space of a matrix? 567 00:34:12,280 --> 00:34:16,010 Range-- I think it's the range. 568 00:34:16,010 --> 00:34:16,600 Yeah. 569 00:34:16,600 --> 00:34:21,460 And of course, all this is fundamental in mathematics. 570 00:34:21,460 --> 00:34:27,909 So of course, everything here has different languages 571 00:34:27,909 --> 00:34:30,190 and different emphasis. 572 00:34:30,190 --> 00:34:33,310 But you see what the emphasis is here. 573 00:34:33,310 --> 00:34:38,080 So you see the proof, that A equals CR just 574 00:34:38,080 --> 00:34:40,560 reveals everything. 575 00:34:40,560 --> 00:34:45,199 So it's our first idea of a factorization of a matrix. 576 00:34:45,199 --> 00:34:51,360 And we've multiplied C times R. I could just say-- 577 00:34:51,360 --> 00:34:55,980 so really, you've seen now the main point of Section 1.1 578 00:34:55,980 --> 00:35:00,750 of the notes to come up with that factorization 579 00:35:00,750 --> 00:35:01,980 and that conclusion. 580 00:35:05,020 --> 00:35:09,280 And you see why C has the same number of-- 581 00:35:09,280 --> 00:35:13,000 the number of columns of C equals the number of rows of R, 582 00:35:13,000 --> 00:35:16,510 and those are the column rank and the row rank. 583 00:35:16,510 --> 00:35:18,970 Yeah, it's just pretty neat. 584 00:35:18,970 --> 00:35:24,670 And here was the special case where those the column space is 585 00:35:24,670 --> 00:35:27,730 all multiples of U-- it's a line through U. 586 00:35:27,730 --> 00:35:30,490 The row space is all multiples of V-- 587 00:35:30,490 --> 00:35:34,200 it's a line through V. And that's the basic building 588 00:35:34,200 --> 00:35:34,700 block. 589 00:35:37,870 --> 00:35:39,820 Can I just say another little word 590 00:35:39,820 --> 00:35:43,310 before I push on beyond CR? 591 00:35:43,310 --> 00:35:45,970 That this has become-- 592 00:35:45,970 --> 00:35:51,800 if you have a giant matrix, like size 10 to the 5th, 593 00:35:51,800 --> 00:35:54,630 you can't put that into fast memory. 594 00:35:54,630 --> 00:35:56,870 It's a mess. 595 00:35:56,870 --> 00:35:59,360 How do you deal with a matrix of size 10 to the 5th, 596 00:35:59,360 --> 00:36:02,670 when you cannot deal with all the entries? 597 00:36:02,670 --> 00:36:06,030 That's just not possible. 598 00:36:06,030 --> 00:36:08,230 Well, you sample it. 599 00:36:08,230 --> 00:36:10,480 So later in the course, we'll be doing 600 00:36:10,480 --> 00:36:12,970 random sampling of a matrix. 601 00:36:12,970 --> 00:36:16,460 So how could you sample a matrix? 602 00:36:16,460 --> 00:36:17,900 So you have a matrix. 603 00:36:17,900 --> 00:36:21,440 Of course you're looking at it, but it's-- 604 00:36:21,440 --> 00:36:26,920 and you want to get some typical columns. 605 00:36:26,920 --> 00:36:28,240 Here's the natural idea. 606 00:36:30,760 --> 00:36:33,640 You just look at A times x. 607 00:36:33,640 --> 00:36:37,900 Let x be a random vector. 608 00:36:37,900 --> 00:36:43,810 Rand of-- so it's got m rows and one column. 609 00:36:43,810 --> 00:36:44,440 It's a vector. 610 00:36:48,880 --> 00:36:51,410 And what can I say about Ax? 611 00:36:54,080 --> 00:36:56,623 It's in the-- what space is it in? 612 00:36:56,623 --> 00:36:57,450 AUDIENCE: Column. 613 00:36:57,450 --> 00:36:58,200 GILBERT STRANG: Column space. 614 00:36:58,200 --> 00:36:58,740 Thanks. 615 00:36:58,740 --> 00:37:02,380 That was the first idea in this lecture. 616 00:37:02,380 --> 00:37:04,120 Ax is in the column space. 617 00:37:04,120 --> 00:37:06,930 So if you want a random vector in the column space, 618 00:37:06,930 --> 00:37:09,210 I wouldn't suggest to just randomly pick 619 00:37:09,210 --> 00:37:11,170 one of the columns. 620 00:37:11,170 --> 00:37:14,440 Better to take a mixture of columns 621 00:37:14,440 --> 00:37:18,520 by taking a random vector x, and looking at Ax. 622 00:37:18,520 --> 00:37:20,680 And if you wanted 100 random vectors, 623 00:37:20,680 --> 00:37:25,540 you'd take a 100 random x's, and that would give you 624 00:37:25,540 --> 00:37:30,460 a pretty good idea, in many cases, of what the column 625 00:37:30,460 --> 00:37:32,290 space looks like. 626 00:37:32,290 --> 00:37:34,410 That would be enough to work with often. 627 00:37:37,390 --> 00:37:40,370 Can I just throw in another question? 628 00:37:40,370 --> 00:37:44,030 So Ax is in the column space of A. 629 00:37:44,030 --> 00:37:46,060 Let me just ask you this question. 630 00:37:46,060 --> 00:37:55,000 Is ABCx-- is that in the column space of A? 631 00:37:55,000 --> 00:37:59,710 Suppose I have matrices A, B, and C, and a vector x, 632 00:37:59,710 --> 00:38:01,540 and I take their product. 633 00:38:01,540 --> 00:38:04,420 Does that give me something in the column space of A? 634 00:38:04,420 --> 00:38:05,710 AUDIENCE: Yes. 635 00:38:05,710 --> 00:38:07,870 GILBERT STRANG: Yes, good for you. 636 00:38:07,870 --> 00:38:09,126 How do you know that? 637 00:38:09,126 --> 00:38:11,360 AUDIENCE: [INAUDIBLE] 638 00:38:11,360 --> 00:38:14,800 GILBERT STRANG: Yeah, it's A times something. 639 00:38:14,800 --> 00:38:16,120 Right. 640 00:38:16,120 --> 00:38:18,100 Putting parentheses in the right place 641 00:38:18,100 --> 00:38:20,050 is the key to linear algebra. 642 00:38:20,050 --> 00:38:23,082 And there it is. 643 00:38:23,082 --> 00:38:24,790 It's a question that just occurred to me. 644 00:38:24,790 --> 00:38:29,040 And I thought, well, I wonder if you'd do it. 645 00:38:29,040 --> 00:38:34,210 So we have still time to multiply matrices. 646 00:38:34,210 --> 00:38:36,840 Oh, I was going to say about C and R-- 647 00:38:36,840 --> 00:38:42,660 so these are real columns from A. But R is-- 648 00:38:42,660 --> 00:38:47,370 the rows are not taken directly from the rows of A. Actually, 649 00:38:47,370 --> 00:38:49,200 there is a name for this. 650 00:38:49,200 --> 00:38:53,580 It's called the row reduced echelon form of the matrix, 651 00:38:53,580 --> 00:38:57,880 and it's a big goal in 18.06. 652 00:38:57,880 --> 00:39:02,970 It has the identity there, and then the other columns 653 00:39:02,970 --> 00:39:04,770 tell you the right combinations. 654 00:39:08,510 --> 00:39:15,500 Another big factorization would be to take columns from A-- 655 00:39:15,500 --> 00:39:17,680 so this is another-- 656 00:39:17,680 --> 00:39:19,070 so I'll put maybe or-- 657 00:39:22,630 --> 00:39:26,810 and we won't be doing this for a month. 658 00:39:26,810 --> 00:39:31,040 We could start to take columns of A 659 00:39:31,040 --> 00:39:34,550 and put them into C, if they were independent. 660 00:39:34,550 --> 00:39:37,910 And suppose I took rows of A. Now I'm 661 00:39:37,910 --> 00:39:43,850 going to take literally rows of A, and put them into R-- 662 00:39:43,850 --> 00:39:47,200 well, shall I call it R twiddle or something, 663 00:39:47,200 --> 00:39:51,260 because it'll be a different R. I'm not 664 00:39:51,260 --> 00:39:55,850 going to use those rows, but I'm going to take two 665 00:39:55,850 --> 00:39:59,630 actual rows of A. Then what? 666 00:39:59,630 --> 00:40:01,260 So that's an important factorization, 667 00:40:01,260 --> 00:40:02,135 but it's not correct. 668 00:40:05,230 --> 00:40:09,620 If I took two other rows of R, it wouldn't work. 669 00:40:09,620 --> 00:40:12,430 So you have to put it in the middle some two 670 00:40:12,430 --> 00:40:17,410 by two matrix U that makes it correct. 671 00:40:17,410 --> 00:40:24,310 You'll see in that Section 1.1 that I got excited 672 00:40:24,310 --> 00:40:28,000 and wrote a page about CUR. 673 00:40:28,000 --> 00:40:32,160 Yeah, so I'll just mention that. 674 00:40:32,160 --> 00:40:34,770 So now I'm ready-- oh, I wanted to say something 675 00:40:34,770 --> 00:40:35,540 about the course. 676 00:40:38,070 --> 00:40:43,210 I got excited thinking about math, but there is this course. 677 00:40:43,210 --> 00:40:44,080 So what's up? 678 00:40:46,930 --> 00:40:50,820 So there'll be linear algebra problems. 679 00:40:50,820 --> 00:40:55,810 But what makes this course special is the other homeworks, 680 00:40:55,810 --> 00:40:58,600 which are online. 681 00:40:58,600 --> 00:41:00,235 And you would use-- 682 00:41:03,090 --> 00:41:03,590 let's see. 683 00:41:03,590 --> 00:41:07,910 In principle, you could use any of the languages-- 684 00:41:07,910 --> 00:41:12,990 MATLAB, Python, which has become the biggest-- 685 00:41:12,990 --> 00:41:16,740 most used for deep learning, or Julia, 686 00:41:16,740 --> 00:41:18,705 which is the hot, new language. 687 00:41:22,910 --> 00:41:28,900 So last time, last year, the problems-- 688 00:41:28,900 --> 00:41:32,290 oh, so what happened last year? 689 00:41:32,290 --> 00:41:35,410 Well, everything in this course is 690 00:41:35,410 --> 00:41:40,090 owed to a professor who visited from the University 691 00:41:40,090 --> 00:41:41,740 of Michigan, Professor Rao-- 692 00:41:41,740 --> 00:41:47,490 Raj Rao, who gave most of the lectures a year ago, 693 00:41:47,490 --> 00:41:50,580 brought these homework problems-- 694 00:41:50,580 --> 00:41:52,770 online homework problems, so that people 695 00:41:52,770 --> 00:41:55,950 brought laptops to class and we did things in class. 696 00:41:58,650 --> 00:42:02,990 So he had and has a very successful course 697 00:42:02,990 --> 00:42:07,500 at EE in Michigan. 698 00:42:07,500 --> 00:42:10,950 But he was a PhD from here, and he came back on a sabbatical, 699 00:42:10,950 --> 00:42:13,980 and he created this-- got us started. 700 00:42:13,980 --> 00:42:15,650 And we really owe a lot to him. 701 00:42:19,130 --> 00:42:23,850 Also, Professor Edelman was involved with this course. 702 00:42:23,850 --> 00:42:26,310 And you maybe know that he created Julia. 703 00:42:26,310 --> 00:42:29,040 How many know what Julia is? 704 00:42:29,040 --> 00:42:30,660 Oh, wonderful. 705 00:42:30,660 --> 00:42:31,920 That will make his day. 706 00:42:34,480 --> 00:42:37,050 He tells me every time I see him, Julia is good. 707 00:42:37,050 --> 00:42:38,910 And I tell him, I believe it. 708 00:42:38,910 --> 00:42:40,680 [LAUGHTER] 709 00:42:40,680 --> 00:42:44,220 Anyway-- and it's become-- 710 00:42:44,220 --> 00:42:47,520 Professor Johnson in 18.06 has used Julia. 711 00:42:47,520 --> 00:42:54,120 And every semester, Steven Johnson gives in the first week 712 00:42:54,120 --> 00:42:58,440 a tutorial on Julia. 713 00:42:58,440 --> 00:43:02,370 And so that's arranged, and I promised to tell you 714 00:43:02,370 --> 00:43:05,910 where and when that is. 715 00:43:05,910 --> 00:43:08,220 So I think if you don't know anything about Julia, 716 00:43:08,220 --> 00:43:09,360 try to go. 717 00:43:09,360 --> 00:43:11,250 It's in Stata. 718 00:43:11,250 --> 00:43:14,570 It's on Friday at 5:00-- 719 00:43:14,570 --> 00:43:16,140 5:00 to 7:00. 720 00:43:16,140 --> 00:43:22,545 So Julia from Professor Johnson. 721 00:43:25,320 --> 00:43:27,210 So he's done this multiple times. 722 00:43:27,210 --> 00:43:28,390 He's good at it. 723 00:43:32,280 --> 00:43:35,130 I don't think we know yet what-- 724 00:43:35,130 --> 00:43:39,090 I guess I'm hoping that you'll have an option to use 725 00:43:39,090 --> 00:43:41,170 any of the three languages. 726 00:43:41,170 --> 00:43:47,880 But the online thing that we give you was created in Julia. 727 00:43:47,880 --> 00:43:51,290 So professor Rao had to learn Julia last spring, 728 00:43:51,290 --> 00:43:53,160 and the class did, too. 729 00:43:53,160 --> 00:43:55,740 And there was a certain amount of bitching about it. 730 00:43:55,740 --> 00:43:57,990 [LAUGHTER] 731 00:43:57,990 --> 00:44:01,750 But I think, with maybe one exception, 732 00:44:01,750 --> 00:44:06,105 who still got an A, everybody was OK 733 00:44:06,105 --> 00:44:08,330 and was glad to learn Julia. 734 00:44:08,330 --> 00:44:12,330 And Professor Rao now uses Julia entirely. 735 00:44:12,330 --> 00:44:16,650 So he's creating a new on ramp with Julia. 736 00:44:16,650 --> 00:44:19,920 MATLAB, by the way, has just issued an on ramp 737 00:44:19,920 --> 00:44:22,450 to deep learning that I'll tell you about, 738 00:44:22,450 --> 00:44:25,380 and probably get a MATLAB-- 739 00:44:25,380 --> 00:44:30,630 somebody from Math Works to say something about it. 740 00:44:30,630 --> 00:44:36,960 So that's what's coming, that we don't quite know exactly how 741 00:44:36,960 --> 00:44:39,150 well organized those homeworks. 742 00:44:39,150 --> 00:44:43,720 We'll just take the first one and see what happens. 743 00:44:43,720 --> 00:44:48,990 So I'll certainly say more about homeworks when-- 744 00:44:48,990 --> 00:44:50,610 maybe even Friday. 745 00:44:50,610 --> 00:44:54,282 But are there other questions that I should answer? 746 00:44:54,282 --> 00:44:55,990 Because some people will be thinking, OK, 747 00:44:55,990 --> 00:45:00,180 am I going to do this or am I going to sit in 6.036, 748 00:45:00,180 --> 00:45:03,960 or some other course in deep learning? 749 00:45:03,960 --> 00:45:06,090 Anything on your mind? 750 00:45:06,090 --> 00:45:08,910 And you can email me. 751 00:45:08,910 --> 00:45:12,310 So we will have a Stellar site and you'll see all the-- 752 00:45:12,310 --> 00:45:14,550 the TAs are still to be named. 753 00:45:14,550 --> 00:45:17,970 But the wonderful thing is that the undergraduates who 754 00:45:17,970 --> 00:45:22,080 took this course last year are volunteering 755 00:45:22,080 --> 00:45:25,170 to be graders for you guys. 756 00:45:25,170 --> 00:45:30,990 So they will know what those online homeworks were about. 757 00:45:30,990 --> 00:45:37,860 So that's a first word about what's coming 758 00:45:37,860 --> 00:45:41,670 and about the language. 759 00:45:41,670 --> 00:45:47,160 I'm going to finish by a very important topic, 760 00:45:47,160 --> 00:45:51,050 multiplying A times B. Oh, look, a clean board. 761 00:45:51,050 --> 00:45:57,110 So now I want to multiply a matrix by a vector. 762 00:45:57,110 --> 00:46:00,680 Everybody knows how to do it. 763 00:46:00,680 --> 00:46:07,650 You take a row of A-- so you take a row of A 764 00:46:07,650 --> 00:46:13,330 and you take a column of B, and you take the dot product. 765 00:46:13,330 --> 00:46:14,720 So you get a dot product. 766 00:46:17,480 --> 00:46:18,380 Row dot column. 767 00:46:24,310 --> 00:46:30,180 That's, again, low level, OK for beginners. 768 00:46:30,180 --> 00:46:36,510 But we want to see that matrix multiplication, AB, in a deeper 769 00:46:36,510 --> 00:46:38,250 way. 770 00:46:38,250 --> 00:46:42,840 And the deeper way is columns times row. 771 00:46:42,840 --> 00:46:48,060 Columns of A, rows of B. Columns times row. 772 00:46:48,060 --> 00:46:49,950 Oh, we had a column times a row. 773 00:46:52,690 --> 00:46:54,355 That was this rank one example. 774 00:46:57,130 --> 00:47:00,040 We had a column times a row, and it produced a matrix. 775 00:47:00,040 --> 00:47:02,040 And that's what it looked like. 776 00:47:02,040 --> 00:47:04,600 And its rank was one. 777 00:47:04,600 --> 00:47:05,890 So those are what-- 778 00:47:10,260 --> 00:47:11,850 so it's a combination of. 779 00:47:14,520 --> 00:47:16,740 It's very like Ax. 780 00:47:16,740 --> 00:47:20,040 I'm really just extending the Ax idea to AB. 781 00:47:22,550 --> 00:47:25,340 So this is the old way. 782 00:47:25,340 --> 00:47:26,750 The new way is columns. 783 00:47:26,750 --> 00:47:32,150 So there's column K. It will multiply. 784 00:47:32,150 --> 00:47:36,980 Sure enough, it multiplies row K. Everybody sees 785 00:47:36,980 --> 00:47:39,420 that it will happen that way. 786 00:47:39,420 --> 00:47:41,210 If you do it the old way, when you 787 00:47:41,210 --> 00:47:44,540 do a dot product of something here, something here, 788 00:47:44,540 --> 00:47:46,280 you're doing these multiplications. 789 00:47:46,280 --> 00:47:50,000 And when you hit column K here, you hit row K there. 790 00:47:50,000 --> 00:47:51,942 So these are connected. 791 00:47:56,370 --> 00:48:08,130 So I get things like column K of A times row K of B. 792 00:48:08,130 --> 00:48:10,010 And I don't know what notation to use, 793 00:48:10,010 --> 00:48:14,430 so I just wrote the words. 794 00:48:14,430 --> 00:48:21,000 But now that's one piece of the final answer, AB. 795 00:48:21,000 --> 00:48:22,650 That's a rank one piece. 796 00:48:22,650 --> 00:48:24,570 That's a building block. 797 00:48:24,570 --> 00:48:31,920 So I add from K equals 1 column one times row one. 798 00:48:31,920 --> 00:48:35,150 Column one of A times row one of B plus-- 799 00:48:35,150 --> 00:48:38,380 da, da, da, plus column K plus-- 800 00:48:38,380 --> 00:48:48,500 and of course, I stop at column n of A times row n of B. 801 00:48:48,500 --> 00:48:51,890 So it's a sum of outer products. 802 00:48:54,560 --> 00:48:57,650 So everybody sees a sum because I 803 00:48:57,650 --> 00:49:05,690 have column one times row 1, column K times row K, 804 00:49:05,690 --> 00:49:10,720 column n times row n, and then I add those pieces. 805 00:49:10,720 --> 00:49:17,440 It's just the generalization of Ax to a matrix B there. 806 00:49:17,440 --> 00:49:30,310 So it's a sum of Column K row K of A row K of B. And maybe, 807 00:49:30,310 --> 00:49:34,780 should we check that that gives us the right answer? 808 00:49:34,780 --> 00:49:38,350 I won't do that here, but all we're doing 809 00:49:38,350 --> 00:49:44,210 is the same multiplications in a different order. 810 00:49:44,210 --> 00:49:46,550 Actually, let's just quit with one minute. 811 00:49:46,550 --> 00:49:49,430 We can figure out how many multiplications are there. 812 00:49:49,430 --> 00:49:52,760 How many more applications to do an m by n 813 00:49:52,760 --> 00:49:58,570 matrix A times an n by p matrix B? 814 00:49:58,570 --> 00:50:02,610 So that's A times B. How many individual numbers-- 815 00:50:02,610 --> 00:50:05,180 because this would determine the cost of it. 816 00:50:05,180 --> 00:50:09,870 How many numbers would we need? 817 00:50:09,870 --> 00:50:18,240 Well, suppose we do it the old way, by inner product, 818 00:50:18,240 --> 00:50:19,440 row times column. 819 00:50:19,440 --> 00:50:23,460 So how many multiplications to do a row times column 820 00:50:23,460 --> 00:50:26,580 and get one entry in the answer? 821 00:50:26,580 --> 00:50:29,060 n, right? 822 00:50:29,060 --> 00:50:31,070 The row has length n, the column has 823 00:50:31,070 --> 00:50:33,860 length n, n multiplications. 824 00:50:33,860 --> 00:50:34,800 So that's n. 825 00:50:34,800 --> 00:50:38,150 And now how many of those do I have to do? 826 00:50:38,150 --> 00:50:39,950 AUDIENCE: mp. 827 00:50:39,950 --> 00:50:41,330 GILBERT STRANG: mp. 828 00:50:41,330 --> 00:50:44,120 Because what's the size of this answer? 829 00:50:44,120 --> 00:50:46,340 The size of this answer is m by p. 830 00:50:49,580 --> 00:50:54,225 So if I do it in that old order, like n multiplication 831 00:50:54,225 --> 00:50:56,310 is to do a dot product. 832 00:50:56,310 --> 00:50:59,250 And I've got this many dot products in the answer. 833 00:50:59,250 --> 00:51:02,540 So I've mnp multiplied. 834 00:51:07,630 --> 00:51:09,120 Now suppose I do it this way. 835 00:51:12,020 --> 00:51:16,440 How many multiplications to do one of those guys? 836 00:51:16,440 --> 00:51:18,450 To multiply our column by a row? 837 00:51:23,716 --> 00:51:29,291 This is an m by 1, and this is a 1 by p. 838 00:51:29,291 --> 00:51:30,980 One column, one row. 839 00:51:30,980 --> 00:51:34,390 How many multiplications for that guy? 840 00:51:34,390 --> 00:51:36,530 mp. 841 00:51:36,530 --> 00:51:40,718 And how many of those rank ones do I have to do? 842 00:51:40,718 --> 00:51:43,640 n. 843 00:51:43,640 --> 00:51:44,940 You got it? 844 00:51:44,940 --> 00:51:46,740 mp times n. 845 00:51:46,740 --> 00:51:56,580 Now, the other way was n times mp. 846 00:51:56,580 --> 00:52:00,330 So it gives the same answer, mnp multiplication. 847 00:52:00,330 --> 00:52:02,940 In fact, they're exactly the same multiplications, just 848 00:52:02,940 --> 00:52:04,000 a different order. 849 00:52:04,000 --> 00:52:04,500 OK. 850 00:52:04,500 --> 00:52:06,540 We're at 1:55. 851 00:52:06,540 --> 00:52:08,760 Thanks for coming today. 852 00:52:08,760 --> 00:52:11,640 I'll talk more about the class and about linear algebra 853 00:52:11,640 --> 00:52:12,570 on Friday. 854 00:52:12,570 --> 00:52:13,984 Thank you.