D4M Session 5 Demo

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: All right. So we're going to now show some of the examples. And so again, they're in our directory here, Examples. We're now in section 2, so we're going to go to Applications. And we have this section called Perfect Power Law. The actual example codes are all these ones called PPL 1, 2, 3, and 4. We have actually a lot of other functions in here, which are actually really useful functions. They're the functions that do all this stuff. And maybe they should actually get folded into the main D4M distribution. They're here right now. They probably could be cleaned up a little bit as part of the homework. But these are, again, very useful functions for allowing you to generate and fit these types of things.

So I have already started my MatLab session. There we are. So we will do the first one. Going along here. Figures. So if we go and look at what we did here-- so the first thing we did is we set our parameters for our Perfect Power Law fit. So we set alpha at 1.3, a D max of 1,000, and then approximately 30 bins. We called our function here, which is in this directory, which is Power Law Distribution, which will create a power law distribution from these three parameters. And then the first thing we did is just plot a distribution. So we go here, then we go to Figure 1. You can see there is that distribution. So our Perfect Power Law distribution with these parameters, very nicely done.

Then moving along, we can compute. This is how you compute the total number of vertices from the distribution. We just sum. It could be the total number of edges. We can sum there. And then we see that the total vertices was 18,187, 84,000 edges with an edge to vertex ratio of 4.6. We now have a function called edges from distribution. So if I pass in the distribution-- essentially the degrees and the counts-- it will then generate a set of vertices that is consistent with that distribution.

And now I'm going to create some permutations, basically, of these vertices. I'm going to create a-- randomly permute the edge order with these two sets here. So if I make a random permutation of the number of edges, that allows me to permute the edges. And I can also randomly permute the vertex labels themselves, and then I can look at these different permutations to see what kind of grass they've created.

So if I don't permute the data, I just have a0 here. I just pass in the vertices that were created. I create a non-permuted adjacency matrix. I can then look at that. And that just shows you-- this is the adjacency matrix. This is in the plot. It just shows you that I just created a vertex that's entirely self loops. Not very interesting, but nevertheless, completely consistent with the perfect power of law degree distribution.

So likewise, if I just permute the edges-- so basically, I take my permutation here, and I just permute that-- this is just not permuting the vertex labels, but just permuting the edges. To the next figure here, 3, and we get this distribution-- basically start vertex, end vertex. Looks fairly random. However, you see the highest degree vertices are up here at 0, 0, because that's the order in which the generator spits them out. It gives you the highest degree vertices first. And so although we've reconnected them with different edges, you see that their vertex labels still have an intrinsic order which is corresponding to vertex degree.

I should say this generates this naturally. It's a very common way to-- if you have arbitrary vertices and you want to put them on an adjacency matrix, one of the first things to do is to reorder them according to degree. You'll always get some kind of structure that emerges there, and it's often an easier way to see what's going on there. And as you can see from the previous data, you saw if you ordered things by degree and you saw this type of [INAUDIBLE] aha, that looks a lot like a power law distribution just from the scatter of the dots.

So moving on here. So now we're just going to permute the vertex labels. So we are not permuting the edges, just the vertex labels. So if you see here-- and then by permuting the vertex labels, it looks random, but fairly sparse. That's because we still-- every single time, we've moved the vertex labels around. But all the edges associated with that vertex pair still will ship around [INAUDIBLE]. We haven't done anything to break them up. Whereas before, we broke up the edges, but we didn't change the labels.

And so then the final one, which is the one that we typically do when we want to create random, is we permute both the vertices and the edges to create a truly random looking adjacency matrix. And there you see you have something. And now what you can also see here is these high degree vertices do stand out. This is a very standard power law distribution. You'll see these dense rows and columns in there that are indicative of these very high degree nodes.

When we just permuted the edges but not the vertices, these high degree rows and columns all got shifted up into the corner of the plot. So again, another way to begin to look at the data in a way to recognize structure. And I think this adjacency matrices, once you start looking at them, you begin to get a comfort zone, just as we do with other types of data. We begin to learn what they look like. This is a very good way to look at the data, because you can look at a fair amount. If I were to actually plot the graph of 18,000 vertices as a traditional graph of the vertices and lines connecting them, it would just be blue. The entire screen would be blue. There would be no way to properly position those vertices.

Here, this is 84,000 data points, and I can still kind of see it a little bit. This is the limit. 100,000 edges is the limit of what you can put on any plot to really see it, unless there's some hidden structure that allows you to really, really move it all together. But the adjacency matrix is a great way to look at fairly large graphs. So we generally do that.

So this is the procedure. Create your perfect parallel distribution, create some edges, create some permutations, and then permute it, and then you off you are. You've just created a randomized perfect power law graph. Again, this is an example of a code where you probably might even just take this program and adjust it to suit your needs. And then, I think, we did that. And then finally, we plotted the degree distribution of a 3 just to show you that I'm not lying. And again, the triangle is the original and the blue was the thing. And you see throughout all those permutations, our degree distribution remained the same. Even though they look-- those would be completely different graphs, their degree distributions are identical.

So let's move on here. So this is just showing-- so we actually rolled this up all into one function here for you. Ran power law matrix, if you give it an alpha, a D max, and an ND, it will do those three steps for you that I had in the previous chart all in one thing and produce an adjacency matrix that's a perfect power law based on these parameters. And again, you saw we have the same number of vertices that we saw and edges and ratio that we saw before. So that's all the same. Now we're going to transform this data, clean up this data by making it unweighted, undirected. We're going to eliminate self loops. We're going take the upper triangular part. Here's another one that's unweighted, no self loops, different versions of them. And then we do-- you can see what those look like.

So if we look at the first one here, this just shows the unweighted, what basically making the data unweighted does to the data set. So the triangles are the original data, and just making it unweighted, how it distorts that data set. This shows you what happens when you make it undirected. So unweighted means we took any cases where we had vertices with more than one connection to them, if something had five connections, now it just gets one connection. And so that was a fairly big distortion.

A perfect example is if you take a person's social network graph, and if you were to make it unweighted, what you're saying is that the connection you have with your spouse is identical with someone that you emailed once or that you friended once. And I think we all agree that that's a fair amount of information that's lost there. And so again, encouraging folks to be aware of that and to be careful of when they're doing it. Again, making it undirected just means we basically-- if I phone you a lot or I cite you a lot in papers,

That's the same as you citing me a lot, which you lose some information there. Again, this shows the kind of distortion that we get from making it undirected. And again, this shows what happens when you do no self loops. Well, there's not a lot of self loops, so we only have affected a few vertices here. So in this case, eliminating self loops is not a terribly distorted-- doesn't really distort the data very much at all.

And then finally, this shows the upper correlation matrix. So when we correlated the two, basically multiplied the adjacency matrix together, again showing what we saw before. Moving on. You can see the plots that got eliminated from my PowerPoint. MATLAB has defeated PowerPoint's attempts to deny you your education. So again, what we're doing here is we're creating a perfect power law. This is a bigger one. I wan a lot of vertices. So this time, we had 50,000 vertices, 329,000 edges with a ratio of 6 and 1/2. We create our vertices. We randomize the edge order, et cetera.

Now we're going to randomly pick a subsample of these. And what is F samp set at? It's 1/40. So I'm going to take 1/40 of all the vertices. Now I'm going to go and compute that degree, and I'm going to basically subsample all of these. And later, what you'll see-- so let's just take a look at that. So this just shows that chart here. So this is the original data. Again, this is the vertex. We're sorting the vertices by degree here. So this is the highest degree vertex. These are the lowest degree vertex. And each vertex is getting a dot here. So we have 50,000 vertices. They all get a dot here, and we've only taken 1/40 of them. And this just shows you here-- If you only take 1/40 of them and then compute their sample, this is what you get.

Now, standard sampling theory would say aha, well, I know how to correct for this. The way I correct for this is I just multiply my sample data by 40. And we took 140, so that means whenever I measure it, the true value should be 40 times higher. So we can look at that in figure 2. So this is the true sample.

So again, we see for our high-degree vertices here-- this is the highest degree vertex here again. So this is the high-degree vertices. These are the low-degree vertices. I don't know if I said that, opposite when I mentioned it before. This is the highest degree vertex. And you see that by sampling the data, we're doing a very good job on the high-degree vertices. We're sampling them just fine. And that's why statistics works. If something is really not rare and you sample, you're going to get a good estimate.

However, for these low-degree vertices over here, what you see-- by multiplying by 40, we're significantly over-estimating their probability. As I say, this is the curve that proves that optimists and pessimists are both correct. There are so many rare things. If the world is a power law distribution, it means that there are so many rare events in the world, some of them are going to happen to you. So it means if you're an optimist, go play the lottery. If you're a pessimist, it means that lightning could hit you, and you better just stay inside. So there's just so many rare things that some really rare things are going to happen to you in your lifetime. Most likely, those rare things are very mundane.

But we can correct for this. And so we have basically a way here of deriving calculations. So we compute the parameters of our distribution, and through these two functions, compute degree correction, and apply degree correction. We can actually go back and say all right, given that we believe the data is power law and we've sampled it, can we then come up with a more uniform correction that basically gives us a better estimate that works at both the high and the low end? And that's what you see here. So basically, we haven't changed. The correction hasn't changed here. But we've downgraded these lower ones.

And essentially, what we're doing is instead of just using the average as the statistic, we're using the median. So we're using, essentially, a quantile-based correction here, a 50th percent quantile-based based correction here. And that causes us to lower the estimates of these vertices. And so it would be a better estimate and allow you to do the sampling of that. Very good.

And our final demo. So now what we're doing is power law fitting. And so we have the routines for doing that. So again, here's our distribution. It's a power law of 1.3. We've set D max to be 2,000 and about 60-ish bins. We create our parallel distribution. It has 50,000 vertices, 329,000 edges. Ratio is the same as the one we did before. We're going to make it undirected-- undirected and unweighted, undirected, unweighted, no self loops, so standard corrections that we do.

We're going to compute the degree distribution of that data and plot it. Or actually, get it there now. I'm going to then-- we have this function called power law fit. So if I compute-- so I compute the degree distribution. So we have this function called out degree which gives us the distribution. And I can find, essentially, the number of values with the one, and the one that-- our maximum, so this is estimating our poor man's slope. So we're computing the slope. We're counting the total number of edges here, and then we have this function called power law fit. Basically, we can plug in what the estimated alpha is, what the number of vertices is, and the number of edges is to find our best fit distribution.

So this basically inverts those formulas I showed you, which is given a degree distribution that sums to a particular number of vertices and sums to a particular number of edges, can you give me a new D max and a new ND, these parameters that don't really have as much meaning, to do that? And so basically, we use, essentially, a combination of three different techniques here. Because this is so nonlinear, and there's this-- basically, remember I talked about integer bins and logarithmic bins?

Well, if you look at that plot, it showed there's that bending. It's a very nasty manifold, the surface of this function. And it has a continuous part and a discrete part. So what we do here is we do essentially a sampled search where we randomly sample, looking for a location. We do a heuristic search, which is a simulated [INAUDIBLE] search. And we also use Broyden's nonlinear-- essentially a variation of Newton's method to all try and find the best set of parameters that will fit this data.

We rarely get an exact match. But you can see here it's choosing different ones. And this gives you how it's doing. From the sample search, this shows you the number of vertices and the number of edges it was able to achieve. The heuristic search didn't do very well at all, and then the Broyden search did a pretty good job, and it got us pretty well. So actually, it ended up comparing all of these, and it ended up choosing the sample search-- this one, this first one I did. It liked that best of all.

So we'll look at that here. So Figure 2 was the original data. Figure 1 shows you the manifold of this space. And so plotted in this coordinate system of n versus m, the dot is the dot that it found. It's actually here in this [INAUDIBLE] these lines show the boundaries. But you can see this very nonlinear manifold here. This is the continuous regime. This is the integer regime. It cusps right at the transition. Again, a very nasty function to try and invert, which is why we used all those different techniques to invert it.

And then Figure 3 shows the results. So this black line shows the original model input that we provided. So that was the true model. The circle shows the data after it was transformed. We made it undirected, unweighted with no self loops. That's this. Alpha is-- this is our poor man's alpha. What you can see is almost identical to the original model. So the poor man's alpha does a very good job of fitting in this case. The triangle shows the model fit. So when we fit the data, we came up with that best fit it shows, and we then created a new distribution. This is what it looked like. And then the plus sign shows us rebinning that data onto the bins from the model. And you see that we've done a very nice job of recovering the original power law distribution even after we did that distortion.

So again, that is the last example. So I want thank you. And then for the homework, a lot of you did Homework 2, which is great. I think Homework 3 wasn't such a great hit. I'm going to definitely rethink that one. But the next homework does not require you to have done homework 3. If you did homework 2, basically, it's just saying compute a degree distribution.

And in fact, you don't even need to have done Homework 2. You can compute a degree distribution on your Homework 2, or you can compute a degree distribution on any data set. For example, today is Halloween. If you go trick or treating with somebody-- yourself, your children, somebody else's children, you can maybe email me the histogram of your candy and plot it on a degree distribution, and maybe compute the poor man's alpha coefficient from that. And the other coefficients from that would be a fun exercise to do. So that'll be the next homework. I'll email that out in the next couple of days.

So look forward. Again, if you send me the homework prior to next-- actually, just a reminder, no class next week. This room has been taken. But again, if you email me the homework prior to this time next week, I will give you feedback on it. You can still send me the homework after that. I just won't give you any feedback on it. So thank you again, and look forward to seeing you in two weeks.

Free Downloads

Video

• iTunes U (MP4 - 56MB)
• Internet Archive (MP4 - 56MB)

Caption

• English-US (SRT)