The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JEREMY KEPNER: Welcome.
Happy Halloween.
And for those of you watching this video at home,
you'll be glad to know the whole audience has joined me
and they're all dressed out in costumes.
And it's a real fun day here as we do this.
So making me not feel alone in my costume.
A lot of moral support there, so that's great.
So, yeah.
So this is Lecture 05 five on Signal Processing on Databases,
just for a recap.
For those of you who missed earlier classes
or are going this out of order on the web,
signal processing really alludes detection theory,
finding things, which alludes to the underlying
mathematical basis of that, which is linear algebra.
And the databases really refers to working
with unstructured data, strings, and other types of things.
Two things that aren't really talked about together,
but we're bringing them together here
because we have lots of new data sets that required it.
And so this talk is probably the one that's
getting most into something that we would say
relates to detection theory, because we're
going to be dealing with a lot with background data models.
And in particular, power laws and methods
of constructing power law data sets,
methods on sampling and fitting, and using that as a basis
for doing the kind of work that you want to do.
So moving in here.
So just your outline.
And so we've got a lot of material to go over here today.
This is all using the data set, when
we get into the data set that we talked
about in the last lecture, which is this Reuters data set.
So we'll be applying some of these ideas to that data set.
So just going to-- introduction here,
and then I'm going to get to sampling
theory, and subsampling theory, various types of distributions.
And then end up with the Reuter's data set.
The overall goal of this lecture is
to really develop a background model for these types of data
sets that is based on what I'm calling a perfect power law.
And then we're going to, basically,
after we can construct a perfect power law,
we're going to sample that power law
and look at what happens when we sample it.
What are the effects of sampling it?
And then we can use the power law
to look at things like deviations and such.
Now you might ask, well, why are we
so concerned about backgrounds and power laws?
And it's because here's sort of the basis of detection
theory on one slide.
So in detection theory, you basically
have a model which consists of noise and the signal,
and you have two hypotheses, H0, which
is there's only noise in your data,
and H1 that there's signal plus noise.
And so, essentially, when you do detection theory, what you're
doing is given these models, you can then
compute optimal filters for answering the question,
is there a signal there or is it just noise?
That's essentially what detection theory boils down to.
In this data, now when we deal with graph theory,
it's obviously not so clean in terms of our dimensions here.
We'll have some kind of high dimensional space
and our signal will be projected into
that high dimensional space.
But nevertheless, the concept is still just as important
that we have noise and a signal.
And that's what we're trying to do here.
Detection theory works in the traditional domains
that we've applied it to because we
have a fairly good model for the background, which tends
to be Gaussian random noise.
It's kind of the fundamental distribution
that we use in lots of our data sets.
And if we didn't have that model,
it would be very difficult for us to proceed
with much of detection theory.
And the Gaussian random noise model
works, because in many respects, if you're
collecting sensor data, you really
will have Gaussian physics going on.
You also-- law of large numbers in terms
of if you have lots of different distributions
that are pulled together, they will end up
beginning to look like a Gaussian as well.
So that's what we have in many of the traditional fields
that we've worked in in signal processing,
but now we're in this new area where a lot of our data
sets arrive from artificial processes, processes that
are a result of human actions.
Be it data on a network, be it data in a social network,
be it other types of data that have a strong sort
of artificial element to them.
We find that the Gaussian model does not
reveal itself in the same way that we've seen in other sets.
So we need this.
We really need a background model,
so we have to do something about it.
So there has been a fair amount of research and literature
to come up with first principles, methods
for creating power law distribution in data sets.
We talked a little bit about these distributions
in the previous lectures.
And they've met with mixed results.
It's been difficult to come up with,
what is the underlying physics of the processes
that result in certain vertices in graphs
having enormous number of edges and other vertices
only having a few?
And so there has been work on that.
I encourage you to look at that literature.
Here we're going to do much more of a-- sort of go
from the reverse direction, which
is let's begin by coming up with some way to construct
a perfect power law.
With no concept of well, what is the underlying physics
motivating this?
Essentially, a basic linear model for a perfect power law.
That we probably can do.
And then we'll go from there.
And linear models are something that we often
use in our business.
And it's a good first starting point.
So along those lines, this is a way
to construct a perfect power law in a matrix.
This is basically a slide of definitions
here, so let me spend a little time going through them.
So we're going to represent our graph or our data
as a random matrix.
Basically of zero where there's no connection between-- this
is a set of vertices connected with another set of vertices.
There are N out of these vertices and N
in of these vertices.
You have a dot here.
The row corresponds to the vertex at the edge left,
and the column corresponds to the vertex
that the edge is going into.
So this adjacency matrix A is just
going to be constructed by randomly filling
this matrix with entries.
And the only real constraint on them is that when you sum A,
we're going to allow multiple edges.
So you can have more-- the values aren't just
zero and one, but they can have more than that.
But when you sum the matrix A, all its values up,
you get a value M, which is the total number of edges
in the graph.
So we have essentially a graph with-- this could
be a bipartite graph or not.
But with essentially N out vertices, N in vertices, and M
total edges.
The perfect power law, we're going
to have essentially two perfect power laws here.
One on the out degree.
So if you sum these rows, and then you do a histogram,
you're going to want to produce a histogram that
looks something like this.
So you have an out degree for each vertex.
And this would show how many vertices have that out degree.
And so our power law says that these points
should fall on a slope with a negative power law
coefficient of alpha out.
So that's essentially the one definition there.
And then when you likewise have another going
in the-- for the other degree.
So the in degrees have their own power law.
So these are the definitions.
This is what we're saying is a perfect power law.
So we're saying a perfect power law has these properties.
Now we're going to attempt to construct it.
We have no physical basis for saying why
the data should look this way.
We're just saying this is a linear model of the data
and we're going to construct it that way.
And again, these can be undirected, multi-edge,
we can allow self-loops and disconnected vertices,
and hyper-edges.
Anything you can get by just randomly throwing down
values onto a matrix.
And again, the only constraint is
that the sum in both directions is
equal to the number of edges.
So given that, can we construct such a thing?
Well, it turns out we can construct such a thing fairly
simply.
And so in MATLAB we can construct a perfect power law
graph with this four line function here.
It will construct a degree distribution
that has this property.
And the three number parameters to this distribution
are alpha, which is the slope, dmax,
which is the maximum degree vertex,
and then this number Nd, which is roughly
proportional to the number of bins
that we're going to have here.
Essentially the number of points.
It's not exactly that, but roughly proportional to that.
And so when you do this little equation here,
the first thing that you will see
is we are going to be creating a logarithmic spacing of bins.
We kind of need to do that here.
But at a certain point, we get below a value
where the spacing will be-- we'll have essentially one
bin per integer.
And so these are two very separate regimes.
You're going to have one which is called the integer
regime, where basically each integer has
one-- is representative.
And then it transitions to a logarithmic regime.
You might say this is somewhat artificial.
It's actually very reflective of what's really going on.
We really see here-- we really have a dmax,
we really have, almost always, a count at 1.
And then we have a count at 2 or 3,
and then they start spreading out.
And so this is just an artificial way
to create this type of distribution,
which is a perfect power law distribution.
So it's a very simple, very efficient code
for creating one of these.
It has a smooth transition from what we call the integer
bins and the logarithmic bins.
And it also gives a very nice what we call poor man slope
estimator.
So there's a lot of research out there
about how do you estimate the slope of your power law.
And there's all kinds of algorithms for doing this.
Well, the simplest way is just to take the two endpoints.
Take the first point and the last point,
and you know you're perfectly fitting two points.
And you could argue you're perfectly fitting
the two most important points.
And you get this nice, simple value for the slope here.
In addition, you can make the argument
that regardless of how you bin the data,
you'll always have these two bins.
You will always have a bin at dmax
and you will always have a bin at 1.
And all the other bins are going to be somewhat
a matter of choice, or of fitting.
And so again, that's another reason to rationalize alpha.
So I would say, if you plot your data
and you have to estimate an alpha,
then I would just say, well, here's what you do.
And it's as good an estimate of alpha as any.
And it's very nicely defined.
So we call that-- when we talk about estimating
the slope here, this is the formula we're going to use.
So far, this code has just constructed
a degree distribution, i.e., the degree, and then
the number of vertices with that degree
will be the outputs of this perfect power law function.
We still have to assign that degree distribution
to an actual set of edges.
OK?
And here's the code that will do that for.
It will say, given a degree distribution,
it will create a set of vertices that do that.
Now, the actual pairing of the vertices into edges
is arbitrary.
And in fact, these are all different adjacency matrices
for the same degree distribution.
That is, every single one of these
has the same degree distribution in both the rows
and the columns.
So the actual which vertices are connected to which
is a second order statistic.
So the degree distribution in this first order [INAUDIBLE],
but how you want to connect those vertices up
is somewhat arbitrary.
And so that's a freedom that you have here.
So for example, if I just take the vertices out of here
and I just say, all right, every single vertex in your list,
I'm just going to pair it up with yourself,
I will get an adjacency matrix that's all diagonals.
Essentially, all self-loops.
If I take that list and just randomly
reorder the vertex labels themselves,
then I get something that looks like this.
If I just randomly reorder the edge pairs,
I get something like this.
And if I randomly relabel both the vertices
and reconnect the vertices into different edges,
I get something like this.
And for the most part, when we talk about our randomly
generating our perfect power laws,
we're going to talk about this.
Which is probably most like what we really encounter.
It's essentially something that's
equivalent to randomly labeling your vertices,
and then randomly taking those vertices
and randomly pairing them together.
So that basically talks about how
we can actually construct a graph from our perfect power
law edge.
Now, so this is a forward model.
Given a set of these three sort of parameters,
we can generate a perfect power law.
But if we're dealing with data, we often
want slightly different parameters.
So as I said, before our three parameters were alpha,
which is greater than 0, dmax, which
is the highest degree in the data, which we're saying
is greater than 1, and then this parameter Nd, which roughly
corresponds to number of bins.
So we can generate a power law model for any of these values
here that satisfy these constraints.
So that's a large number.
However, what we'll typically see
is that we want to use these parameters.
So we'll want to have an alpha, a number of vertices,
and a number of edges, is more often the parameters we
want to work with.
And we can compute those by inverting these formulas.
That is, if we compute the degree,
we can sum it to compute the number of vertices.
And likewise, we can sum the distribution times the degree
to get the number of edges.
So given these, an alpha and our model, we can invert these.
All right?
And what you see here is for a given value of alpha,
the allowed values of N and M, given--
that is, the values of N and M that will be a power law.
So what you see is that not all combinations of vertices,
vertex count and edge count, can be constructed in a power law.
There's a band here.
This is a logarithmic graph.
It's a wide band.
But there's a band here of allowable data
that will produce that.
And typically, what you see is kind
of the middle of this band is like a ratio of around 10,
which happens to be the magic number that we see
in lots of our data sets when people say,
I have power law data.
Someone will ask you, well, what's your to vertex to ratio?
And [INAUDIBLE] we say, it's like 8, or 10, or 20,
or something like that.
And again, you see it's because in order for it
to be power law data, at least according to this model,
it has to fall into this band here.
You'll also see this is a very nonlinear function here.
And we'll get into fitting that later.
And it's a nasty, nasty function to invert,
because we have integer data, and data
that's almost continuous.
And it's a nasty, nasty bus-- we can do it,
but it's kind of a nasty business.
But given an alpha and an N that are consistent,
we can actually then generate a dmax and an Nd that
will best fit those parameters.
So let's do an example here.
So I didn't just dress up in this crazy outfit for nothing.
We have a whole Halloween theme to our lecture today.
So this is-- when I go trick or treating
with my daughter, of course, our favorite thing
is to do the distribution of the candy when we're done.
And so this shows last year's candy distribution.
We'll see how it varies.
As you can see, Hershey's chocolate
bars, not surprisingly, extremely popular.
What else is popular here?
Swedish fish, not so popular.
Nestle's Crunch bars, not so popular.
Again, I actually found this somewhat-- this list
hasn't change since when I went trick or treating.
This is a tough list to break into.
Getting a new candy that makes it to Halloween-worthy candy
is pretty hard.
So this year shows the distribution of all the candy
that we collected.
And here are some basic information.
So we had 77 pieces of candy, or distinct edges.
We had 19 types of candy.
Our edge to vertex ratio was 4.
The dmax was 15.
So we had 15 Hershey's Kisses.
N1, we had eight types of candy that we only got one of.
And then our power slope was alpha.
And then our fit parameters to this,
when we actually fit, where we got 77, 21, and M/N of 3.7.
And this shows you the data.
So this is the candy degree.
And this is the number.
And this shows you what we measured.
This is the poor man's slope here.
This is the model.
And then one thing we can do is actually--
which is very helpful-- is we can
re-bin the measured data using the bins extracted
from the model.
And that gets you these red x's here, or plus signs here.
And we'll discover that's very important.
Because the data you have is often
very rarely binned in a way that's
proper for seeing the proper distribution.
And we can actually use this model
to come up with what a better set of bins
would be, and then bin the data with respect to that.
So that's just an example of this in actual practice.
So now that we have a mechanism for generating perfect power
law, let's see what happens when we sample it.
Let's see what happens when we do
the things to it that we typically
do to clean up our data.
I bring this up because in standard graph theory, as I've
talked about in previous lectures,
we often have what we call random,
undirected Erdos-Renyi graphs, which
are basically vertices without direction.
And usually the edges are unweighted.
So we just have a 0 or a 1.
So very simplified graphs.
I'm actually going to-- getting a little hot here
in the top hat.
So a lot of our graph theory is based on these types of graphs.
And as we've talked about before that our data tends
to not look like that.
So one of the things we do so that we
can apply the theory to that data is that we often
make with data look like that.
So we'll often make with data undirected.
We'll often make the data basically unweighted
and other types of things so we can apply all the theory
that we've developed over the last several decades
on these particular types of very well studied graphs.
So now that we have a perfect power law graph,
we can see what happens if we apply those same corrections
to the data.
And so here's what we see.
So we generated a perfect power law graph.
The alpha is 1.3.
The dmax was 1,000.
Our Nd was 50.
This generated a data set with 18,000 vertices and 84,000
edges.
And so here's a very simple way to make it.
We're going to make it undirected by basically taking
the matrix and adding its transpose,
and then taking the upper diagonal.
This is actually the best way to make an adjacency matrix
undirected, to take that upper portion,
because a lot of the statistics that-- basically it
saves you having to deal with a lot of annoying factors of 2.
So a lot of times, we'll just do A plus A transpose,
but then you get these annoying factors of 2 lying around.
And so this is a way to sort of not do that.
So we're getting rid of-- we've made it undirected.
We're made it undirected by doing that.
We're going to make it unweighted by basically
converting everything to a 0 or 1, and then back to double.
So that makes it unweighted.
And then we're getting rid of the diagonal,
so that eliminates self-loops.
So we've done all these things.
We've cleaned up our data in this way.
So what happens?
Well, so the triangles were the input model.
Well, now we've cleaned up our data,
and we see this sort of mess that we've done to our data.
And in fact, I'll call this-- in keeping with our Halloween
theme here, we'll call this our witch's broom distribution
here.
And if anybody's looked at degree redistributions on data,
it will be like you'll recognize this shape instantly
because you have this bendiness coming up here.
And then sort of fanning out down here.
Very common thing that we see in the data sets that we plot.
And in fact, there's not an insignificant amount
of literature devoted to trying to understand these bumps
and wiggles, and do they really mean something
underlying about the physical phenomenon that's taking place.
And while it's the case that those bumps and wiggles may
actually be representative of some physical phenomenon,
based on this, we also have to concede
the fact it's also consistent with our cleaning up procedure.
That is, the thing we're trying to do to make our data better
is introducing nonlinear phenomenon
on the data, which we may confuse with real phenomena.
So this is very much a cautionary tale.
And so based on that, I certainly
encourage people not to clean up their data in that way,
and keep the directedness.
Don't throw away the self-loops.
Keep the weightedness.
Do your degree distributions in this way.
And live with the fact that that's what your data really
is like and try and understand it that way,
rather than trying cleaning up in this way.
Sometimes you have no choice.
The algorithms that you have will only
work on data being that's been cleaned up in this way.
But you have to recognize you are
introducing a new phenomena.
It's a highly non-linear process, this cleaning up,
and you have to be careful about that.
However, given that we've done this,
is there a way that we can recover the original power law?
So we can try that.
So we have here is the data that we-- the original data that we
cleaned up is now these circles.
OK?
And we're going to take that data set
and compute an alpha and an N and M from it
from using pour inversion formulas.
And then compute what the power law of that would be.
So that's the triangles.
So here's our poor man's alpha fit.
This is our model.
This is what the model is.
These triangles here is the model, we're saying.
And then we can say, aha, let's use the bins that
came from this model to re-bin these circles
onto these red plus signs here.
So that's our new data set.
And what you see is that we've done
a pretty good job of recovering the original power law.
So if we had data that we observed to look like this,
we wouldn't be sure it was a power law.
Like, we don't know.
Say, well, what's this bend here?
And what's this fanning out here?
But then if you go through this process and re-bin it,
you can be like, oh, no, that really looks like a power law.
And so that's a way of recovering the power
law that we may have lost through some filtering
procedure.
Here's another example.
So what we're going to do is essentially take our matrix
and compute the correlation of it.
We talked a lot about-- if we say
I have an incidence matrix, we multiply it to do correlation.
In this case, we're treating our random matrix
as not an adjacency matrix, but as
an incidence matrix, a randomly generated incidence matrix.
And so these are, again, the parameters that we use.
We're converting it to all unweighted, all 0's and 1's.
And then we are correlating it with itself
to construct the adjacency matrix.
Taking the upper diagonal, and then removing the diagonal.
And this is the result of what we see.
So here's our input model.
Again, the triangles.
And then this is the measured-- what we get out from there.
And if you saw this, you might be like, wow,
that's a really good power law.
In fact, I've certainly seen data-- I mean, most the time
I would see, yep, that is a power law distribution.
We absolutely have a power law distribution.
However, we then apply our procedure.
So again, we have our measured data.
OK, we're going to do our parameters here.
Get our poor man's alpha parameter.
And then fit, the triangles are the new fit.
OK.
And then we use those bins to re-bin.
And we see here that when we actually re-bin the data,
we get something that looks very much not like a power law
distribution.
So there's an example of the reverse.
Before we had data that didn't look like a power law,
but when we re-binned it, we recovered the power law.
Here we have data that may-- just
sort of in this random binning may look like a power law.
We actually see it has this bump.
And then continuing with our Halloween theme,
we can call this the witch's nose distribution,
because it comes along here as this giant bump
and then goes back to a power law.
And this actually, there's meaning for this.
And we will see this later in the actual data.
But this is not just that when you do these correlation
matrices, certain types of them, particularly self-correlations,
very likely will produce this type of distribution.
But again, even though we have this bump,
you would still argue that our linear power law is still
a very good first order fit.
So we still captured most of the dynamic range
of the distribution.
And this is now a delta from that.
And so we're very comfortable with that, right?
We start with our linear models.
That models most of the data.
And then we have a second order.
If we wanted to, we could go in and come up
with some kind of second order.
Subtract the linear model from here
and you would see some kind of hump distribution here.
And you could then model your data as a linear model,
plus some kind of correction.
Again, very classic signal processing way
to deal with our data, and certainly seems
as relevant here as anywhere else.
Let's see here.
And again, so the power law can be preserved
as we talked about there.
So moving on, another phenomenon that's
often documented in the literature
is called the densification.
In fact, there's many papers written on what
is called densification.
This is the observation that if you construct a graph,
and you compute the ratio of the edges
to vertices over time, that ratio will go up.
And there's a lot of research talking
about the physical phenomenon that might
produce that type of effect.
And so while that physical phenomenon might be there,
it's also a byproduct of just sampling the data.
So for instance here, what we're going to do
is we created our perfect power law graph
and we're going to sample it.
We're basically going to take subsamples of that data.
And we're going to do it in little chunks, about 10%
of the data at a time.
And the triangles and the circles
show when we look at each set of data independently.
And then we have these lines that
show what happens when we do it cumulatively.
We basically take 10% of the data, then 20% of the data,
then 30% and move on here.
And we have two different ways of sampling our data here.
Random is, I'm just taking that whole matrix
and I'm randomly picking edges out of it.
And what you see is that each sample
has a relatively low edge to vertex distribution.
But as you add more and more and more up, it gets denser.
And this is just simply the fact that given
a finite number of vertices, you eventually,
if you kept on adding edges and edges and edges,
eventually it would go-- this would become infinite.
If you add an infinite number of edges
to a finite [INAUDIBLE] vertices,
then it will get denser and denser and denser and denser.
And this is sort of a byproduct of treating these as 0 and 1's.
And not recognizing.
So this just naturally occurs through sampling.
The linear sampling here is where, basically, I'm
taking whole rows at a time.
Or I could have taken whole columns at a time.
So I'm taking each row and eventually, all right, I'm
dropping them down.
And there you say it's constant.
So I'm basically for each-- I'm essentially
taking a whole vertex and adding it at a time.
And here the sampling is somewhat independent of it.
The densification-- if you sample whole rows,
the density of that will be the same as if you did it.
So this is just good to know.
And good to know about sampling and these phenomena
can take place, and then how sampling
can play an important role in the data that we observe.
Another phenomenon that's been studied extensively
is what happens to the slope of the degree distribution
as you add data.
And again, we do this exact same type of sampling.
And you see here that when we just-- this data had a slope,
I believe, of 1.3.
And you can see if we sample it randomly, just
take random vertices, that the slope starts out very,
very high.
And each sample stays high.
But when we start accumulating them,
they start converging on the true value-- converging
from above.
And likewise, when you do linear sampling,
you have a direction in the opposite way.
And they both end up converging onto the true value here
of 1.3.
And so again, this just shows that the slope of your degree
distribution is also very much a function of the sampling.
It could also be a function of the underlying phenomenon.
But again, a cautionary tale that one
needs to be very aware of how one's sampling.
And again, these perfect power law data sets
are a very useful tool for doing that.
So if you have a real data set and you're
sampling in some way, and you want
to know what is maybe real phenomenon versus what
is sampling effects, if you go and generate a perfect power
law that's an approximation of this data set,
you can then very quickly see which phenomena are just
a result of sampling a perfect power law,
and which phenomena are maybe indicative of some deeper
underlying correlations between the data.
So again, a very useful tool here.
Moving on, we're going to talk about subsampling.
And one of the problems that we have is very large data sets.
And often we can't compute the degree distribution
on the entire data set.
Or we can't-- and this has sort of been a bread and butter
of signal processing for years, where if we want to compute
a background model, that we don't just simply sum up all
the data.
That we randomly select data from the data
set, and we use that as a model of our background.
And that's a much more efficient way,
from a computational and data handling perspective,
than simply computing the mean or the variance
based on the entire data set.
So again, we need good background estimation in order
to do our anomaly detection.
And again, it's prohibitive to traverse the data.
So the question is, can we add accurately
estimate the background from a sample?
So let's see what happens.
We have a perfect power law.
We can look at what happens when we sample that.
So we've generated a power law.
OK.
And this is not-- I've changed-- this may look like the degree
distribution, but it's actually a different plot.
So this is showing every single vertex in the data set.
All right.
And this shows the N degree of that vertex.
And we've sorted them.
So the highest degree vertex is over here
and the lowest degree vertex is over here.
OK.
So this is all the vertices.
And so this is the true data.
And this is what happens when we take a 1/40 sample.
I just say, I'm only going to take 1/40 of the edges.
What does it look like?
Some relatively simple math here, which I won't go over.
But it's there for you.
We can actually come up a correction
that allows us to sample that data based
on the median distributions.
I apologize for the slides.
We will correct them for the web when we go out there.
All right.
So moving on.
So one of the things we talk about sampling,
and we talk mainly about single distributions.
We can also talk about joint distributions.
So we can actually use the degree
as a way of labeling the vertices and look at them.
So it's a way of compressing-- if we just say,
we're going to label each vertex by its degree,
this is a way of compressing many vertices into a smaller
dimensional space way to look at that.
And we can then count the correlations.
We can look at the distribution of how many edges are there
from vertices of this degree to vertices of that degree?
And so it's a tool for projecting our data
and understanding what's going on.
And we can also then re-bin that data
with a power law, which will make it more easily understood.
So if we look here, we see that we had the degree distribution.
So this shows us for a perfect power law data,
for vertices with this degree in and this degree out, how many
edges there were between them.
And as you see here, obviously, there
was a lot of vertices here between low degree edges,
and not so much here.
But this is somewhat a misnomer because
of the way the data comes out.
We're not really binning it properly.
However, if we go and fit a perfect power law to this data,
and then pick a new set of bins based
on those, so re-bin the data, we can see here
that we get a much smoother uniform distribution.
So while here we may have thought
that this was an artificially low dense-- not dense region,
this was artificially high-- what
you see here is when you re-bin this
that there's a very smooth distribution from what
we expect for our perfect power law here.
That this is a fairly uniform distribution
with respect to that.
And so basically, it essentially puts more bins
where we actually have data as opposed
to wasting bins over here where we don't really have any data.
Using our perfect power law model,
we can actually compute analytically
what this should be.
So here's an example of what that looks like.
And again, very similar to what we see.
And given the data, and a model for the data,
we can then compute the ratio of the observed to the model
to get a sense of what data is unusual
versus what we expect from a perfect power-- or linear fit.
And we see that very clearly here.
So this is the data just dividing the data by the model
here.
And you see again, you see all this data
appears like this is the ratio, the log of the ratio.
And so it's a-- zero means that it's essentially
the ratio is 1, so all of this is expected.
And then we see all these fluctuations here.
Things that are higher than we expected
and things that are lower than we expected.
And so this is the classic-- the time where you see this most is
whenever they show you a map of the United States by county
of some-- maybe like it's-- a classic is like a cancer
cluster, or heart disease, or any phenomena.
And what you see is that certain counties in the western part
of the United States are extremely healthy,
and certain counties are just deadly.
And it's just a fact that they're
very sparsely populated.
And so they are dealing with small numbers effect here.
So basically, this is just showing you oscillations
between 0 and 1 here.
So what we call Poisson sampling.
And so it makes it very difficult
to know what those are.
However, if we re-bin the data and then divide by the model,
we see that the vast majority of our data, as expected,
is in this normal regime.
Another thing we can actually do is
we can look at like the most unexpected data set.
So we can look at the most typical data set.
Oh, these got moved here, didn't they.
I don't know what's going on with PowerPoint today.
But this shows us the surpluses, the deficits,
and the most typical.
So here's like the most extreme bin, the most underrepresented
bin, and sort of the most average bin.
And these are the areas that they correspond to
in the real data set.
So you can use this to find extremes
based on a statistical test.
And again, you see that here again.
We have these different-- that shows
what the measured over expected is versus measured.
And you can see, this is the original data set
and this is the re-bin data set.
And you see that the re-binning removes
a lot of these very sparse points over here
and gives you very narrow distribution around what you
expected.
You can actually go and find selected edges if you want.
So this just shows the different types of edges.
The maximum.
If you actually wanted to go and look at those edges,
these would be maximum, these would be the minimum,
and these are the other types.
So it's a useful thing.
You can, again, go, say all right,
we found that if we have an artificially high correlation
between vertices with this degree and this degree,
we can then backtrack and find out
which specific vertices there are, and see if that's
anything that's interesting.
We can also use this plot to look at these questions of edge
order.
And so, hopefully, this will work today.
So here's is a-- if I randomly select vertices
and I compute their degree versus-- so basically, what
we did before.
We subsample and then compute over what is expected.
And we play this.
And you can just see here we get this sort of-- when we randomly
select the vertices, we basically
get kind of a typical-- each sample
looks very much the same.
Again, up there in these high degrees,
we have this Poisson sampling effect
where we have some-- this is we have
no vertices, so we get a 0, which is lower than expected.
And if we have like two vertices,
we get higher than expected.
And so again, we still have that Poisson effect.
Interestingly, if we do linear sampling, which
we take whole rows at the time, we
get a very different type of phenomenon.
So you see you get these-- whenever
you do run into a high degree row, by definition,
it is unusual.
Which again, means you have to be very careful.
You're going to run into this high degree row
eventually by sampling, but you have to be careful.
It's like, oh, my goodness.
This is a very, very unusual type of thing.
So again, cautionary tale about sampling.
All right.
So we've talked a lot about the theory here.
Let's get into some real data.
So this is our Reuter's data again.
I showed it to you the other day.
The various document distributions we had.
In this case, 800,000 documents, 47,000 extracted entities,
for a total of 6,000,000, essentially, edges.
So it's a bipartite graph [INAUDIBLE] between documents.
And four different types.
And we can now look at the different degree
distributions of the different classes and see what we have.
So the first one we want to look at are the locations.
And so we look at the distribution of the documents
and the entities.
So basically, imagine we took-- so we're very clear here.
So what I'm doing is just taking this part of the matrix,
and the distribution of the documents is summing this way.
And the distribution locations is summing up and down
along the columns.
So for each one of these types, we
can do those different types.
We have, essentially, two different degree distributions.
One associated with the documents and one
associated with the entities.
So this shows us our document distribution.
So we have the measured data.
We have our fit.
We have our model, and then our re-binning.
And you know, you could say that this is approximately a power
law, and that when you re-bin it,
that this sort of S-shaped effect
is still there, which probably means it's really there.
There's something going on in the data that really
is making this bowing effect.
And so that's really there.
Likewise here, we have our model.
We have the measured data.
We have our model, alpha, which is the blue line.
We have the model fit.
And then our re-binning.
And again, you could say the power law
model's a pretty good fit, but again, we
have something going on here.
Some kind of phenomenon going on there.
And we can then do this for each type.
We can look at the organizations.
And again, we don't have as many organizations.
Again, similar types of phenomenon here.
This is so sparse, it's difficult to really say
what's going on here.
Of course, we do people, which is, of course,
people always are the first thing
that you talk about when you talk about power law
distributions.
And again, very nicely, we have our measured data and our fit.
And again, we see this sort of bent broom distribution
here, bending, and then with this fan out effect here.
But then when we model it and re-bin it,
we get something that looks much more like a true power law.
And you can see that very nicely here
with the actual person data.
A very good power law.
So this tells us that regardless of what this underlying data
looks like, this probably really is a power law distribution.
And then we have the times.
And again, a similar type of thing here.
Different types of things.
And we actually have a little spike here.
The Reuter's data has a certain collection
of times associated with the actual filing of events.
There's only 35 of them, so we do get a little spike here,
which is actually what we expect.
Although you wouldn't see that really clearly
in the true data, but when you re-bin it,
this bump comes out fairly clearly.
So again, proper binning extremely important.
We can look at our correlations as well.
So let's just look at the person-person correlations.
And again, what we saw here is sort of-- we
see this is the raw data.
So something that looks very much like a power law.
But when we go through our re-binning process,
we see kind of this witch's nose effect here really in the data.
So this would tell you to first order it's a power law,
but you actually have this correlation taking place here,
right here.
So that's something that really seems to be taking place.
You see the same thing when we do time.
We can look at document.
Let's now look at sampling.
So again, this is the same sampling that we did.
We're going to now basically look
at the document densification.
So this is selecting whole rows.
We select a whole document, so we're getting a whole row.
And this just shows you the four different types here.
And you see that they behave exactly as expected.
Each individual sample is reflective
of the overall densification, because you're
taking whole rows.
OK?
Now let's take entities.
So now we're cutting across these.
And now you see something that looks
much more like random sampling.
When you randomly select an entity,
that's sort of a random set of documents.
And again, the individual samples,
but when you start summing them up,
you see that they get denser and denser and denser.
So individual document has a sort
of a good sample of the overall distribution.
When you pick a person, as you take a higher
fraction of the data, you're going
to get more and more and more with that.
So again, all consistent with what we saw before.
A little noisier to see here, but trust me,
the power law exponent also behaves exactly as we expected.
Likewise, this is essentially a linear sampling,
and here's the random sampling behaving exactly
as we expected.
We can look at the joint distributions.
So here's the actual-- this is the location cross-correlation.
So they're showing you basically the document versus entity
degree distributions.
So this is the measured.
And it's been re-binned.
This is measured divided by the expected,
so the expected re-bins.
This is measured divided by the model.
And here's the model, and expected divided by the model.
And you can basically compare all these different types
to create different statistical tests.
And actually, we can find here the measured re-binned divided
by the expected re-binned.
We can get our surpluses and deficits and other types
of things in the actual data.
And so here you see something that
maybe looks like an artificially high grouping here,
some artificially low, and some expected.
And you can then use these as ways
to actually go and find anomalous documents.
Or documents are like-- you say, look,
this is the most typical type of thing.
I mean, people are like, well, why would you try and find
the most normal?
Well, a lot of times, you want to be able to,
in terms of summarization, be like here's
a very representative set of documents.
They have the statistical properties.
This is very consistent with everything else.
Because people will [INAUDIBLE] like what that mean to be that?
So again, a very useful way to look at the data.
Do the same thing with organization.
Again, basically get the measured re-binned,
the measured divided by the expected,
the expected re-binned, the model, refit the model,
and the various different ratios of all of them, which
you can do to find various outliers and such.
Again, a very useful.
Interesting that it picked the most representative one
way up here.
So that's an interesting-- so here's a very unusual
representative sample.
Persons, you can do the same thing.
And I guess one of the things I'm also trying to do
is that you see that these all don't look the same.
Right?
Hopefully, from looking at these, it's like this one,
the location distribution is clearly
a little similar to the organization distribution,
but the person distribution looks very different.
And the time distribution looks very different.
So you see that you have to be careful about-- sometimes we'll
just take all these different categories
and lump them together in one big distribution.
And here's a situation, like these are really
pretty different things.
And you really want to treat them
as four distinct classes of types of things.
So as an example of selected edges,
this just shows the most typical.
This just shows the various entities that were selected,
and very representative document.
And this is a very low degree, various entities.
So this might show you entities-- here's
the kind of surplus example.
And we could go in and actually finding those edges.
Same with the person.
Very generic people, higher degree.
Here's a very low.
So this person, Jeremy Smith, is very unusual in terms
of what they connected with.
Surplus ones here.
Can't really read that over there, but just
to give you examples of things that you can actually
use this to go in and actually find things.
All right.
So that brings again to the lecture part.
So just, again, developing this background model's
very important for the graphs.
Based on the perfect power law, gives us
a very simple heuristic for creating a linear model.
We can then really quantify the effects of sampling,
which is very important.
Again, traditional sampling approaches
can easily create nonlinear phenomenon
that we have to be careful of and be aware of.
It allows us to develop techniques
for comparing real data with power law fits
that we can then use as statistical tests
for finding unusual bits of data.
It's a very classic detection theory here.
Come up with a model for the background.
Create a linear fit for that background.
Then use that model to then quantify
the data to see which things are unusual.
Again, just using very classic detection
theory, the techniques.
Again, having a background model being
the linchpin of doing that.
And I'm not saying that this-- this is very recent work.
This parallel model-- something we did in the last year or so.
We just published it this summer.
I can't guarantee that in three or four years from now
that people will still be using this model,
because it is very new.
But I think this is representative
of the kinds of things that will be in three or four
or five years, that people-- it may not be this,
but something like this to characterize our data.
And so I think it's very useful there.
All right.
So with that, we will take a short break
and then we will show the example code
and talk about the assignment.
So very good.
Thank you very much.