D4M Session 3 Lecture

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation, or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

JEREMY KEPNER: All right, we're going to get started here, everyone. And I know we have a lot going on here at the lab today, and I'm very pleased that those of you decided to choose us today over all the other things that are going on. So that's great, I'm very pleased. I wanted to say I think I've provided feedback to everyone who submitted their earlier first homework over the weekend, and I was very impressed and pleased by those of you who did the homework. I think you really embraced the spirit of the homework very well.

I know the homework that just happened is one that kind of really builds on that. For those of you who didn't do the first one, then you're kind of left out a colon. But I would like to give my congratulations for those who took on that second homework, which was significantly-- definitely required a fair amount of contemplation to sort of attempt at homework. So I want to thank those of you who did that.

So we're going to get started here. And so we're going to bring up the slides here. So we're going to be getting a little bit more. We're kind of entering the second of the three sections. We had these first three courses here that really was the first part.

Did a lot with motivation and a lot with theory, and now we're going to do stuff that's a little bit more-- you know, we'll be continuing that trend. We'll still always have plenty of motivation and a little bit of theory, but getting more and more towards things that feel like the things you might actually really do. And so we're kind of entering into that next phase of the course here. So let's get started here. So this is the lecture 03.

Pull up an example here from some real data, and talking about that. Here is the outline of the course. Actually just before we're doing that, sort of somewhat mandatory for those of you who are just joining us or are watching on the web and have jumped to add to this lecture, the title of the class is Signal Processing on Databases, is two words that we don't normally see together, or two phrases.

Signal processing, which is really alluding to Detection Theory, Formal Detection theory, and its linear algebraic foundations. And databases, which is really alluding to dealing with strings, and unstructured data, and graphs. And so we're, in this course, talking about a technology, D4M, which allows us to pull these two views together. And so again we're now on the lecture 03, and we will continue on this journey that we have started.

Here's the outline. A little bit of history, I'm going to talk about how the web became the thing it is today, which I think talks a lot about why we invented the D4M technology. So it's going to talk about the specific gap that it fills, and talk a little bit about the D4M technology, some of the results that we've had. And then, hopefully today, the demo is more substantive, and so we'll spend more time on that than we did last time, where last time was more serious than the demo was really fairly small.

So for those of you who don't remember back in the ancient primordial days of the web, back in the early 90s, this is what it looked like. So if I talk about the hardware side of the web, it was quite simple. So there we go. The hardware side of the web was all Sun boxes.

We all had our Sun workstations, and there were clients, and there were servers, and there were databases. And the databases that we had at the time were SQL databases. Oracle was sort of really kind of just beginning to get going then. Sybase was sort of a dominant player at that stage.

This is sort of an example of the very first modern web page, or website, that was built. It was built by-- I'm sure there are other people who are doing very similar things at same time, so it's among the first. It's difficult to-- not going to be like Al Gore and say I invented the internet. But a friend of mine, Don Beaudry and I, were working for a company called Vision Intelligence Corporation, which is now part of GE.

And we had a data set and a SQL database, and we wanted to make it available. And there are a lot of thick client tools that came with our Sybase package for doing that, but we found them a little bit clunky. And so a friend of mine, Don, he downloaded this-- he said, hey I found this new software. I found a beta release of this software called NCSA Mosaic.

And so for those who don't know, NCSA Mosaic was the first browser, and it was invented at the National Center for Supercomputing Applications. And they released it. And it was awesome. It changed everything.

Just so you know, prior to that, and not to disrespect anybody who invented anything, but HTTP and HTML were dying. They had actually come out in 1989, and everyone like, this is silly. What's this? Why do I have to type this HTTP and then this HTML, and it's just very clunky and stuff like.

That and it was getting its clock pretty well cleaned by another technology called Gopher, which was developed in the University of Minnesota, which was a much easier interface. The server was a lot better. It was a lot easier to do links then in HTML. And it was just a menu driven, text based way to do that.

Well Mosaic created a GUI front end to HTML and allowed you to have pictures, and that changed everything. That just absolutely-- and it just sort of caught on like wildfire. But it was just a beta at that point in time, and I don't think you could really do pictures, but it rendered fonts nicely.

And Gopher did not. Gopher was like just plain text. So you could have like underline, in bold, and the links were very clear. You could click at them. You could even have indents and bullets. You could imagine organizing with something that would look like a web page today.

And so we had that, and they're like, oh well we'll talk to our database using that. We'll use you HTT put to post a request. The Mosaic server was so bad, we actually just used the Gopher server to talk to the Mosaic browser.

And so we would do puts, and then we used a language called perl, which many of you know, because it actually had a direct binding to Sybase. And so we would take that, format it, and create SQL queries. And then it would send us back strings, which we would then format as HTML, and input to the browser, and it was great. It was like very nice, and it was sort of the first modern web page that we did in 1992. And essentially a good fraction of the internet kind of looks like this today.

Our conclusion at that time was that this browser was a really lousy GUI, and that HTTP was really trying to be a file system, but it was a really bad file system. And Perl was really bad for analysis. And SQL really wasn't-- it's good for credit cards transactions, but it wasn't really good for data in and out.

So our conclusion was this is an awful lot of work just to use some documents, and it won't catch on. Our conclusion was it won't catch on. So with that conclusion, maybe I'm underscoring my judgment for the entire course that we decided this was a bad idea.

So after that we had sort of what we call the Cambrian Explosion of the internet, which is we had an explosion. On the hardware side, we had big, giant data centers merging servers and databases, which is great. We had all these new types of hardware technologies here, laptops, and tablets, and iPhones, and all that kind of stuff.

So the hardware technology went forward in leaps and bounds. And then the browsers really took off, all of them. We have all these different browsers. You can still see, I have the mosaic logo back there still a little bit, because they're still showing.

For those of you who don't know, the author of Mosaic and NCSA was a guy by the name of Marc Andreessen. He left NCSA and formed a company called Netscape, which actually, at a moment in time was the dominant-- they were the internet. They got crushed, although I think Marc Andreessen made a fair amount of money.

But they did open source all their software, and so they basically rewrote Mosaic from the ground up, and that became Mozilla and Firefox and all that. So that's still around today. Safari may have borrowed from this. I don't really know.

When Netscape was dominating the internet, Microsoft needed to-- first they thought the internet was going to be a fad. And then they're like, all right this is serious. We need a browser. And so Mosaic had actually been spun out of the University of Illinois into a company called Spyglass, which may actually still exist today. And then Microsoft bought the browser, which was the original NCSA Mosaic code, and that became in Internet Explorer. So it's still around today.

This is Chrome, and as far as I know chrome is a complete rewrite from the ground up. And probably some of the other, like Opera, and other browsers that around today are completely-- but they still use it. It's a browser to use.

HTML still uses the basic put get syntax. SQL, we're still using it today. We have all whole explosion of technologies, Oracle being very dominant, but my SQL, SQL Server, or other types of things, Sybase still around, very common.

And we still have two major servers out there, Apache and Windows. So Apache was the Mosaic server. And just so you know, Apache was-- it's a patchy server, meaning it was very buggy. So it was the a patchy server, and that became Apache.

So that was the new Mosaic server, became Apache because it was a patchy server. And I bet they had borrowed code from Gopher, so that's why the Gopher, I think, is still showing there a little bit. We've had a proliferation of languages that are very similar. So people, in terms of gluing these two together, Java, et cetera, all that type of stuff.

But the important thing to recognize is that this whole architecture was not put together, because somehow we felt the web browser was the optimal tool, or SQL was the optimal tool, or these languages are the optimal tool. They were just things that were laying down to could be repurposed for this job. And so that's why they're not necessarily optimal things. But nevertheless, our conclusion has to be slightly revised that it would not catch on, because obviously it did catch on. The desire to share information was so great that it overcame the high barrier to entry just to set up on a web page or something like that.

Now things have changed significantly. We're not the first people to observe that this original architecture is somewhat broken for this type of model. And so today the web looks a lot more like this. You see all of our awesome client hardware which is great.

We have servers and databases that have now moved in type of the data center into shipping containers and other types of things, which is great. Increasingly, our interfaces will look a lot more like video games. I would tell you that a typical app on the iPad feels a lot more like a video game than a browser, which is great, so we're really kind of moving beyond the browser is the dominant way for getting our information.

And companies like Google and others recognize that, if you're just presenting data to people, and you're not doing say credit card transactions, then a lot of the features of SQL in those types of databases are maybe more than you need, and you can actually make these sort of very scalable databases more simply that are called triple stores. And so Google Bigtable being the original sort of one. Riak is another. Cassandra is another. HBase is another. Accumulo, which is the one we use in this class, is another.

So this whole new technology space has really driven that and has made a lot more scalable. But unfortunately, in the middle here, we're still very much left with much of the same technology. And this is kind of the last piece, this sort of glue. And so our conclusion is now that it's a lot of work to view a lot of data, but it's a pretty great view, and we can really view a lot of data.

But we still have this middle problem. There's this gap. And so we're trying to address that gap. And so we address that gap with D4M. So D4M is formally designed from the ground up to be something that is really designed to work on triples, the kind of data we store in these modern databases, and do the kind of analysis in less code than you would otherwise.

So D4M, which stands for Dynamic Distributed Dimension Data model, which is sometimes in quotes, databases for Matlab, because it happens to be written in Matlab. It connects the triple stores, which can view their data as very large, sparse matrices of triples with the mathematical concepts of associative arrays that we talked about last class. That's essentially what D4M does for you.

So that's kind of one sort of holistic view of the technology space that we're dealing with here. Another way to view at it, which I'm going to talk a little bit about how the technologies that we have brought this to you. So I'm going to tell you a little bit about LLGrid and what we do there, because you're using your LLGrid accounts for this, and so I want to tell you a little bit about that.

So LLGrid is the core supercomputer at Lincoln Laboratory. And we're here to help with all your computing needs. Likewise, we're here to help with your D4M needs. If you have a project, and you think it would be useful to use D4M in that project, please contact us.

You don't have to just take this class and think that you're on your own. I mean, we are a service. We will help you do the schemas, get the code going, get it sort of adapted to your problem. That's what we do.

LLGrid, we have actually well over this, but just so I don't have to keep changing the slide. We always say it's about 500 users, about 2000 processors. It's more than that. To our knowledge, it's really the world's only desktop interactive supercomputer.

It's dramatically easier to use than any other supercomputer. And as a result, as an organization, we have the highest fraction of staff using supercomputing on a regular basis of any organization, just because it's so easy. And it's also kind of been the foundation, the vision, for supercomputing in the entire state of Massachusetts. All of our ideas have been adopted by other universities in the state and moving forward.

Not really the technology that we're talking about today, but something that is really kind of the bread and butter of LLGrid, is our Parallel Matlab technology, which is now going to be celebrating its 10th anniversary this year. And it's really a way to allow people to do parallel programming relatively easily. D4M interacts seamlessly with this technology, and so you will also be doing parallel database codes with this technology.

So we have this thing called Parallel Matlab. If you're an LLGrid user, you are entitled to, or you almost always get the book on Parallel Matlab. And it allows you to take things like matrices and chop them up very easily to do complicated calculations. And almost every single parallel program that you want to write can be written with just the addition of a handful of additional functions.

And we get to use what is called a distributed arrays parallel program model, which is generally recognized as by far the best of all the parallel programming models, if you are comfortable with a multi-dimensional arrays. Everyone at MIT is, so this is easy.

However this is not generally true. Thinking in matrices is not a universal skill, or universal training, and so it is a bit of-- sometimes we do say this is software for the 1%. So you know, it is not for everyone. But if you do have this knowledge and background, it allows you to do things very naturally. And as you know, we've talked about all of our associated arrays as looking at data in matrices, and so these two technologies come together very nicely.

And we are really the only place in the world where you can routinely use this model. This is the dominant model that people use when they do parallel programming, which is generally-- and you might say, well why do I say it's the best? Well we run contests. There's a contest every year where we bake off the best parallel programming models, and distributed arrays in high level environments wins every year. And so it's kind of the best model if you can understand them.

It wouldn't be a class on computing in 2012 unless we talked about the cloud, so we definitely need to talk about the cloud. And to make it simple for people, we're going to divide the cloud into sort of two halves. You can subdivide the cloud into many, many different components. There's what we call utility cloud computing, which is what most people use the cloud for.

So gmail, what are enterprise services, calendars online, all that kind of stuff, basic data sharing, photo sharing, and stuff like that, that's what we call utility cloud computing. You can do human resources on the cloud. You can do accounting on the cloud. All this type of stuff that's on the cloud, and it's a very, very successful business.

What we mostly focus on is what is called data intensive cloud computing. It's based on Hadoop and other types of technologies. But even still, a lot of that is technologies that are used by large scale cloud companies, but they don't really share with you.

Google, although it's often called Hadoop, Google doesn't actually use Hadoop. Hadoop is based on a small part of the Google infrastructure. Google's infrastructure is vastly larger than that. And they don't just let anybody log into and say, oh go ahead mine all the Google data.

They expose it to services, allowing you to do some of that, but the core technology that they do, they don't expose. In all large companies, they don't expose that to you. They give you selected services and such. But this is really what we're focused on is, this data intensive computing.

If I want to further subdivided the cloud from an implementer's perspective, if you own a lot of computing hardware, I mean hundreds of kilowatts or megawatts of computing hardware, you're most likely doing one of these four things. You could be doing traditional supercomputing, which we all know and love, which is sort of closest to what we do in LLGrid. You could do a traditional database management system, so every single time you use a credit card, make a purchase, you're probably connecting to a traditional database managed system.

You could be doing enterprise computing, which mostly, this day and age, is run out of VMware. So this is the LLGrid logo. And so you could be doing that. So that's all the things I earlier mentioned. Or this new kid on the block, big data, which is all the buzz today, using java and MapReduce and other types of things.

The important thing to recognize is these four areas, each of these are multi tens of billions of dollars industry. The IT world has gotten large enough that we now see specialization down to the hardware level. The hardware, the processors, that these folks use is different. Likewise for these folks and these folks. You see specialization of chips, now, to these different types of things.

So each of these at the center of a multi-billion dollar ecosystem, and they each have pros and cons. And sometimes you can do your mission wholly in one, but sometimes you have to cross. And when you do cross, it can be quite a challenge, because these worlds are not necessarily compatible.

And I would say, just to help people to understand Hadoop, one must recognize that Java is the first class citizen in the Hadoop world. The entire infrastructure is written in Java. It's designed so that if you only know Java, you can actually administer and manage a cluster, which is why it's become so popular. There are so many Java programmers, and now they just know Java, and they downloaded Hadoop, and they have rudimentary access to some computer systems they can sort of cobble together a cluster.

I should say the same thing occurred in the C in the Fortran commune in the mid 90s. There was a technology called NPI, in fact this is the NPI logo today, which allowed you, if you knew C, Fortran had rudimentary access on a network of workstations, you could create a cluster with NPI. And that really sort of began the whole sort of parallel cluster computing revolution. This does the same thing in Java, and so for Java programmers it's been a huge success in that regard.

Just so you know where we sit-in with that, so we have a LLGrid here, which has really made traditional supercomputing feel interactive and on demand and elastic, so that's one of the things that we do that's LLGrid, very unique. You launch your job. It runs immediately. That is different than the way every other supercomputing center in the country is run. Most supercomputing centers are run by, you write your program, and you launch it, and you wait until the queue says that it is run.

And in fact, some centers will even use, as a metric of success, how long their wait is. It's kind of like a restaurant, right? How many years do you have to wait to get a good reservation, or a good doctor right? And I remember someone saying, oh our system is so successful that our wait is a week.

You hit submit and you will have to wait a week to run your job. That's not really consistent with the way we do business around here. You're under very tight deadlines. So we created a very different type of system 10 years ago, which sort of feels more like what you people are trying to do in the cloud.

In the Hadoop community, there's been a lot of work to try and make databases. What these technologies give you is sort of efficient search, and so there are a number of databases that have been developed that sit on top of Hadoop, HBase being one, and Accumulo being another. Which to my knowledge is the highest performance open source triple store in the world, and probably will remain that way for a very long time. And so we will be using the Accumulo database.

And then we've created bindings to that. So we have our D4M technology, which allows you to bind you to databases. And we also have something called LLGrid MapReduce. MapReduce is sort of the core programming model within the Hadoop community. It's such a trivial model, it almost doesn't even deserve a name.

It's basically like, write a program given a list of files. Each program runs on each file independently. I mean it's a very, very simple model. It's been around since the beginning of computing, but the name has become popularized here, and we now have a way to do MapReduce right in LLGrid. It's very easy, and people enjoy it.

You can, of course, do the same thing with other technology in LLGrid, but for those people, particularly if you're writing job or-- not Matlab. We don't recommend people use this for Matlab, because we have better technologies for doing that. But if you're using Python, or Java, or other programs, then MapReduce is there for you.

As we move towards combining-- so our big vision here is to combine big compute and big data, and so we have a whole stack here as we deal with new applications, texts, cyber, bio, other types of things, the new APIs that people write, the new types of distributed databases. So it's really affecting everything. Just an example of the Hadoop architecture.

You've had this course. You should at least have a few minutes so that if someone asked, well what is Hadoop, you know what it does. This is the basic architecture. So you have a Hadoop MapReduce job. You submit it, and it goes to a JobTracker.

The JobTracker then breaks that up into subtasks, each of which has its own tracker. Those tasks then get sent to the DataNodes in the architecture, and it gets those DataNodes, and NameNode keeps track of the actual names of the things. Just a very simple a Hadoop cluster. This is the different types of the nodes that are in, and processes in architecture.

Hadoop's strengths and weaknesses, well it does allow you to distribute processing a large data. Best use case is, let's say you had enormous number of log files, and you decided you only wanted to search them once, for one string. That's the perfect use case for Hadoop. If you decide, though, that you would like to actually do more than one search, then it doesn't necessarily always make sense. So it's basically designed to run grep on an enormous number of files.

And I kid you not, there are companies today that actually do this, like at production scale. All their analytics are done by just grepping enormous numbers of log files. Needless to say that the entire database community cringes, and rolls over in their graves if they are not alive, at this. Because we have solved this problem by indexing our data, and that allows you to do fast search. And people, that's why people have invented databases like HBase and Accumulo, which would sit on top of Hadoop because they recognize that, really, if you're going to search more than once, you should index your data.

Again, it is very scalable. It is fundamentally designed to have extremely unreliable hardware in it. So it is quite resilient. It does this resiliency at a cost. It relies on a significant amount of replication. Typically the standard replication in a Hadoop cluster is a factor of 3.

If you're in the high performance storage business, this also makes you cringe, because you're paying 3x for your storage. We have techniques like RAID, which allow us to do that. But again, the expertise required to set up a cluster and do 3x replication is relatively low, and so that makes it very appealing to many, many, many folks, and so it's very useful for that.

Some of the difficulties, the scheduler in Hadoop is very immature. Schedulers are very well defined. There's about two or three standard schedulers in the supercomputing community. They're each about 20 years old. They all have the same 200 features, and you need about 200 features to really do a proper scheduler.

Hadoop is about four or five years old, and it's got about that many features. And so you do often have to deal with collisions. It's easy for one user to hog and monopolise the entire cluster. You're often dealing with various overheads from the scheduler itself. And this is well known to the Hadoop community. They're working on it actively, and we'll see all that-- it's not an easy multi-user environment.

And as I've said before, it fundamentally relies on the fact that the JVM is on every node. Because when you're sending a program to every node, the interpreter for that must exist on every node. And by definition, the only language that you're guaranteed to have on every node in a Hadoop cluster is Java. Any other language, any other tool, has to be installed and become a part of the image for the entire system so that it can be run on every node. So that is something that one has to be aware of.

And we've certainly seen it's like, oh I wrote these great programs in another language. And it's like, well can't I just run them? It's like no, that doesn't exist. Even distributing a fat binary can be difficult, because it has to be distributed through the Hadoop Distributed File system. All data is distributed there, and that is not-- you have to essentially write a wrapper program that then recognizes where to get your binary, and then exacts that thing and it can be tricky.

But the basic LLGrid MapReduce architecture simplifies this significantly, although this maybe doesn't look like it. Essentially you call LLGrid MapReduce. It launches a bunch of what are called mapper tasks. It runs them on your different input files. When they have done, they've created output files. And then runs another program. If you specified, it then combines the output.

So that's the basic model of MapReduce, which is you have what's called a map program and a reduced program, and the map program is given a list of files. It runs those programs on at a time. Each one of them generates an outpoint, and a reduce program pulls them all together.

All right, so that's a little tutorial on Hadoop, because I felt an obligation to explain that to you. We couldn't be a big data course without having doing that. But it's very simple, and very popular, and I expect it to maintain it's popular for a very long time.

I think it will evolve and get better, but for the vast majority of people on planet Earth, this is really the most accessible form of parallel computing technology that they have available. So we should all be aware of Hadoop, and its existence, and what it can do. Because many of our customers use Hadoop, and we need to work with them.

All right, so getting back now to D4M. I think I've mentioned a lot of this before. The core concept of D4M is multi-dimensional associative array. Again, D4M is designed to sort of overcomes. For those of us who have more mathematical expertise, we can do much more sophisticated things than you might be able to do in Hadoop.

Again, it allows you to look at your data in four ways at once. You can view it as 2D matrices, and reference rows and columns with strings, and have values that are strings. It's one to one with a triple store, so you can easily connect to databases. Again, it looks like matrices so you can do linear algebra. And also between the duality between adjacency matrices and graphs, you can also think about your data as graphs.

This is composable, as I've said before. Almost all of our operations that I perform an associated rate of return another associative array, and so we can do things like add them, subtract them, and them, or them, multiply them, and we can do very complicated queries very easily. And these can work on associative arrays. And if we're bound to tables, they can also do that.

Speaking of tables, I've already talked about our schema in this class that we always use. So if your standard data might look like this, say this is a cyber record of a source IP, domain, and destination IP, this is sort of the standard tabular view. We explode it by taking the source, pending the value to it which creates this very large, sparse table here which will naturally go into our triple store.

Of course, by itself it doesn't really gain as anything, because most tables either are row based or column based. The databases that we are working with a row based, which allow you to do fast look up of row key. However, once we've exploded the schema, if we also store the transpose, we now can index everything here quickly and effectively to the user. It looks like we have indexed every single string in the database with one schema, which is very nice, very powerful stepping stone for getting results.

All right, I'm going to talk a little bit about what we have done here. So I'm just going to go over some basic analytics and show you a little bit more code. So for example, here is our table. So this could be viewed as the sparse matrix. We have various source IPs here. And I want to just compute some very elementary statistics on this data.

I should say, computing things like sums and averages may sound like not very sophisticated statistic. It's still extraordinarily powerful, and we are amazed at how valuable it is, because it usually shows you right away the bad data in your data. And you will always have bad data. And this is probably the first bit of value add we provide to almost any customer that we work with on D4M, is it's the first time anybody's actually sort of looked at their data in totality, and been able to do these types of things.

And so usually the first thing we do is like, all right we've loaded their data in schema, and we've done some basic sum. And then we'll say, do you know that you have the following stuck on switches? You know, 8% of your data all has this value in this column, which absolutely can't be right, you know? And 8% could be a low enough number that you might not just encounter it through routine traversal, but it will stand out in just the most rudimentary histogram, extremely.

And so that's an incredibly valuable piece of [INAUDIBLE]. Oh my goodness, they'll usually be like, there's something broken in our system. Or you'll be like, you have this column, but it's almost never filled in, and it sure looks like it should be filled in, so these switches are stuck off.

I encourage, when you do the first thing, to just kind of do the basic statistics on the data and see where things are working and where things are broken. And usually you can then either fix them. And most of the time when we tell a customer, it's like, we're just letting you know these are issues in your data. We can work around them. We can sort of ignore them and still proceed, or you can fix them yourselves.

And almost always they're like, no we want to fix that ourselves. That's something that's fundamentally broken in our system. We want to make sure that that data is correct. Because usually that data is flowing into all other places doing a being acted on, and so that's one thing we're going to do.

So we're going to do some basic statistics here. We're going to compute how many times each column type appears. We're going to compute how many times each type appears. We'll compute some covariance matrices, just some very, very simple types of things here.

All right, let's move on here. So this is the basic implementation. So we're going to give it a set of rows that we're looking at here. This could be very large. We have a table binding, T, so we say, please go and get me all those rows.

So we get the whole swath of rows, and that will return that as an associative array. And then, normally I shorten this is double logi, since the table always contain strings, it return strings values, the first thing we knew it was going to get rid of those, because we're going to do math. So we turn them into logicals, and then we turn them back into double so we can do math.

You can actually pass regular expressions. You can also do that starts with command. We want to get the source IP. We're just interested in source IP domain. And the first thing we do is find some popular columns. So we just type sum, and that shows up popular columns. If we want to find popular pairs, it's covariance matrix calculation there, or square in and find domains with many destination IPs.

An example in this data set is this was the most popular in the data set that we had. These were the most popular things that appeared here. And you can see, this is all fairly reasonable stuff, LLBean, lot of New England Patriots fans in this data set, a lot of ads, Staples. These are the types of things it shows you right away.

Here's the covariance matrix of this data. So as you can see, it's symmetric. Obviously there is a diagonal, and it has the bipartite structure. That is, there is no destination to destination IP links. And we have source IP the destination, domain, and all that type of stuff exactly.

The covariance matrix is often extremely helpful thing to look at, because it will be the first time it shows you the full interrelated structure of your data. And you can quickly identify dense rows, dense columns, blocks, chunks, some things where you want to look at, chunks that aren't going to be interesting because they're very, very dense or very, very sparse. Again a very, very just sort of survey type of tool.

A little shout out here to our colleagues in the other group, who developed structured knowledge space. Structured knowledge space is a very powerful analytic data set. I kind of tell people it's like the Google search where when you type a search key, it starts guessing what your next answer is.

But Google does that based on sort of a long sort of analysis of all the different types of searches people have done, and it can make a good guess. SKS does that much more deeply, in that it maintains a database on a collection of documents. And in this case, if you type Afghanistan, will then actually go and count the occurrences of different types of entities in those documents, and then show you what the possible next key word choices can be, so in the actual data itself, not based on a set of cached queries and other types of things.

I don't really know what Google actually does. I'm just guessing. I'm sure they do use some data for guiding their search, but this is a fairly sophisticated analytic. And one of the big sort of wins for we knew we were on the right track with D4M is that this has been implemented in hundreds or thousands of lines of job at SQL. And I'm going to show you how we implement it in one line in D4M.

So let's go over that algorithm. So here's an example of what my data might look like. I have a bunch of documents here. I have a bunch of entities, where an entity appears in a document. I have a dot. I'm going to have my associative array over here, which is matching from the strings to the reals.

And my two facets are just going to be two columns names here, so I'm going to pick a facet y1 and y2. So I'm going to pick these two columns. All right, so basically that's what this does. This says get me y1 and get me y2.

I'm using this bar here to say I'm going to knock away the facet name afterwards so I can and them together. So that gets me all the documents that basically contain both UN and Carl, and then I can go and compute the counts that you saw in the previous page there just by doing this matrix multiply. So and them, I transpose them, and I matrix multiply to get them together, and then there we go.

All right, so now I'm going to get into the demo, and I think we have plenty of time for that. I'll sort of set it up, and they'll take short break. People can have cookies, and then we will get into the demo, which really begins to show you a way to use D4M on real data.

So the data that we're going to work with here is the Reuters Corpus. So this was a corpus of data released in 2000 to help spur research in this community. We're very grateful for Reuters to do this. They gave the data to NIST who actually stewards the data sets. It's a set of Reuters news reports over this particular time frame, the mid 90s, and it's like 800,000 total.

And what we have done, we're not actually working with the straight Reuters data, we've run it through various parsers that have extracted just the entities, so the people, the places, and the organization. So it's sort of a summary. It's a very terse summarization of the data, if that. It's really sort of a derived product.

The data is power law, so if we look at the documents per entity, you see here that certain people, places, and organizations, appear. A few of them appear lots of places, and most of them appear in just a few documents, and you see the same thing when you look at the entities per documents. So there's essentially, could view this is computing the in degrees and the out degrees of this bipartite graphs. And it has the classic power law shape that we expect to see in our data.

So with that, we're going to go in the demo. And just to summarize, so just recall, the evolution of the web is really created a new class of technologies. We really see the web moving towards game style interfaces, triple store databases, and technologies like D4M for doing analysis, very new technology.

And just so that we, for the record, there is no assignment this week. So for those of you who are still doing the assignments, yay. The example, actually won't do both of these today, we'll just do one of them. Oh and I keep making this mistake. I'll fix it. I'll fix it later.

But it's in the examples directory. We've now moved on to the second subfolder, apps, entity analysis. And you guys are encouraged to run those examples, which we will now do shortly. So we will take a short five minute break, and then continue on with the demo.

 

Free Downloads

Video

• iTunes U (MP4 - 83MB)
• Internet Archive (MP4 - 83MB)

Caption

• English-US (SRT)