The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high-quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JOSH TENENBAUM: We're going to--
I'm just going to give a bunch of examples of things
that we in our field have done.
Most of them are things that I've played some role in.
Maybe it was a thesis project of a student.
But they're meant to be representative of a broader
set of things that many people have been working
on developing this toolkit.
And we're going to start from the beginning in a sense-- just
some very simple things that we did
to try to look at ways in which probabilistic, generative
models can inform people's basic cognitive processes.
And then build up to more interestingly kinds
of symbolically structured models, hierarchical models,
and ultimately to these probabilistic programs
for common sense.
So when I say a lot of people have been doing this,
I mean here's just a small number of these people.
Every year or two, I try to update this slide.
But it's very much historically dated with people
that I knew when I was in grad school basically.
There's a lot of really great work
by younger people who maybe their names haven't
appeared on this slide.
So those dot dot dots are extremely serious.
And a lot of the best stuff is not included on here.
But in the last couple of decades,
across basically all the different areas
of cognitive science that cover basically
all the different things that cognition does,
there's been great progress building serious mathematical--
and what we could call reverse engineering models in the sense
that they are quantitative models of human cognition,
but they are phrased in the terms of engineering,
the same things you would use to build a robot
to do these things, at least in principle.
And it's been developing this toolkit
of probabalistic generative models.
I want to start off by telling you
a little bit about some work that I did together
with Tom Griffiths.
So Tom is now a senior faculty member at Berkeley,
one of the leaders in this field--
as well as a leading person in machine learning, actually.
One of the great things that he's done
is to take inspiration from human learning
and develop fundamentally new kinds of probabilistic models,
in non-parametric Bayes in particular,
inspired by human learning.
But when Tom was a grad student, we worked together.
He was my first student.
We're almost the same age.
So at this point, we're more like senior colleagues
than student advisor.
But I'll tell you about some work
we did back when he was a student,
and I was just starting off.
And we were both together trying to tackle this problem
and trying to see, OK, what are the prospects for understanding
even very basic cognitive intuitions, like senses
of similarity or the most basic kinds of causal discovery
intuitions like we were talking about before,
using some kind of idea of probabilistic inference
in a generative model?
And at the time-- remember in the introduction
I was talking about how there's been this back
and forth discourse over the decades of people saying, yeah,
rah rah, statistics, and, statistics,
those are trivial and uninteresting?
And at the time we started to do this, at least
in cognitive psychology, the idea
that cognition could be seen as some kind
of sophisticated statistical inference
was very much not a popular idea.
But we thought that it was fundamentally right
in some ways.
And it was at the time--
again, this was work we were doing in the early 2000s
when it was very clear in machine learning and AI
already how transformative these ideas were
in building intelligent machines or starting
to build intelligent machines.
So it seemed clear to us that at least it
was a good hypothesis worth exploring and taking much more
seriously than psychologists had much before that.
That this also could describe basic aspects
of human thinking.
So I'll give you a couple examples of what we did here.
Here's a simple kind of causal inference from coincidences,
much like what you saw going on in the video game.
There's no time in this.
It's really mostly just space, or maybe a little bit of time.
The motivation was not a video game, but imagine--
to put a real world context on it--
what's sometimes called cancer clusters or rare disease
clusters.
You can read about these often in the newspaper, where
somebody has seen some evidence suggestive of some maybe
hidden environmental cause--
maybe it's a toxic chemical leak or something--
that seems to be responsible for-- or maybe they
don't have a cause.
They just see a suspicious coincidence
of some very rare disease, a few cases that seem surprisingly
clustered in space and time.
So for example, let's say this is one square mile of a city.
And each dot represents one case of some very rare disease that
occurred in the span of a year.
And you look at this.
And you might think that, well, it
doesn't look like those dots are completely, uniformly, randomly
distributed over there.
Maybe there's some weird thing going on
in the upper left or northwest corner--
some who knows what--
making people sick.
So let me just ask you.
On a scale of 0 to 10, where 10 means you're sure
there's some kind of thing going on
and some special cause in some part of this map.
And 0 means no, you're quite sure there's nothing going on.
It's just random.
What do you say?
To what extent does this give evidence for some hidden cause?
So give me a number between 0 and 10.
AUDIENCE: 5.
JOSH TENENBAUM: OK, great.
5, 2, 7.
I heard a few examples of each of those.
Perfect.
That's exactly what people do.
You could do the same thing on Mechanical Turk,
and get 10 times as much data, and pay a lot more.
It would be the same.
I'll show you the data in a second.
But here's the model that we built.
So again, this model is a very simple kind
of generative model of a hidden cause
that various people in statistics
have worked with for a while.
We're basically modeling a hidden cause as a mixture.
Or I mean it's a generative model,
so we have to model the whole data.
When we say there's a hidden cause,
we don't necessarily mean that everything is caused by this.
It's just that the data we see in this picture
is a mixture of whatever the normal random thing going on is
plus possibly some spatially localized cause that
has some unknown position, unknown extent.
Maybe it's a very big region.
And some unknown intensity-- maybe
it's causing a lot of cases or not that many.
The hypothesis space maybe is best visualized like this.
Each of these squares is a different hypothesis
of a mixture density or a mixture model--
which is a mixture of just whatever the normal uniform
process is that causes a disease unrelated to space and then
some kind of just Gaussian bump, which can vary in location,
size, and intensity, that is the possible hidden cause
of some of these cases.
And what the model that we propose says
is that your sense of this spatial coincidence--
like when you look at a pattern of dots really and you see,
oh, that looks like there's a hidden cluster there somewhere.
It's basically you're trying to see whether something like one
of those things on the right is going on as opposed to the null
hypothesis of just pure randomness.
So we take this log likelihood ratio,
or log probability, where we're comparing
the probability of the data under the hypothesis
that there's some interesting hidden cause, one
of the things on the right, versus the alternative
hypothesis that it's just random,
which is just the simple, completely uniform density.
And what makes this a little bit interesting computationally
is that there's an infinite number of these possibilities
on the right.
There's an infinite number of different locations
and sizes and intensities of the Gaussian.
And you have to integrate over all of them.
So again, there's not going to be a whole lot
of mathematical details here.
But you can read about this stuff
if you want to read these papers that we had here.
But for those of you who are familiar with this,
with working with latent variable models,
effectively what you're doing is just
integrating either analytically or in a simulation over all
the possible models and sort of trying
to compute on average how much does the evidence support
something like what you see on the right, one of those cluster
possibilities, versus just uniform density.
And now what I'm showing you is that model
compared to people's judgments on an experiment.
So in this experiment, we showed people patterns
like the one you just saw.
The one you saw is this one here.
But in the different stimuli, we varied parameters
that we thought would be relevant.
So we varied how many points were there total,
how strong the cluster was in various ways, whether it was
very tightly clustered or very big,
the relative number of points in the cluster versus not.
So what you can see here, for example,
is it's a very similar kind of geometry,
except here this is a sort of biggish cluster.
And then we're making basically there's four points that look
clustered and two that aren't.
And in these cases, we just make the four points
more tightly clustered.
Here, what we're doing is we're going
from having no points that look clustered to having almost all
of the points looking clustered and just varying
the ratio of clustered points to non-clustered points.
Here, we're just changing the overall number.
So notice that this one is basically the same as this one.
So again, at both of these, we've
got four clustered points and two seemingly non-clustered
ones.
And here we just scale up in set-- or scale
up from four to two, to eight and four.
And here we scale it down to two and one,
and various other manipulations.
And what you can see is that they
have various systematic effects on people's judgments.
So what I'm calling the data there
is the average of about 150 people
who did the same judgment you did--
0 to 10.
What you can see is the one I gave you was this one here.
And the average judgment was almost exactly five.
And if you look at the variance, it
looks just like what you saw here.
Some people say two or three.
Some people say seven.
I chose one that was right in the middle.
The interesting thing is that, while you maybe
felt like you were guessing-- and if you just
listened to what everyone else was saying,
maybe it sounds like we're just shouting out random numbers--
that's not what you're doing.
On that one, it looks like it, because it's
right on the threshold.
But if you look over all these different patterns, what
you see is that sometimes people give much higher numbers
than others.
Sometimes people give much lower number than others.
And the details, that variation, both
within these different manipulations we did
and across them, are almost perfectly
captured by this very simple probabilistic generative
model for a latent cause.
So the model here is-- this is the predictions that model I
showed you is making, where again,
basically, a high bar means there's
strong evidence in favor of the hidden latent cause hypothesis.
Some, one, or more--
some cluster-- that low bar means strong evidence
for the alternative hypothesis.
The scale is a bit arbitrary.
And it's a log probability ratio scale.
So I'm not going to comment on the scale.
But importantly, it's the same scale across all of these.
So a big difference is, it's the same big difference
in both cases.
And I don't think this is fairly good evidence
that this model is capturing your sense
of spatial coincidence and showing that it's not
just random or arbitrary, but it's actually
a very rational measure of how much evidence there is
in the data for a hidden cause.
Here's the same model now applied
to a different data set that we actually
collected a few years before, which
just varies the same kinds of parameters,
but has a lot more points.
And the same model works in those cases, too.
The differences are a little more subtle
with these more points.
So I'll give you one other example of this sort of thing.
Like the one I just showed you, we're
taking a fairly simple statistical model.
This one, as you'll see, isn't even really causal.
This one at least, that I showed you, is causal.
The advantage of this other one is
that it's both a kind of textbook statistics example,
it's one where people do something more interesting
than what's in the textbook.
Although you can extend the textbook analysis to make
it look like what people do.
And unlike in this case here, you
can actually measure the empirical statistic.
You can go out, and instead of just like positing,
here's a simple model of what a latent environmental cause
would be like, you can actually go and measure
all the relevant probability distributions
and compare people not just with a notional model,
but with what, in some stronger sense,
is the rational thing to do, if you
were doing some kind of intuitive Bayesian inference.
So these are, again stuff that Tom Griffiths
did with me, in an in and then after grad school.
We asked people to make the following kind
of everyday prediction.
So we said, suppose you read about a movie that's
made $60 million to date.
How much money will it make in total?
Or you see that something's been baking
in the oven for 34 minutes.
How long until it's ready?
You meet someone who's 78 years old.
How long will they live?
Your friend quotes to you from line 17 of his favorite poem.
How long is the poem?
Or you meet a US congressman who has served for 11 years.
How long will he serve in total?
So in each of these cases, you're
encountering some phenomenon or event
in the world with some unknown total duration.
We'll call that t, total.
And all we know is that t, total, is somewhere
between zero and infinity.
We might have a prior on it, as you'll see in a second.
But we don't know very much about this particular t,
total except you get one example, one piece of data,
some t, which we'll just assume is just randomly sampled
between zero and t, total.
So all we know is that whatever these things are,
it's something randomly chosen, less than the total extent
or duration of these events.
And now we can ask, what can you guess
about the total extent or duration from that one
observation?
Or in mathematical terms, there is some unknown interval
from zero up to some maximal value.
You can put a prior on what that interval is.
And you have to guess the interval from one sampled point
sampled randomly within it.
It's also very similar-- and another reason
we studied this-- to the problem of learning
a concept from one example.
When you're learning what horses are from one example,
or when you're learning what that piece of rock climbing
question is-- what's a cam--
from one example, or what's a tufa from one example.
You can think, there's some region
in the space of all possible objects or something,
or some set out there.
And you get one or a few sample points,
and you have to figure out the extent of the region.
It's basically the same kind of problem, mathematically.
But what's cool about this is we can
measure the priors for these different classes of events
and compare people with an optimal Bayesian inference.
And you see something kind of striking.
So here's, on the top--
I'm showing two different kinds of data here.
On the top are just empirical statistics of events
you can measure in the world; nothing behavioral, nothing
about cognition.
On the bottom, I'm showing some behavioral data
and comparing it with model predictions that
are based on the statistics that are measured on top.
So what we have in each column is
one of these classes of events, like movie grosses in dollars.
You can get this data from iMDB, the Internet Movie Database.
You can see that most movies make $100 million or less.
There's sort of a power law.
But a few movies make hundreds, or even many hundreds,
maybe a billion dollars even, these days.
Similarly with poems, they have a power law distribution
of length.
So most poems are pretty short.
They fit on a page or less.
But there are some epic poems, or
some multi-page-- many, many hundreds of lines.
And they fall off with a long tail.
Lifespans, movie runtimes are kind of unimodal, almost
Gaussian-- not exactly.
Those red curves, histograms' bars,
show the empirical statistics that we
measured from public data.
And the red curves show just the best fit
of a simple parametric model, like a Gaussian or a power law
distribution that I'm mentioning.
House of representatives-- how long people serve in the House
has this kind of gamma, or particular gamma
called an Erlang shape with a little bit
of an incumbent effect.
Cake baking times-- so remember we asked how long
is this cake going to bake for.
They don't have any simple parametric form when you go in
and look at cookbooks.
But you see, there's something systematic there.
There's a lot of things that are supposed
to bake for exactly an hour.
There are some which have a smaller, or a shorter,
but broad mode.
And then there's a few epic 90-minute cakes out there.
So that's all the empirical statistics.
Now what you're seeing on the bottom is people's--
well, on the y-axis, the vertical axis,
you have the average-- it's a median--
of a bunch of human predictions for the total extent
of any one of these things, like your guess
of the total length of a poem given that,
basically, there is a line 17 in it.
And on the x-axis, what you're seeing is that one data point,
the one value of t, which is, all you know
is that it's somewhere between zero and t, total.
So different groups of subjects were
given five different values.
So you see five black dots, which correspond to what
five different subgroups of subjects
said for each of these possible t values.
And then the black and red curves
are the model fit, which comes from taking
a certain kind of Bayesian optimal prediction, where
the prior is what's specified on the top-- that's
the prior on t, total.
The likelihood is a sort of uniform random density.
So it's just saying t is just a uniform random sample from zero
up to t, total.
You put those together to compute a posterior.
And then you-- the particular estimator we're using
is what's called the posterior median.
So we're looking at the median of the exterior
and comparing that with a median of human subjects.
And what you can see is that it's almost a perfect fit.
And it doesn't really matter whether you
take the red curve, which is what comes from approximating
the prior with one of these simple parametric models,
or the black one, which comes from just
taking the empirical histogram.
Although, for the cake baking times,
you really can only go for the empirical one.
Because there is no simple parametric one.
That's why you just see a jagged black line there.
But it's interesting that it's almost a perfect fit.
There are a couple--
just like somebody asked in Demis's talk--
there's one or two cases we found where this model doesn't
work, sometimes dramatically, and sometimes a little bit.
And they're all interesting.
But I have time to talk about it.
That's one of the things I decided to skip.
If you'd like to talk about it, I'm happy to do that.
But most of the time, in most of the cases we've studied,
these are representative.
And I think, again, all of the failure cases
are quite interesting ones.
That point to, this is one of the many things
we need to go beyond.
But the interesting thing isn't just
that the curves fit the data, but the fact
that the actual shape is different in each case.
Depending on the prior of this different classes of events,
you get a fundamentally different, or qualitatively
different, prediction function.
Sometimes it's linear.
Sometimes it's non-linear.
Sometimes it has some weird shape.
And really, quite surprisingly to us,
people seem to be sensitive to that.
So they seem to predict in ways that are reflective of not only
the optimal Bayesian thing to do,
but the optimal Bayesian thing to do from the optimal prior,
from the correct prior.
And I certainly don't want to suggest that people always
do this.
But it was very interesting to us
that for just a bunch of everyday events, and really,
the places where this analysis works best
are ones, again, where we think people actually might plausibly
have good reasons to have the relevant experiences
with these everyday events, they seem
to be sensitive to both the statistics in the sense of just
what's going on in the world and doing
the right statistical prediction.
So that's what we did.
10 years ago or so, that was like the state
of the art for us.
And then we wanted to know, well, OK,
can we take these sorts of ideas and scale them up
to some actually interesting cognitive problems,
like say, for example, learning words for object categories.
And we did some of that.
I'll show you a little bit of that
before showing you what I think was missing there.
I mean, in a lot of ways, this is a harder problem.
I mean, it's very similar, as I said.
It's basically like, there's just like the problem I just
showed you, where there was an unknown total extent
or duration, and you got one random sample from it,
here there is some un--
imagine the space of all possible objects--
could be a manifold or described by a bunch of knobs.
I mean, these are all generated from some computer program.
If these were real, biological things,
they would be generated from DNA or whatever it is.
But there's some huge, maybe interestingly structured, space
of all possible objects.
And within that space is some subset, some region or subset,
somehow described that is the set of tufas.
And somehow you're able to grasp that subset, more or less,
if you get its boundaries, to be able to say yes or no
as you did at the beginning of the lecture
from just, in this case, a few points-- three points--
randomly sampled from somewhere in that region.
It would work just as well if I showed you one of them,
basically.
So in some sense, it's the same problem.
But it's much harder, because here, the space
was this one dimensional thing.
It was just a number.
Whereas here, we don't know what's
the dimensionality of the space of objects.
We don't know how to describe the regions.
Here we knew how to describe the regions.
They were just intervals with a lower bound at zero
and an upper bound at some unknown thing.
And the hypothesis space of possible regions
was just all the possible upper bounds of this event duration.
Here we don't know how to describe this space.
We don't know how to describe the regions that
correspond to object concepts.
We don't know how to put a price on those hypotheses.
But in some work that we did-- in particular,
some work that I did with Fei Xu, who is also
a professor at Berkeley.
We were colleagues and friends in graduate school.
We sort of did what we could at the time.
So we made some guesses about what that hypothesis space--
what that space might be like, what the hypothesis space might
be like, how to put some priors, and so, on there.
Used exactly the same likelihood,
which was just this very simple idea that the observed examples
are a uniform random draw from some subset of the world.
And you have to figure out what that subset is.
And we were able to make some progress.
So what we did was we said, well, like in biology,
perhaps-- and if you saw-- how many people saw Surya Ganguli's
lecture yesterday morning?
Cool.
I sort of tailored this for assuming that you probably
had seen that.
Because there's a lot of similarities, or parallels,
which is neat.
And it's, again, part of engaging on generative models
and neural networks.
As you saw him do, you'll get my version of this.
So also, like he mentioned, there
are actual processes in the world which generate objects--
something like this.
We know about evolution--
produces basically tree-structured groups,
which we call species, or genus, or something like that, or just
taxa, or something.
There's groups of organisms that have
a common evolutionary descent.
That's the way a biologist might describe it.
And we know, these days, a lot about the mechanisms
that produce that.
Even going back 100 or 200 years,
say, to Darwin, we knew something
about the mechanisms that produced it,
even if we didn't know the genetic details, ideas
of something like mutation, variation, natural selection
as a kind of mechanistic account,
about right up there with Newton and forces.
But anyway, scientists can describe some process
that generates trees.
And maybe people have some intuition,
just like people seem to have some intuitions
about these statistics of everyday events,
maybe they have some intuitions, somehow,
about the causal processes in the world, which
give rise to groups and groups and subgroups.
And they can use that to set up a hypothesis space.
And the way we went about this is,
we have no idea how to describe people's internal mental models
of these things, but you can do some simple--
there are simple ways to get this picture
by just basically asking people to judge similarity
and doing hierarchical clustering.
So this is a tree that we built up by just asking people--
getting some subjective similarity metric
and then doing hierarchical clustering, which we thought
could roughly approximate maybe the internal hierarchy
that our mental models impose on this.
Were you raising your hand or just-- no.
OK.
Cool.
We ultimately found this dissatisfying,
because we don't really know what the features are.
We don't really know if this is the right tree
or how people built it up.
But it actually worked pretty well,
in the sense that we could build up this tree.
We could then assume that the hypotheses for concepts
just corresponded to branches of the tree.
And then you could--
again, to put it just intuitively,
the way you do this learning from one or a few examples,
let's say that you see those few tufas over there.
You're basically asking, which branch of the tree
do I think-- those are randomly drawn from some internal branch
of the tree, some subtree.
Which subtree is it?
And intuitively, if you see those things and you say,
well, they are randomly drawn from some branch,
maybe it's the one that I've circled.
That sounds like a better bet, for example, than this one
here, or maybe this one, which would
include one of these things, but not the others.
So that's probably unlikely.
And it's probably a better bet than, say,
this branch, or this branch, or these ones, which are logically
compatible, but somehow it would have been sort
of a suspicious coincidence.
If the set of tufas had really been this branch here,
or this one here, then it would have been quite a coincidence
that the first three examples you saw
were all clustered over there in one corner.
And what we showed was that, that kind of model,
where that suspicious coincidence came out
from the same kinds of things I've just
been showing you for the causal clustering example,
and for the interval thing, it's the same Bayesian math.
But now with this tree-structured hypothesis
space, that was actually-- did a pretty good job
of capturing people's judgments.
We gave people one or a few examples of these concepts
that, the examples could be more narrowly or broadly spread,
just like you saw in the clustering thing,
but just sort of less extensive.
We did this with adults.
We did this with kids.
And I won't really go into any of the details.
but If you're interested, check out these various Xu
and Tenenbaum papers.
That's the main one there.
And you know, the model kind of worked.
But ultimately, we found it dissatisfying.
Because we couldn't really explain--
we didn't really know what the hypothesis space was.
We didn't really know how people were building up this tree.
And so we did a few things.
We-- meaning I with some other people--
turned to other problems where we had a better idea, maybe,
of the feature space and the hypothesis space,
but the same kind of ideas could be explored and developed.
And then ultimately-- and I'll show you this maybe before
lunch, or maybe after lunch--
we went back and tackled the problem of learning concepts
from examples with other cases where we could get a better
handle on really knowing what the representations that people
were using were, and also where we could compare with machines
in much more compelling apples and oranges ways.
In some sense here, there's no machine, as far as I know,
that can solve this problem as well as our model.
On the other hand, that's, again, it's
just very much like the issue that came up
when we were talking about--
I guess maybe it was with you, Tyler--
when we were talking about the deep learning-- or with you,
Leo--
the deep reinforcement network.
A machine that's looking at this just as pixels
is missing so much of what we bring
to it, which is, we see these things
as three-dimensional objects.
And just like the cam in rock climbing,
or any of those other examples I gave
before, I think that's essential to the abilities
that people are doing.
The generative model we build, this tree
is based not on pixels, or even on ConvNet features,
but on a sense of the three-dimensional objects,
its parts, and their relations to each other.
And so, fundamentally, until we know how to perceive objects
better, this is not going to be comparable between humans
and machines on equal terms.
But I'll show you a little bit later
some still pretty quite interesting, but simpler,
visual concepts that you can still learn and generalize
from one example, but where they are comparable in equal terms.
But first I want to tell you a little bit about these--
yet another cognitive judgment, which
like the word learning, or concept learning cases,
involved generalizing from a few examples.
They also involve using prior knowledge.
But they're ones where maybe we have
some way of capturing people's prior knowledge by using
the right combination of statistical inference
on some kind of symbolically structured bottle.
So you can already see, as--
I mean, just sort of to show the narrative here.
The examples I was giving here, this
doesn't require any symbolic structure.
All that stuff I was talking at the beginning,
about how we have to combine statistical inference,
sophisticated statistical inference,
with sophisticated symbolic representations,
you don't need any of that here.
All the representations could just
be counting up numbers or using simple probability
distributions that statisticians have
worked with for over 100 years.
Once we start to go here, now we have
to define a model with some interesting structure,
like a branching tree structure, and so on.
And as you'll see, we can quickly
get to lots more interesting causal,
compositionally-structured generative models
in similar kinds of tasks.
And in particular, we were looking for--
for a few years, we were very interested
in these property induction tasks.
So this was-- it happened to be--
I mean, I think this was a coincidence.
Or maybe we were both influenced by Susan Carey, actually.
So the work that Surya talked about, that he was trying
to explain as a theoretician-- remember,
Surya and Andrew Saxe, they were trying
to give the theory of these neural network
models that Jay McClelland and Tim
Rogers had built in the early 2000s,
around the same time we were doing this work.
And they were inspired by some of Susan Carey's work
on children's intuitive biology, as well as
other people out there in cognitive psychology--
for example, Lance Rips, and Smith, Madine.
Many, many cognitive psychologists
studied things like this--
Dan Osherson.
They often talked about this as a kind of inductive reasoning,
or property induction, where the idea was--
so it might look different from the task I've given you before,
but actually, it's deeply related.
The task was often presented to people like an argument
with premises and a conclusion, kind
of like a traditional deductive syllogism, like all men are
mortal, Socrates is a man, therefore Socrates is mortal.
But these are inductive in that there
is no-- you can't conclude with deductive certainty
the conclusion follows from the premises
or is falsified by the premise, but rather you just
make a good guess.
The statements above the line provide some, more or less,
good or bad evidence for the statement
below the line being true.
These studies were often done with-- they
could be done with just sort of familiar biological properties,
like having hairy legs or being bigger than a breadbox.
I mean, it's also-- it's very much the same kind of thing
that Tom Mitchell was talking about, as you'll start to see.
There's another reason why I wanted to cover this.
We worked on these things because we
wanted to be able to engage with the same kinds of things
that people like Jay McClelland and Tom Mitchell
were thinking about, coming from different perspectives.
Remember, Tom Mitchell showed you
his way of classifying brain representations of semantics
with matrices of objects and 20-question-like features that
included things like is it hairy, or is it alive,
or does it eggs, or is it bigger than a car,
or bigger than a breadbox, or whatever.
Any one of these things--
basically, we're getting at the same thing.
Here there's just what's--
often these experiments with humans
were done with so-called blank predicates, something
that sounded vaguely biological, but was basically made up,
or that most people wouldn't know much about.
Does anyone know anything about T9 hormones?
I hope so, because I made it up.
But some of them were just done with things
that were real, but not known to most people.
So if I tell you that gorillas and seals both have
T9 hormones, you might think it's sort of, fairly plausible
that horses have T9 hormones, maybe more so than if I
hadn't told you anything.
Maybe you think that argument is more
plausible than the one on the right; given
that gorillas and seals have T9 or hormones, that anteaters
have hormones.
So maybe you think horses are somehow
more similar to gorillas and seals than anteaters are.
I don't know.
Maybe.
Maybe a little bit.
If I made that bees--
gorillas and seals have T9 on hormones.
Does that make you think it's likely that bees have
T9 hormones, or pine trees?
The farther the conclusion category
gets from the premises, the less plausible it seems.
Maybe the one on the lower right also seems not very plausible,
or not as plausible.
Because if I tell you that gorillas
have T9 hormones, chimps, monkeys, and baboons
all have T9 on hormones, maybe you
think that it's only primates or something.
So they're not a very-- it's, again,
one of these typicality-suspicious
coincidence businesses.
So again, you can think of it as-- you
can do these experiments in various ways.
I won't really go through the details,
but it basically involves giving people
a bunch of different sets of examples, just like--
I mean, in some sense, the important thing to get
is that abstractly it has the same character of all
the other tasks you've seen.
You're giving people one or a few examples, which we're
going to treat as random draws from some concept,
or some region in some larger space.
In this case, the examples are the different premise
categories, like gorillas and seals
are examples of the concept of having T9 hormones.
Or gorillas, chimps, monkeys, and baboons
are an example of a concept.
We're going to put a prior on possible extents
of that concept, and then ask what kind of inferences
people make from that prior, to figure out
what other things are in that concept.
So are horses in that same concept?
Or are anteaters?
Or are horses in it more or less, depending on the examples
you give?
And what's the nature of that prior?
And what's good about this is that,
kind of like the everyday prediction task--
the lines of the poems, or the movie grosses, or the cake
baking-- we can actually sort of go out and measure
some features that are plausibly relevant,
to set up a plausibly relevant prior,
unlike the interesting object cases.
But like the interesting object cases,
there are some interesting hierarchical and other kinds
of causal compositional structures
that people seem to be using that we
can capture in our models.
So here, again, the kinds of experiments-- these features
were generated many years ago by Osherson and colleagues.
But it's very similar to the 20 questions game
that Tom Mitchell used.
And I don't remember if Surya talked
about where these features came from,
that he talked a lot about a matrix of objects and features.
I don't know if he talked about where they come from.
But actually, psychologists spent a while coming up
with ways to get people to just tell you
a bunch of features of animals.
This is, again, it's meant to capture the knowledge
that maybe a kid would get from maybe plausibly reading
books and going to the zoo.
We know that elephants are gray.
They're hairless.
They have tough skin.
They're big.
They have a bulbous body shape.
They have long legs.
These are all mostly relative to other animals.
They have a tail.
They have tusks.
They might be smelly, compared to other animals--
smellier than average is sort of what that means.
They walk, as opposed to fly.
They're slow, as opposed to fast.
They're strong, as opposed to weak.
It's that kind of business.
So basically what that gives you is this big matrix.
Again, the same kind of thing that you
saw in Surya's talk, the same kind of thing
that Tom Mitchell is using to help
classify things, the same kind of thing
that basically everybody in machine learning
uses-- a matrix of data with objects, maybe as rows,
and features, or attributes, as columns.
And the problem here is--
the problem of learning is to say--
the problem of learning and generalizing from one example
is to take a new property, which is a new concept, which
is like a new column here, to get
one or a few examples of that concept, which is basically
just filling in one or a few entries in that column,
and figure out how to fill in the others, to decide,
do you or don't you have that property, somehow
building knowledge that you can generalize
from your prior experience, which
could be captured by, say, all the other features
that you know about objects.
So that's the way that you might set up
this problem, which again, looks like a lot of other problems
of, say, semi-supervised learning or sparse matrix
completion.
It's a problem in which we can, or at least
we thought we could, compare humans
and many different algorithms, and even theory,
like from Surya's talk.
And that seemed very appealing to us.
What we thought, though, that people were doing,
which is maybe a little different than what--
or somewhat different-- well, quite different than what Jay
McClelland thought people were doing--
maybe a little bit more like what Susan Carey or some
of the earlier psychologists thought people were doing--
was something like this.
That the way we solve this problem,
the way we bridged from our prior experience to new things
we wanted to learn was not, say, by just computing
the second order of statistics and correlations,
and compressing that through some bottleneck hidden layer,
but by building a more interesting
structured probabilistic model that was, in some form,
causal--
in some form-- in some form, compositional
and hierarchical-- something kind of like this.
And this is a good example of a hierarchical generative model.
There's three layers of structure here.
The bottom layer is the observable layer.
So the arrows in these generative models point down,
often, usually, where the thing on the bottom is the thing
you observe, the data of your experience.
And then the stuff above it are various levels of structure
that your mind is positing to explain it.
So here we have two levels of structure.
The level above this is sort of this tree in your head.
The idea-- it's like a certain kind of graph structure,
where the objects, or the species, are the leaf nodes.
And there's some internal nodes corresponding,
maybe to higher level taxa, or groups, or something.
You might have words for these, too, like mammal,
or primate, or animal.
And the idea is that there's some kind
of probabilistic model that you can describe,
maybe even a causal one on top of that symbolic structure,
that tree, that produces the data that's more directly
observable, the observable features,
including the things you've only sparsely
observed and want to fill in.
And then you might also have higher levels of structure.
Like if you want to explain, how did you
learn that tree in the first place,
maybe it's because you have some kind of generative model
for that generative model.
So here I'm just using words to describe it,
but I'll show you some other stuff in a--
or I'll show you something more formal a little bit later.
But you could say, well, maybe the way
I figure out that there's a tree structure is
by having a hypothesis--
the way I figure out that there's
that particular tree-structured graphical model of this domain
is by having the more general hypothesis
that there is some latent hierarchy of species.
And I just have to figure out which one it is.
So you could formulate this as a hierarchical inference
by saying that what we're calling
the form, the form of the model, it's
like a hypothesis space of models, which are themselves
hypothesis spaces of possible observed patterns of feature
correlation.
And that, that higher level knowledge,
puts some kind of a generative model on these graph
structures, where each graph structure then
puts a generative model on the data you can observe.
And then you could have even higher levels
of this sort of thing.
And then learning could go on at any or all levels
of this hierarchy, higher than the level of experience.
So just to show you a little bit about how this kind of thing
works, what we're calling the probability of the data
given the structure is actually exactly the same, really,
as the model that Surya and Andrew Saxe used.
The difference is that we were suggesting--
may be right, may be wrong--
that something like this generative model
was actually in your head.
Surya presented a very simple abstraction
of evolutionary branching process, a kind of diffusion
over the tree, where properties could turn on or off.
And we built basically that same kind of model.
And we said, maybe you have something in your head
as a model of, again, the distribution of properties,
or features, or attributes over the leaf nodes of the tree.
So if you have this kind of statistical model.
If you think that there's something like a tree
structure, and properties are produced over the leaf nodes
by some kind of switching, on-and-off, mutation-like
process, then you can do something like in this picture
here.
You can take an observe a set of features in that matrix
and learn the best tree.
You can figure out that thing I'm showing on the top,
that structure, which is, in some sense, the best
guess of a tree structure-- a latent tree structure-- which
if you then define some kind of diffusion mutation
process over that tree, would produce with high probability
distributions of features like those shown there.
If I gave you a very different tree
it would produce other patterns of correlation.
And it's just like Surya said, it
can be all captured by the second order statistics
of feature correlations.
The nice thing about this is that now this
also gives a distribution on new properties.
So if I observe--
because each column is conditionally independent
given that model.
Each column is an independent sample
from that generative model.
And the idea is if I observe a new property,
and I want to say, well, which other things have this, well,
I can make a guess on using that probabilistic model.
I can say, all right, given that I
know the value of this function over the tree,
this stochastic process, at some points, what
do I think the most likely values are at other points?
And basically, what you get is, again, like in the diffusion
process, a kind of similarity-based generalization
with a tree-structured metric, that nearby points in the tree
are likely to have the same value.
So in particular, things that are near to, say, species one
and nine are probably going to have the same property,
and others maybe less so.
And you build that model.
And it's really quite striking how much
it matches people's intuitions.
So now you're seeing the kinds of plots I was showing you
before, where--
all my data plots look like this.
Whenever I'm showing the scatterplot, by default,
the y-axis is the average of a bunch of people's judgments,
and the x-axis is the model predictions on the same units
or scale.
And each of these scatterplots is
from a different experiment-- not done
by us, done by other people, like Osherson and Smith
from a couple of decades ago.
But they all sort of have the same kind of form,
where each dot is a different set of examples,
or a different argument.
And what typically varied within an experiment--
you vary the examples.
And you fix constant the conclusion category.
And you see, basically, how much evidential support
to different sets of two or three examples
gives to a certain conclusion.
And it's really, again, quite striking that--
sometimes in a more categorical way,
sometimes in a more graded way-- but basically,
people's average judgments here just
line up quite well with the sort of Bayesian inference
on this tree-structured generative model.
These are just examples of the kinds of stimuli here.
Now, we can compare.
One of the reasons why we were interested
in this was to compare, again, many different approaches.
So here I'm going to show you a comparison with just
a variant of our approach.
It's the same kind of hierarchical Bayesian model,
but now the structure isn't a tree,
it's a low-dimensional Euclidean space.
You can define the same kinds of proximity smoothness thing.
I mean, again, it's more a standard in machine learning.
It's related to Gaussian processes.
It's much more like neural networks.
You could think of this as kind of like a Bayesian version
of a bottleneck hidden layer with two dimensions,
or a small number of dimensions.
The pictures that Surya showed you were all higher dimensions
than two dimensions in the latent space,
or the hidden variable space, of the neural network,
the hidden layer space.
But when he compress it down to two dimensions,
it looks pretty good.
So it's the same kind of idea.
Now what you're saying is you're going to find,
not the best tree that explains all these features,
but the best two-dimensional space.
Maybe it looks like this.
Where, again, the probabilistic model
says that things which are relatively-- things that
are closer in this two-dimensional space
are more likely to have the same feature value.
So you're basically explaining all the pairwise feature
correlations by distance in this space.
It's similar.
Importantly it's not as causal and compositional.
The tree models something about, possibly, the causal processes
of how organisms come to be.
If I told you that, oh, there's this--
that I told you about a subspecies, like whatever--
what's a good example--
different breeds of dogs.
Or I told you that, oh, well, there's not just wolves,
but there's the gray-tailed wolf and the red-tailed wolf.
Red-tailed wolf?
I don't know.
Again, they're probably similar, but they might--
one red-tailed wolf, whatever that is, more
similar to another red-tailed wolf,
probably has more features in common
than with a gray-tailed wolf, and probably more
to the gray-tailed wolf than to a dog.
The nice thing about a tree is I can tell you these things,
and you can, in your mind--
maybe you'll never forget that there's a red-tailed wolf.
There isn't.
I just made it up.
But if you ever find yourself thinking
about red-tailed wolves and whether their properties are
more or less similar to each other than to gray-tailed
wolves, or less so to dogs, or so on,
it's because I just said some things,
and you grew out your tree in your mind.
That's a lot harder to do in a low-dimensional space.
And it turns out that, that model also
fits this data less well.
So here I'm just showing two of those experiments.
Some of them are well fit by that model,
but others are less well fit.
Now, that's not to say that they wouldn't
be good for other things.
So we also did some experiments.
This was experiments that we did.
Oh, actually, I forgot to say, really importantly,
this was all worked done by Charles Kemp, who's
now a professor at CMU.
And it was part of the stuff that he did in his PhD thesis.
So we were interested in this as a way, not to study trees,
but to study a range of different kinds of structures.
And it is true, going back, I guess, to the question
you asked, this is what I was referring to about
low-dimensional manifolds.
There are some kinds of knowledge representations
we have which might have a low-dimensional spatial
structure, in particular, like mental maps of the world.
So our intuitive models of the Earth's surface, and things
which might be distributed over the Earth's surface
spatially, a two-dimensional map is probably
a good one for that.
So here we considered a similar kind of concept
learning from a few examples task, where we said--
but now we put it like this.
We said, suppose that a certain kind
of Native American artifact has been
found in sites near city x.
How likely is it also to be found in sites near city y?
Or we could say sites near city x and y, how about city z.
And we told people that different Native American
tribes maybe had-- some lived in a very small area, some
lived in a very big area.
Some lived in one place, some another place.
Some lived here, and then moved there.
We just told people very vague things that taps into people's
probably badly remembered, and very distorted,
versions of American history that would basically suggests
that there should be some kind of similar kind
of spatial diffusion process, but now
in your 2D mental map of cities.
So again, there's no claim that there's any reality to this,
or fine-grained reality.
But we thought it would sort of roughly correspond
to people's internal causal generative
models of archeology.
Again, I think it says something about the way
human intelligence works that none of us
are archaeologists, probably, but we still have these ideas.
And it turned out that, here, a spatially structured model
actually works a lot better.
Again, it shouldn't be surprising.
It's just showing that actually, the way-- the judgments
people make when they're making inferences from a few examples,
just like you saw with the predicting
the everyday events, but now in the much more
interestingly structured domain, is
sensitive to the different kinds of environmental statistics.
There it was different power laws versus Gaussian's of cake
bake-- or of movie grosses versus lifetimes or something.
Here it's other stuff.
It's more interestingly structured kinds of knowledge.
But you see the same kind of picture.
And we thought that was interesting,
and again, suggests some of the ways
that we are starting to put these tools together,
putting together probabilistic generative models
with some kind of interestingly structured knowledge.
Now, again, as you saw from Surya,
and as Jay McClellan and Tim Rogers worked on,
you can try to capture a lot of this stuff
with neural networks.
The neat thing about the neural networks that these guys have
worked on is that exactly the same neural network can
capture this kind of thing, and it
can capture this kind of thing.
So you can train the very same hidden multilayer neural
network with one matrix of object and features.
And the very same neural network can
predict the tree-structured patterns for animals
and their properties, as well as the spatially-structured
patterns for Native American artifacts and their cities.
The catch is that it doesn't do either of them that well.
It doesn't do as well as the tree-structured models do
for peop--
when I say either, it doesn't do that well, I
mean, in capturing people's judgments.
It doesn't do as well as the best tree-structured models do
for people's concepts of animals and their properties.
And it doesn't do as well as the best spacial structures.
But again, it's in the same spirit as the DeepMind networks
for playing lots of Atari games.
The idea there is to have the same network solve
all these different tasks.
And in some sense, I think that's a good idea.
I just think that the architecture should
have a more flexible structure.
So we would also say, in some sense,
the same architecture is solving all these different tasks.
It's just that this is one setting of it.
And this is another setting of it.
And where they differ is in the kind of structure
that-- well, they differ in the fact
that they explicitly represent structure in the world.
And they explicitly represent different kinds of structure.
And they explicitly represent that different kinds
of structure are appropriate to different kinds of domains
in the world and our intuitions about the causal processes that
are at work producing the data.
And I think that, again, that's sort
of the difference between the pattern classification
and the understanding or explaining
view of intelligence.
The explanations, of course, go a lot beyond different ways
that similarity can be structured.
So one of the kind of nice things-- oh,
and I guess another--
two other points beyond that.
One is that to get the neural networks
to do that, you have to train them with a lot of data.
Remember, Surya, as Tommy pushed him on in that talk,
Surya was very concerned with modeling
the dynamics of learning in the sense of the optimization time
course, how the weights change over time.
But he was usually looking at infinite data.
So he was assuming that you had, effectively,
an infinite number of columns of any of these matrices.
So you could perfectly compute the statistics.
And another important thing about the difference
being the neural network models and the ones I was showing you
is that, suppose you want to train the model, not
on an infinite matrix, but on a small finite one,
and maybe one with missing data.
It's a lot harder to get the-- the neural network will
do a much poorer job capturing the structure than these more
structured models.
And again, in a way that's familiar with--
have you guys talked about bias-variance dilemma?
So it's that same kind of idea that you probably
heard about from Lorenzo.
Was it Lorenzo or one of the machin learni--
OK.
So it's that same kind of idea, but now applying
in this interesting case of structured estimation
of generative models for the world,
that if you have relatively little data, and sparse data,
then having a more structured inductive bi--
having the inductive bias that comes from a more structured
representation is going to be much more valuable when
you have sparse and noisy data.
The key-- and again, this is something that Charles and I
were really interested in-- is we wanted to--
like the DeepMind people, like the connectionists,
we wanted to build general purpose semantic cognition,
wanted to build general purpose learning and reasoning systems.
And we wanted to somehow figure out
how you could have the best of both worlds, how you could have
a system that relatively quickly could come
to get the right kind of strong constraint-inductive bias
in some domain, and a different one for a different domain,
yet could learn in a flexible way
to capture the different structure in different domains.
More on that in a little bit.
But the other thing I wanted to talk about here
is just ways in which our mental models,
our causal and compositional ones,
go beyond just similarity.
I guess, since time is short-- well,
I was planning to go through this relatively quickly.
But anyway, mostly I'll just gesture towards this.
And if you're interested, you could
read the papers that Charles has, or his thesis.
But here, there's a long history of asking people
to make these kind of judgments, in which
the basis for the judgment isn't something like similarity,
but some other kind of causal reasoning.
So for example, consider these things here.
Poodles can bite through wire, therefore German shepherds
can bite through wire.
Is that a strong argument or weak?
Compare that with, dobermans can bite through wire, therefore
German shepherds can bite through wire.
So how many people think that the top argument is a stronger
one?
How many people think the bottom line is a stronger one?
So that's typical.
About twice as many people prefer the top one.
Because intuitively-- do I have a little thing
that will appear?
Intuitively, anyone want to explain why you thought so?
AUDIENCE: Poodles are really small.
JOSH TENENBAUM: Poodles are small or weak.
Yes.
And German shepherds are big and strong.
And what about dobermans?
AUDIENCE: They're just as big as German shepherds.
JOSH TENENBAUM: Yeah.
That's right.
So they're more similar to German shepherds,
because they're both big and strong.
But notice that something very different is going on here.
It's not about similarity.
It's sort of anti-similarity.
But it's not just anti-similarity.
Suppose I said, German shepherds can bite through wire,
therefore poodles can bite through wire.
Is that a good argument?
AUDIENCE: No.
It's an argument against.
JOSH TENENBAUM: No.
It's sort of a terrible argument, right?
So there's some kind of asymmetric dimensional
reasoning going on.
Or similarly, if I said, which of these seems better
intuitively; Salmon carry some bacteria,
therefore grizzly bears are likely to carry it,
versus grizzly bears carry this, therefore
salmon are likely to carry it.
How many people say salmon, therefore grizzly bears?
How many people say grizzly bears, therefore salmon?
How do you know?
Those who-- yeah, you're right.
I mean, you're right in that's what people say.
I don't know if it's right.
Again, I made it up.
But why did you say that, those of you who said salmon?
AUDIENCE: Bears eat salmon.
JOSH TENENBAUM: Bears eat salmon.
Yeah.
So assuming that's true, so we're told or see on TV,
then yeah.
So anyway, these are these different kinds
of things that are going on.
And to cut to the chase, what we showed
is that you could capture these different patterns of reasoning
with, again, the same kind of thing, but different.
It's also a hierarchical generative model.
It also has, the key level of the hierarchy
is some kind of directed graphical structure
that generates distribution on observable properties.
But it's a fundamentally different kind of structure.
It's not just a tree or a space.
It might be a different kind of graph
and a different kind of process.
So to be a little bit more technical, the things
I showed you with the tree and the low-dimensional space,
they had a different geometry to the graph,
but the same stochastic process operating over it.
It was, in both cases, basically a diffusion process.
Whereas to get the kinds of reasoning that you saw here,
you need a different kind of graph.
In one case it's like a chain to capture a dimension of strength
or size, say.
In the other case, it's some kind of food web thing.
It's not a tree.
It's that kind of directed network.
But you also need a different process.
So the ways-- the kind of probability model
to find that out is different.
And it's easy to see on the--
for example-- on the reasoning with these threshold things,
like the strength properties, if you
compare a 1D chain with just symmetric diffusion,
you get a much worse fit people's
judgments than if you'd used what we called this drift
threshold thing, which is basically a way of saying,
OK, I don't know.
There's some mapping from strength to being
able to bite through wire.
I don't know exactly what it is.
But the higher up you go on one, it's
probably more likely that you can bite--
that you can do the other.
So that provides a wonderful model of people's judgments
on these kind of tasks.
But that sort of diffusion process,
like if it was like mutation in biology, then that
would provide a very bad model.
That's the second row here.
Similarly, this sort of directed kind of noisy transmission
process on a food web does a great way of modeling people's
judgments about diseases, but not
a very good way of modeling people's judgments
about these biological properties.
But the tree models you saw before that
do a great job of modeling people's
judgments about the properties of animals,
they do a lousy job of modeling these disease judgments.
So we have this picture emerging that, at the time,
was very satisfying to us.
That, hey, we can take this domain of, say,
animals and their properties, or the various things we
can reason about, and there's a lot of different ways
we can reason about just this one domain.
And by building these structured probabilistic models
with different kinds of graphs structures
that capture different kinds of causal processes,
we could really describe a lot of different kinds
of reasoning.
And we saw this as part of a theme
that a lot of other people were working on.
So this is-- I mentioned this before,
but now I'm just sort of throwing it all out there.
A lot of people at the time-- again,
this is maybe somewhere between 5 to 10 years ago--
more like six or seven years ago--
we're extremely interested in this general view
of common sense reasoning and semantic cognition
by basically taking big matrices and boiling them
down to some kind of graph structure.
In some form, that's what Tom Mitchell was doing,
not just in the talk you saw, but remember,
he said there's this other stuff he does--
this thing called NELL, the Never Ending Language Learner.
I'm showing a little glimpse of that
up there from a New York Times piece on it in the upper right.
In some ways, in a sort of at least more implicit way,
it's what the neural networks that Jay McClelland, Tim
Rogers, Surya were talking about do.
And we thought-- you know, we had good reason
to think that our approach was more
like what people were doing than some of these others.
But I then came to see-- and this was around the time when
CBMM was actually getting started--
that none of these were going to work.
Like the whole thing was just not going to work.
Liz was one of the main people who convinced me of this.
But you could just read the New York Times
article on Tom Mitchell's piece, and you can see what's missing.
So there's Tom, remember.
This was an article from 2010.
Just to set the chronology right, that was right around--
a little bit after Charles had finished all that nice work
I showed you, which again, I still think is valuable.
I think it is capturing something
about what's going on.
It was very appealing to people, like at Google,
because these knowledge graphs are very much like the way,
around the same time, Google was starting
to try to put more semantics into web search--
again, connected to the work that Tom was doing.
And there was this nice article in the New York
Times talking about how they built their system by reading
the web.
But the best part of it was describing
one of the mistakes their system made.
So let me just show this to you.
About knowledge that's obvious to a person,
but not to a computer-- again, it's
Tom Mitchell himself describing this.
And the challenge of, that's where NELL has to be headed,
is how to make the things that are obvious to people
obvious to computers.
He gives this example of a bug that
happened in NELL's early life.
The research team noticed that--
oh, let's skip down there.
So, a particular example--
when Dr. Mitchell scanned the baked goods category recently,
he noticed a clear pattern.
NELL was at first quite accurate, easily
identifying all kinds of pies, breads, cakes, and cookies
as baked goods.
But things went awry after NELL's noun phrase classifier
decided internet cookies was a baked good.
NELL had read the sentence "I deleted my internet cookies."
And again, think of that as, it's kind
of like a simple proposition.
It's like, OK.
But the way it parses that is cookies
are things that can be deleted, the same way you can
say horses have T9 hormones.
It's basically just a matrix.
And the concept is internet cookies.
And then there's the property of can be deleted,
or something like that.
And it knows something about natural language processing.
So it can see--
and it's trying to be intelligent.
Oh, internet cookies.
Well, maybe like chocolate chip cookies and oatmeal raisin
cookies, those were a kind of cookies.
Basically, that's what it did.
Or no, actually did the opposite.
[LAUGHS]
It said-- when it read "I deleted my files,"
it decided files was probably a baked good, too.
Well, first it decided internet cookies was a baked good,
like those other cookies.
And then it decided that files were a baked goods.
And it started this whole avalanche of mistakes,
Dr. Mitchell said.
He corrected the internet cookies error
and restarted NELL's bakery education.
[LAUGHS] I mean, like, OK.
Now rerun without that problem.
So the point, the lesson Tom draws from this,
and that the article talks about,
is, oh, well, we still need some assistance.
We have to go back and, by hand, set these things.
But the key thing is that, really--
I think the message this is telling
us is no human child would ever make this mistake.
Human children learn in this way.
They don't need this kind of assistance.
It's true that, as Tom says, you and I don't
learn in isolation either.
So, all of the things we've been talking about,
about learning from prior knowledge and so on, are true.
But there's a basic kind of common sense thing
that this is missing, which is that at the time
a child is learning anything about--
by the time a child is learning anything about computers,
and files, and so on, they understand well before that,
like back in early infancy, from say, work that Liz has done,
and many others, that cookies, in the sense of baked goods,
are a physical object, a kind of food, a thing you eat.
Files, email-- not a physical object.
And there's all sorts of interesting stuff
to understand about how kids learn that a book can be both
a no-- a novel is both a story and it's also
a physical object, and so a lot of that stuff.
But there's a basic common sense understanding
of the world as consisting of physical objects,
and for example, agents and their goals.
You heard a little bit about this from us,
from me and Tomer, on the first day.
And that's where I want to turn to next.
And this is just one of many examples that we realized,
as cool as this system is, as great as all this stuff is,
just trying to approach semantic knowledge and common sense
reasoning as some kind of big matrix completion
without a much more fundamental grasp of the ways in which
the world is real to a human mind,
well before they're learning anything about language
or any of this higher level stuff,
it was just not going to work, in the same way
that I think if you want to build a system that learns
to play a video game, even remotely like the way
a human does,
there's a lot of more basic stuff you have to build on.
And it's the same basic stuff, I would argue.
A cool thing about Atari video games
is that, even though they were very low resolution,
very low-bit color displays, with very big pixels, what
makes your ability to learn that game work is
the same kind of thing that makes the ability, even as
a young child, to not make this mistake.
And it's the kind of thing that Liz and people
in her field of developmental psychology--
in particular, infant research-- have been studying really
excitingly for a couple of decades.
That, I think, is as transformative for the topic
of intelligence in brains, minds, and machines
as anything.
So that's what motivated the work we've
been doing in the last few years and the main work we're
trying to do in the center.
And it also goes hand-in-hand with the ways in which
we've realized that we have to take what we've learned how
to do with building problematic models over interesting
symbolically-structured representations
and so on, but move way beyond what you could call--
I mean, we need better, even more interesting,
symbolic representations.
In particular, we need to move beyond graphs
and stochastic processes defined over graphs to programs.
So that's where the probabilistic programs
come back into the mix.
So again, you already saw this.
And I'm trying to close the loop back
to what we're doing in CBMM.
I've given you about 10 to 15 years
of background in our field of how we got to this, why
we think this is interesting and important,
and why we think we need to--
why we've developed a certain toolkit of ideas,
and why we think we needed to keep extending it.
And I think, as you saw before, and as you'll see,
this also, in some ways--
I think we're getting more and more to the interesting part
of common sense.
But in another way, we're getting back to the problems
I started off with and what a lot
of other people at this summer school
have an interest in, which is things
like much more basic aspects of visual perception.
I think the heart of real intelligence and common sense
reasoning that we're talking about
is directly connected to vision and other sense modalities,
and how we get around in the world and plan our actions,
and the very basic kinds of goal social
understandings that you saw in those little videos
of the red and blue ball, or that you
see if you're trying to do action recognition and action
understanding.
So in some sense, it's gotten more cognitive.
But it's also, by getting to the root of our common sense
knowledge, it makes better contact with vision,
with neuroscience research.
And so I think it's a super exciting development
in what we're doing for the larger Brains, Minds,
and Machines agenda.
So again, now we're saying, OK, let's
try to understand the way in which--
even these kids playing with blocks,
the world is real to them.
It's not just a big matrix of data.
That is a thing in their hands.
And they have an understanding of what
a thing is before they start compiling lists of properties.
And they're playing with somebody else.
That hand is attached to a person, who has goals.
It's not just a big matrix of rows and columns.
It's an agent with goals, and even a mind.
And they understand those things before they
start to learn a lot of other things, like words for objects,
and advanced game-playing behavior, and so on.
And when we want to talk about learning,
we still are interested in one-shot learning,
or very rapid learning from a few examples.
And we're still interested in how prior knowledge guides
that, and how that knowledge can be built.
But we want to do it in this context.
We want to study in the context of, say,
how you learn how magnets work, or how you learn
how a touchscreen device works-- really interesting kinds
of grounded physical causes.
So this is what we have, or what I've come
to call the common sense core.
Liz, are you going to talk about core knowledge at all?
so there's a phrase that Liz likes
to use called core knowledge.
And this is definitely meant to evoke that.
And it's inspired by it.
I guess I changed it a little bit,
because I wanted it to mean something
a little bit different.
And I think, again, to anticipate a little bit,
the main difference is--
I don't know.
What's the main difference?
The main difference is that, in the same way
that lots of people look at me and say, oh, he's
the Bayesian guy, lots of people look at Liz
and say, oh, she's the nativist gal or something.
And it's true that, compared to a lot of other people,
I tend to be more interested, and have
done more work prominently associated
with Bayesian inference.
But by no means do I think that's the whole story.
And part of what I tried to show you, and will keep showing you,
is ways in which that's only really
the beginning of the story.
And Liz is prominently associated,
and you'll see some of this, with really fascinating
discoveries that key high level concepts, key kinds
of real knowledge, are present, in some sense,
as early as you can look, and in some form, I think,
very plausibly, have to be due to some kind of innately
unfolding genetic program that builds a mind the same way it
builds a brain.
But just as we'll hear from her, that's,
in some ways, only the beginning, or only one part
of a much richer, more interesting story
that she's been developing.
But for that, among other reasons,
I'm calling it a little different.
And I'm trying to emphasize the connection to what people in AI
call common sense reasoning.
Because I really do think this is the heart of common sense.
It's this intuitive physics and intuitive psychology.
So again, you saw us already give an intro to this.
Maybe what I'll just do is show you a little bit more
of the-- well, are you going to talk about the stuff at all?
LIZ SPELKE: I guess.
Yeah.
JOSH TENENBAUM: Well, OK.
So this is work-- some of this is based on Liz's work.
Some of this is based on Renée Baillargeon,
a close colleague of hers, and many other people out there.
And I wasn't really going to go into the details.
And maybe, Liz, we can decide whether you want to do this
or not.
But what they've shown is that, even prior
to the time when kids are learning words for objects,
all of this stuff with infants, two months, four months, eight
months--
at this age, kids have, at best, some vague statistical
associations of words to kinds of objects.
But they already have a great deal
of much more abstract understanding
of physical objects.
So I won't-- maybe I should not go into the details of it.
But you saw it in that nice video of the baby playing
with the cups.
And there's really interesting, sort
of rough, developmental timelines.
One of the things we're trying to figure out in CBMM
is to actually get much, much clearer picture on this.
But at least if you look across a bunch of different studies,
sometimes by one lab, sometimes up by multiple labs,
you see ways in which, say, going from two months to five
months, or five months to 12 months,
kids seem to-- their intuitive physics of objects
is getting a little bit more sophisticated.
So for example, they tend to understand--
in some form, they understand a little bit of how collisions
conserve momentum, a little bit, by five months or six months--
according to one of Baillargeon's studies--
in the sense that if they see a ball roll down a ramp
and hit another one, and the second one goes
a certain distance, if a bigger object comes,
they're not too surprised if this one goes farther.
But if a little object hits it, then they are surprised.
So they expect a bigger object to be able to move it
more than a little object.
But a two-month-old doesn't understand that.
Although a two-month-old does understand-- this is, again,
from Liz's work--
that if an object is colluded by a screen,
it hasn't disappeared, and that if an object
is rolling towards a wall, and that wall looks solid,
that the object can't go through it, and that if it somehow--
when the screen is removed, as you see on the upper left--
appears on the other side of the screen, that's
very surprising to them.
I think-- I'm sure what Liz will talk about, among other things,
are the methods they use, the looking time methods to reveal
this.
And I think there's really--
this is one of the two main insights that I,
and I think our whole field, needs
to learn from developmental psychology,
is how much of a basic understanding of physics
like this is present very early.
And it doesn't matter whether it's--
in some sense, it doesn't matter for the points
I want to make here, how much or in what way this is innate,
or how the genetics and the experience interact.
I mean, that does matter.
And that, that's something we want to understand,
and we are hoping to try to understand
in the hopefully not-too-distant future.
But for the purpose of understanding
what is the heart of common sense,
how are we going to build these causal, compositional,
generative models to really get at intelligence,
the main thing is that it should be about this kind of stuff.
That's the main focus.
And then the other big insight from developmental psychology,
which has to do with how we build this stuff,
is this idea sometimes called the child as scientist.
The basic idea is that, just as this early commonsense
knowledge is something like a scientific theory,
something like a good scientific theory, the way Newton's laws
are a better scientific theory than Kepler's laws
because of how they capture the causal structure of the world
in a compositional way.
That's another way to sum up what
I'm trying to say about children's early knowledge.
But also, the way children build their knowledge is something
like the way scientists build their knowledge, which
is, well, they do experiments, of course.
We normally call that play.
That's one of Laura's Schulz's big ideas.
But it's not just about the experiments.
I mean, Newton didn't really do any experiments.
He just thought.
And that's another thing you'll hear from Laura, and also
from Tomer, is that a lot of children's learning
looks less like, say, stochastic gradient descent, and more
like scratching your head and trying to make sense of,
well, that's really funny.
Why does this happen here?
Why does that happen over there?
Or how can I explain what seemed to be
diverse patterns of phenomena with some common underlying
principles, and making analogies between things,
and then trying out, oh, well, if that's right,
then it would make this prediction.
And the kid doesn't have to be conscious of that the way
scientists maybe are.
That process of coming up with theories
and considering variations, trying them out,
seeing what kinds of new experiences you can create
for yourself--
call them an experiment, or call them
just a game, or playing with a toy,
but that dynamic is the real heart of how children learn
and build the knowledge from the early stages
to what we come to have as adults.
Those two insights of what we start with and how we grow,
I think, are hugely powerful and hugely important
for anything we want to do in capturing-- making machines
that learn like humans, or making
computational models that really get at the heart of how
we come to be smart.