The following content is provided under a Creative
Commons license.
Your support will help MIT Open CourseWare
continue to offer high-quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu
WILLIAM BONVILLIAN: Rasheed, why don't you
give us that point that you were making to me about DARPA.
RASHEED: Yes, so DARPA relies very heavily on this program
manager structure.
So they give these program managers a lot of power
and a lot of leeway to do whatever they want.
But they have-- they have to have a very distinct vision
from the outset, like really when they
start to be able to do things.
And so I was concerned that these program managers were
going to basically be coming from the same places
over and over again, and so have very similar visions.
And you're going to be able to miss out
on a lot of different talent that
exists outside, or a lot of different new projects,
or new fields, maybe, just like you're talking
about-- these new collaboration between biology
and kind of humans and computer science, things like that.
But that was really my main concern,
was program managers weren't being chosen as--
equitably isn't the word, but that the selection
process of these program managers was actually--
might be too narrow.
WILLIAM BONVILLIAN: But they do get visionaries.
STUDENT: Sometimes.
I mean, I think we mentioned that they only
got white males [INAUDIBLE].
WILLIAM BONVILLIAN: OK, so I'm going
to move us to Tammy Carleton.
And Tammy's big perception about DARPA
is this vision perception, that that's a critical factor that
makes DARPA different.
And there is a process that DARPA
has developed for this radical technological visioning.
And essentially, she argues that there's six stages that they
use to kind of get there.
First, recruitment of great talent--
and the talent, they hear about talent
from this DARPA network, which is now quite huge.
Because all of the people that get DARPA contracts and awards,
they stay in this kind of DARPA family.
And typically, researchers love to work for DARPA.
Because working on DARPA-hard problems is really intriguing.
So that's a community that tends to know each other,
and be in communication, as well as
the community of former DARPA program managers and officials.
So a lot of the talent comes through that network,
which kind of knows what's going to work.
So the recruitment and the vision get united in a way,
in that recruitment process.
Then comes the stage of the vision formulation.
The program manager has got to figure out
what they're going to do with their four or five years.
And they've got to formulate a radical vision,
a right/left vision, what's going to come out
of that pipeline, and what's the fundamental research that
going to get me there.
And it's driving for that breakthrough that's
the organizing proposition.
Then, they've got to go through defining
a program and its launch.
Often, they will have a workshop,
an invitation-only workshop, where
they'll assemble 25, 30, 40 of the kind of best thinkers
on a project area, and meet for a day or two days.
I've watched one of those, which got held up at MIT as part
of the beginnings of creating this Biological Technologies
office.
And they brought in just an amazing collection,
from MIT and from elsewhere, of thinkers
about what some of the early projects
might be to achieve some of these capabilities.
So the Deputy Director of DARPA came up,
and kind of helped run this workshop
and think this through.
The office wasn't set up for a while
until the next DARPA director, but it really
helped contribute to that portfolio.
And it was a really serious effort.
So the program manager develops a portfolio approach
of standing up a series of groups
that are going to be working on this, and inter-relating them.
And the program manager is also going
to be responsible for making this technology transfer.
So in other words, it's not enough just
to do something cool.
You've got to move it into the implementation stage.
And how is that going to happen?
So the program manager has got to undertake
all of these various steps.
So the criteria for being DARPA hard
is that the project has to be technologically very
challenging.
It's got to extend beyond the current limits of technology
and knowledge.
Otherwise, they don't really want to do it.
It's got to be actionable, which means it can be made.
It can be built. It can be produced.
So if it can't be, why do it?
You've got to be able to demonstrate
that it's actionable.
Typically multidisciplinary-- that it's
going to draw on a variety of areas of expertise.
It's got to be far reaching.
It's got to change things.
It's got to be very ambitious at a big scale.
"Don't do little things" is kind of a DARPA sub-rule.
And various techniques to flesh out the vision
include those expert-only workshops,
and a proof of concept effort.
Sometimes, if it's some idea that looks promising,
they'll do what DARPA calls a "seedling," just put
a little money on it, and let somebody play
with it, a real thinker play with it,
and see what it might materialize into.
So they've got that seedling capability.
There is no training system for program managers.
They don't go through any kind of training.
They just arrive.
And they kind of learn from other program managers
what they're supposed to do.
So there's a very strong kind of DARPA culture
about how it operates, that gets communicated by word of mouth
and kind of learning by doing.
And it's informal exchange, but there is a very strong kind
of operating rules set.
DARPA program managers really have
to be quite entrepreneurial.
And Martine, you were raising this earlier.
They've got to stand up a whole new territory.
And to do that, they've got to be
highly entrepreneurial about how to bring that about.
And the decision to stand up a vision program--
this is not set up through peer review.
There isn't consensus decision-making at DARPA.
That program manager has got to have a really strong vision,
and it's got to get really organized.
And then it's got to withstand the office director
and then the DARPA director really tearing through it,
and really analyzing it.
So once you get that, though, the approvals
can be very fast, once the decision is
made to stand something up.
And that the program manager really is--
to go back to that term that you liked, Steph--
a vision champion.
They've got to champion this vision and make it happen.
That's what the program manager has got to do.
So DARPA starts with division.
That is not how industry operates.
That's not how other government agencies operate.
Industry works largely on what's called a "stage gate system."
So industry is going to want to do typically
more incremental kind of advances,
because the advances can snap onto markets that it's already
got, and economic opportunities that it already sees,
and technologies that it's already starting to explore.
So it's typically going to want to do
more incremental advances.
And it's only going to do those incremental advances if they
can optimize them economically.
So an industry R&D project has got
to go through a series of stage gates, which essentially
is a way of tearing down a large menu of stuff
and getting to the most economically optimal.
Industry stage gate process is not a technology visioning
system.
It's not deciding on a vision and doing
whatever is necessary to get there, which
is the way DARPA operates.
It's a much more limited set of procedural stage gates
that a technology idea has got to go through.
So DARPA just working on a very different kind
of organizational model from the way in which things are done,
either at other agencies or within industry.
And starting with this vision up front
is a really interesting organizational idea,
that you all should kind of keep in mind when you're
operating out there, and setting up your own companies,
and doing your own startups.
So Max, you've got Tammy Carleton's piece.
MAX: Right.
The key step within Carleton's five steps
that I thought was most interesting
was the fifth one, technology transfer, which
is supposed to bridge the R&D--
the valley of death, as it were, that
enables a different scientific idea to be commercialized,
and to actually affect people all over the world, eventually.
While reading, I kind of realized
that DARPA seems like they have a pretty decent organization.
They get things done relatively efficiently.
They have a lot of money.
Why is it they have--
and I actually just decided to start googling,
to try to figure out if DARPA had done anything related
to fusion.
And I couldn't find anything.
So I'm curious, why not?
WILLIAM BONVILLIAN: Well, actually,
DARPA deferred on energy technologies
to ARPA-E, when ARPA-E got stood up,
because DARPA views ARPA-E as a compatriot.
And ARPA-E interestingly, has done some extremely interesting
work on fusion.
STUDENT: I know Lockheed [INAUDIBLE] fusion--
WILLIAM BONVILLIAN: Right.
STUDENT: [INAUDIBLE]
MAX: Yeah, but Lockheed Martin's--
WILLIAM BONVILLIAN: Well, they have the Skunk Works.
MAX: They do.
WILLIAM BONVILLIAN: And DARPA's done an enormous amount of work
with the Skunk Works over time, including stealth.
MAX: Granted, yes-- they definitely have a facility,
and they definitely have the project managers.
I just feel that it's a bit of an area
outside of their expertise.
They're going to accidentally make it fly and stuff.
WILLIAM BONVILLIAN: Right.
And look-- another issue for DARPA is-- remember,
these technologies have got to get stood up
in the lifetime of the project.
MAX: Yeah, five years.
WILLIAM BONVILLIAN: Like a five year project.
So if you can't do it within five years,
it's very hard to put it on the DARPA agenda.
MAX: Well, you mentioned that some of the projects
would get handed off to other project managers.
WILLIAM BONVILLIAN: Yes.
MAX: So that's how long a manager lasts, right?
WILLIAM BONVILLIAN: Yes.
MAX: But it's not always how long a project lasts.
WILLIAM BONVILLIAN: Yes.
Sometimes, there can be intergenerational transfers.
And then Licklider developed a different kind of system.
He created a community that kept continuing
with the same kind of ethos, working
on that same set of agenda items,
working them down one after another,
and making progress year after year on each of them.
So that's a different kind of way
that DARPA is going to be organized.
But typically, once the program manager
leaves who had the vision, their job
is to get that implemented in their, in effect,
DARPA lifetime.
And if they haven't, then it's hard for DARPA
to continue that, which we could view
as a weakness of the program.
On the other hand, you could view it as a strength.
MAX: Yeah, because then they make sure that things--
that if they are working, then they keep--
they may proceed with it, or they pass it off
to someone else.
WILLIAM BONVILLIAN: Yes.
MAX: So speaking of Licklider, actually, while I was reading,
I noticed that Licklider had mentioned that he heavily
relied on standardized tests when picking candidates,
so like that recruitment of great talent.
Given that literally everyone in this room
has experience with standardized testing, and at least
I can't speak for everyone, but I don't like it very much.
I wonder--
STUDENT: [INAUDIBLE] two tests.
MAX: Which two?
It was ACTs.
STUDENT: One was like a psych test,
and the other one was like a [INAUDIBLE]..
STUDENT 1: GRE.
We did the GRE, it said.
STUDENT: Well, there's another one,
but it seemed like a psychology test.
I'm not sure.
STUDENT 1: So I think that she mentioned
that he was the only one who asked for test scores at all.
And one was the GRE, which is kind of weird.
WILLIAM BONVILLIAN: Remember, he was a psychologist coming out
of the testing world.
So that's probably part of the reason.
MAX: I'll chalk it up to [INAUDIBLE],, because there's
only so much a standardized test can tell you.
They won't really tell you how you work with people.
They won't tell you anything about how
you work on anything that's not related to, I don't know,
math or literature.
WILLIAM BONVILLIAN: He did come up with a pretty talented team,
though.
MAX: Yeah, that's true.
STUDENT: I mean, you get pretty far on psychology tests,
especially when it comes to intertia.
Because you have to figure out, for really good visionary
leaders, what they'll figure out is
they're intellectuals, introverts, extroverts,
and they have a supporting side to their psyche.
And they're kind of like the avatar of team groups.
So it's pretty good for if you want to find out good leaders.
And I know it's been pretty effective in manageent theory
when you use psych tests.
It's also really good to figure out
who is going to be a really good scientist,
because it's usually highly introverted
and comes up with new insights that doesn't follow the crowd.
So I can see why that's important.
The GRE might be, but also, if your score is low,
but if you've come out with a good body of work,
then they can probably assume that you
are pretty smart, even if you didn't do well
on standardized tests.
STUDENT 2: Because I'm looking up her LinkedIn.
And it says that she has-- her background is in communication.
And then she got a Master's in Public Relations.
And then she did a PhD in Mechanical Engineering Design.
But that must have been back in the day when the D School
at Stanford accepted non-engineers
and non-scientists, because now they don't.
So it would be curious to think about the ways in which
her background, in particular, might give her that insight
that an engineer might not have about this particular field,
going back to the question of multidisciplinary great groups.
But I don't know to what extent, for example,
someone like me might be accepted in maybe Lilly's lab.
LILLY: One of the most famous women in my field
has an undergraduate degree in German.
So you never know.
STUDENT: Lloyd Blankfein, the CEO of J P Morgan, I think--
his degree's in history.
And he almost barely didn't pass.
He said he still gets nightmares,
because he didn't fulfill all the requirements.
WILLIAM BONVILLIAN: All right, so we're
going to move out of the testing world back to the DARPA world.
MAX: Sure.
So one of the things that Carleton pointed out
was that as DARPA has aged, so, too, have its project managers.
Even though they cycle between new managers every four or five
years, the average age of the managers that they would pick
was going up steadily.
So I'm curious if anyone has thoughts
on what the effect of that would be on their managerial ability,
or on DARPA's direction as a whole.
STUDENT: Could you repeat that first part?
MAX: Yeah.
So the average age of the project managers
has been increasing, as DARPA has--
STUDENT: What's the average age?
MAX: I don't remember the numbers.
STUDENT: Is it 27?
WILLIAM BONVILLIAN: No, it's a lot older than that.
STUDENT: It's a lot older?
MAX: Yeah.
WILLIAM BONVILLIAN: One of the reasons, Max,
may be that it's--
I mean, watching MIT, it's very hard for younger faculty
to walk out of the place before they get tenure.
It's hard enough to walk out once you get tenure,
but it's impossible to walk out before you get tenure.
And because you're establishing a whole--
your research portfolio and your whole researching system,
and it's very hard to walk out of that.
In addition, when you go to DARPA, you've got a conflict.
Once you've worked there, you have a conflict
of interest with DARPA.
You're not going to be able to deal
with the agency for a certain period of time.
So if they happen to be funding your research before,
they're not going to be able to continue funding your research
once you've worked there.
So it's gotten more complicated for DARPA
to recruit at the university level.
It's gotten a lot more difficult.
So the route between MIT and DARPA,
frankly, used to be a lot easier to manage.
And when I look at my colleagues here who have gone to DARPA,
they are more typically fairly far along,
and very assured about their career, and their research
foundation, and their writing.
They've done that stuff, and they
can take a three or five year break,
and get that really big, visionary thing
that they've wanted to work on forever done
when they go to DARPA.
But it's harder when you're trying to administer
a whole research portfolio.
Again, with the IPA authority, you've
got a certain freedom to come back to your organization
and keep the day-to-day stuff going.
But it's still not simple.
So that may be one part of the reason
that you tend to get somewhat older faculty than they
probably used to at the beginning.
STUDENT 1: Licklider wasn't especially young
when he was the director.
WILLIAM BONVILLIAN: Right, he wasn't young.
STUDENT 1: Or the project manager.
So I think age does affect the perspective that you bring in.
But having older people isn't a bad thing.
I think in the tech world, it's seen
as like if you're not young and hip,
you're going to be out of touch with what's going to work.
But I don't know if that's a--
I think the points that Bill made
about having to kind of establish yourself and figure
out what your vision is, you have more time when you're
older to have a vision.
LILLY: Yeah, and perspective as well.
Just thinking about advisors I've had over the years,
and younger or newer advisors versus more
established advisors-- they definitely
have a different perspective.
And yeah, NASA is actually having the same phenomenon,
with having directors who are progressively older and older.
So that's very sim-- they can't--
it's very difficult to draw someone away
from a mid-range career if they're being very successful,
and having success getting grants.
And it's hard to get them into administration or directorships
or management.
STUDENT: What would the difference
between young and older directors be?
LILLY: In general, actually, the more established and older
advisors that I've had have had a lot more
perspicacity with respect to what projects are worthwhile,
actually, or--
WILLIAM BONVILLIAN: Through long experience,
you're suggesting, because they've seen a lot.
LILLY: Yeah, that's the impression I get.
WILLIAM BONVILLIAN: Right.
Next, how about another question?
MAX: Sure.
All right.
So Carleton mentioned that DARPA,
in order to save time when they are trying
to develop these projects, they try
to discourage consensus, which I didn't really see as a--
I think it was the second, the next slide you have.
Yeah.
So I understand why, because then it
ensures that whatever you're trying to develop,
it gets done quickly.
But just because it gets done quickly
doesn't really mean that it gets done right.
So I guess I was curious what people thought on that,
specifically that decision.
WILLIAM BONVILLIAN: It's a very interesting point, Max.
And actually, ARPA-E-- and we'll talk about it
when we get to ARPA-E and the energy class,
but ARPA-E does have a process.
It's a much smaller agency, so it's 300 million, not 3 billion
like DARPA.
So it's like a big DARPA office, sizable DARPA office,
one of its five or six offices.
So ARPA-E does have a consensus process.
They bring in-- they call them "project directors"
but they're program managers.
And that community, really, you have
to present your vision to that community.
And that's a very tough-minded group, and a very sophisticated
group.
And the director is quite sophisticated at ARPA-E
historically, as well.
So they did a variation on this.
At DARPA, it's much more, convince your office director,
and then the office director will work with you
on convincing the director of DARPA.
It's a simpler, more straightforward process.
You do get support from other program
managers, who will have a lot of advice and ideas for you.
So it's a pretty supportive operation.
Nobody's in competition with each other.
They're there to help each other,
so that's a positive, although again, they're
competing to achieve their vision.
And everybody kind of knows who's
getting their vision done, and kind of who is
in trouble with their vision.
So that's kind of known and understood.
There's a certain kind of competition in that way,
but it's not a direct one-on-one competition with each other.
But at ARPA-E, there is more of a consensus decision making.
And Luyao when she was telling us about Xerox PARC,
remember, your whole discussion about the dealer process
and having to bring your idea to the whole team sitting around
on beanbags, who would tear it apart--
that's kind of a consensus process.
It's a definite, tough-minded review process
that DARPA doesn't really quite have.
So there may be strengths and weaknesses in both models.
Max, what do you think on that?
MAX: Yeah, I could see there'd be strengths and weaknesses
to both.
But it just feels if you have less consensus, than you have--
if you have fewer people that are trying
to discuss these ideas, then there
might be some critical flaw that you might not--
that you might just miss, just because there are fewer people
with more diverse experiences.
Of course, if you have a decent idea of the science behind it,
and it's not something as untested
as fusion, for example, then I could
see it working pretty well.
I mean, if you had to do that for--
I could see it having problems with something untested
like stealth.
But apparently, they made it work.
So maybe they're passing my expectations.
STUDENT: [INAUDIBLE] Silicon Valley perspective,
like it's a pretty common mantra to say
you want to have an idea that people kind of say
is not a good idea.
Because the idea is, if you're coming up
with something that's pretty innovative,
most likely people won't agree with it,
or it doesn't make sense, or you have an insight
that other people don't get.
Also, if you're an expert, you probably
know some things that other people don't know,
that you'd have to bring them up to speed on, especially if it's
in a completely new area.
And so the history has been, they're
just saying that for new fields, who is the expert?
It wouldn't be a PhD, because the field
hasn't been invented yet.
It's this person who just started working on it.
So Bill Gates was an expert on software for personal computers
before that even became a thing.
So it's very common to-- consensus
is actually really bad, because you put down these ideas based
on new insights.
So.
LILLY: Yeah, and another issue with consensus
is you have to assume, or you have
to have a group in which all members have the personality
type that they will concede to someone else's idea
to get consensus, even if it's not
their pet project or their favorite.
You know what I mean?
Some people aren't disposed to do that, no matter what.
WILLIAM BONVILLIAN: About a closing thought on this, Max?
MAX: Sure.
So overall, I feel that DARPA seems
to have been a pretty great success.
They've made lots of really cool projects.
They do it very quickly.
Not sure how cheaply they do it, but they get it done.
And because of that continued success,
I feel that that's probably why they continue
to get so much funding.
I really try to emphasize that technology transfer aspect,
because it ensures that the technology that they develop
isn't just it's a nice, new toy for the military to have.
It ensures that whatever they make, from the internet,
GPS, et cetera, that everyone can use it,
and that it makes our entire society better.
So I guess that's pretty much just why I like DARPA.
WILLIAM BONVILLIAN: Great.
There's a lot of MIT folks and people saying, what do you do?
All right, let me do a quick wrap up of today's class.
So we did Mitch Waldrop's book.
And I really wanted to kind of portray
the governmental role in supporting the earlier stage
of the evolution of computing.
It was a proving ground for new concepts,
and designs, and architectures.
The Defense Department created an initial market
for a lot of the new products, and services,
and in fact, whole industries.
This greatly expanded university research capabilities,
the computing revolution.
And it was done to further governmental missions.
But it had, obviously, dramatic societal effects.
So it featured governmental agencies
playing a pretty central role.
DARPA, as we discussed was critical,
but some of the others were necessary and needed.
And the governmental agency sponsored
these industry-university collaborations
that led to great strength in the university side,
and focus areas around them.
So in the reading that we did from the textbook,
the central point here was that DARPA
was uniquely able to combine an innovation institutional role,
as well as a great groups innovation
at the face-to-face level kind of role.
That's a remarkable accomplishment.
And then DARPA's ability, as we talked about in that reading
as well, to innovate in a legacy sector,
to do a frontier sector like IT, but also
innovate within a legacy sector, and do things like stealth,
and UAVs--
that's a pretty fascinating organization,
that's flexible, and interesting,
and able to do great stuff.
Glenn Fong's piece was really about DARPA
playing the central institutional mobilization
role for the IT revolution.
And Tammy Carleton's piece really
told us about this kind of central importance
of technology visioning as a key modus operandi, that DARPA
been able to operate in.
Any closing questions?
Good luck with your paper outlines.
Let me know if you have questions.
And I look forward to seeing you on Tuesday.