The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from 100 of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
RICHARD EBERHARDT: Hi, Pablo.
PABLO SUAREZ: Good to be back.
RICHARD EBERHARDT: Nice having you here.
So we wanted to talk about a number
of high-level postmortemy things for how the class went.
So we've got some high-level things we wanted to talk about.
We want to talk about the collaboration between MIT
and the Red Cross/Red Crescent group and your folks--
how student preparedness went, in particular, comparing
those teams that got a lot of support, like client support,
like Snap, of course.
But also, some of the other teams got a little bit more,
and then the unsupported teams, which
aren't exactly unsupported, but they got less,
so that's largely the Heatwave Team,
I think was the main one that we want to talk about there.
So collaboration what do you all think about how
we talked to each other?
How we met with each other-- things like that?
PHILIP TAN: I like the way how we
started in the beginning of the semester.
We had to talk-- and we had some Skype issues.
But I think the idea of kicking off the entire class
with the initial presentation of this
is what your final project is going to be.
It's going to take us a while to get there,
but you come into class understanding
this is what it's going to be.
And Pablo being the person to describe
why this is going to be important,
I think, set lot of expectations well.
So I think later on, when we actually got to that project,
the degree to which each team actually
could have been supported actually
became more of an issue.
The teams that you could directly talk with--
and you were going to be the primary user of the game--
clearly benefited.
But you were right there from the beginning.
These are the teams that, maybe, they did receive support,
or some of them didn't receive support,
but weren't talking with people who
had been there right from the start, I think maybe
had felt a little bit more disconnected
from the main thrust of the class.
At least when they got to that final assignment.
So something I like to think about for future classes
is the extent to which-- if we're
going to repeat this format-- where the primary client comes
in right at the beginning to set the tone for the entire class.
How do we represent more of the variety of different things
that you can do right at the beginning?
So that when students think about what they're
going to be doing throughout the entire semester,
they are actually keeping full scope of the project in mind,
not just the one example that might be presented to them
on the first day of class.
PABLO SUAREZ: And we also may want
to have something like an engagement meter.
Because I wasn't aware of the fact
that some teams were feeling disengaged.
I would show up, some people have questions,
some other have less questions.
And from my perspective, those who have questions I spent time
and it was about the same chronometric time per group.
I could see that some were more lost,
but I wasn't aware that what was needed
was more fire in their belly.
So if we could--
RICHARD EBERHARDT: I couldn't see that either.
PABLO SUAREZ: Right.
RICHARD EBERHARDT: The students never reported it to us.
SARA VERRILLI: Yeah, I was going to say,
when we asked the students to do their postmortems,
they were really clear about how this had been a big motivation,
and interest had been a big problem for them.
But they never mentioned it to us during check-in meetings
when we were doing play testing, when we were working with them.
So that's the sort of thing that we
need to-- probably one of the things
you do is probably to warn them about that.
These are hard topics, and they can become boring topics
very quickly.
And they need to step up and let us know when they think
they're facing a brick wall.
RICHARD EBERHARDT: I think the clues were there.
If we had means and a way to remind ourselves,
like a checklist or something simple like that, just to say,
are you engaged?
What's your motivation?
And, again, that's assuming that we even
want that to be our job.
I think for anything client based,
it has be our, probably, our job.
SARA VERRILLI: I think that when we're
asking them to work on projects where they have relatively
little control over the topic, I think we actually do need to.
Because I think part of it was I don't
think the teams knew how much freedom they
had to deviate from the topic they had chosen
and the specific.
They thought they had chosen this thing,
and now they were stuck with it.
Even though one team was like, we
chose a topic that just wasn't deep enough.
There wasn't enough there.
And if they had come and talked to us,
we could have given them the freedom
to well, go change to a new topic, or how can we go--
RICHARD EBERHARDT: Go deeper.
SARA VERRILLI: --how can we go deeper in it?
Right.
But we were not sufficiently warned.
PHILIP TAN: But that's where rapport
with someone who knows a lot about the domain can help.
And if they don't feel like they have that rapport or someone
who they can readily contact to ask even seemingly
silly questions, then we're not going
to be able to get down to actually ask
the important questions.
Because sometimes they can't tell.
They can't necessarily tell what looks like flagging motivation
on a part of the team or what just
seems like end-of-semester pressures on the students,
right?
They haven't been working on enough projects
to be able to tell the difference.
And so we do need to help them through that, I think.
SARA VERRILLI: By that you mean the students, then?
PHILIP TAN: The students.
I mean the students.
PABLO SUAREZ: And one of the things you may want to consider
is we know that both in the real life-- in the future,
in their future, as well as in future courses--
there will be limited ability of those
who are experts to interact with the game design team.
The game design team, you took the effort--
and I thought it was very well-balanced-- to have some
who know programming, some who know narrative, some who
know user interface, and so on.
They will, by default, not have expertise
on heatwaves or cholera or etc or climate science.
They cannot assume that all they need to know--
including the appetite for engagement--
will be only in the client.
MIT is a bubbling powerhouse of engaging stuff.
And if they could, as part of what they are told-- they
will have to do in the future, and what
they have to do in this course, to reach out
to engaging people, fellow students, and junior faculty
or whoever may give them some time-- to just go and say,
what is exciting about this?
What are the cutting-edge areas?
What are the institutional challenges.
So that they also have to fuel their own appetite.
RICHARD EBERHARDT: Having an assignment
or having part of that final project being like here's
the research component that we expect you to do,
but we didn't put an underline over it.
SARA VERRILLI: We didn't make it explicit that they needed to--
RICHARD EBERHARDT: We did when it was in trouble.
Like for Saving Gora Gora, they were in big trouble.
We told them a number-- you told them, we told them.
I think you'd be surprised what they came up with.
Because they actually, I think, they took the baton
and went with it.
SARA VERRILLI: They did that [INAUDIBLE].
PHILIP TAN: Nice.
RICHARD EBERHARDT: The other cholera team did not,
and they were given the same-- almost
the exact same-- feedback.
That is the thing.
PHILIP TAN: I think there is a huge social barrier
between the undergraduates and the rest of the institute.
The noble laureates, and even the post-docs here,
that's a lot of social pressure keeping students away
from that kind of interaction.
You're all right.
All of these things do exist here at MIT, but some of this
is reinforced by professors themselves--
professors who keep their doors closed
when they in their office, things like that.
And so I do think we have to do give them the license,
and sometimes that's pointing them to specific people.
RICHARD EBERHARDT: Yeah, giving them resources like here's
a list.
One of these might pan out for you.
The others might not.
And that's OK, just do what you can, kind of prime them for it.
PHILIP TAN: It's hard to be a teenager in this space.
RICHARD EBERHARDT: Actually, since we're
talking about the student preparedness
in that research like what we saw between what we consider
the supported teams versus the unsupported teams, which
may or may not be fair-- could be the motivated
teams versus the unmotivated teams.
Snap, in particular, really--
PABLO SUAREZ: [INAUDIBLE] correlation.
RICHARD EBERHARDT: --really well-supported.
They actually reached out to you all for help with the game.
You actually even did testing for them that was actually
coming from you and Jana, yeah, because you
were running workshops.
PABLO SUAREZ: Right.
RICHARD EBERHARDT: How did those workshops run-- how did you
interact with the team?
PABLO SUAREZ: So, first of all, I
think it's important to distinguish not only what you
are calling supportive versus unsupportive,
but also Snap versus everything else.
Because Snap was a game concept that
preexisted the course that needed refinement
and digital interface.
So from that perspective, it's not surprising
that they have something clearly defined that they
could run faster and deeper.
And the group was a group that engaged and went
deep and went fast.
The interaction we had with them,
also, very importantly, we have events
that we could use the product.
So it was much easier to give feedback
saying when we do this, we expect
people that would get bored, why don't we try this other thing.
RICHARD EBERHARDT: It's milestones,
which we give them milestones.
But there is no weight behind them.
These ones have an actual, if it doesn't work by then,
someone's going to have egg on their face, right?
PABLO SUAREZ: And from that perspective, we know this,
but we can try to do more of giving students
more proportion of work that we know-- that we, the clients
know-- that we have milestones.
And that is limited by how much of the real world
gives us milestones to abide by.
But we can try that.
I do not feel comfortable with the label supportive versus
unsupportive because it sounds like some got time and support
and some others didn't.
I think it's more about the form of the relationship
and how the time was used or the opportunity to use the time
was used.
Because I have no doubt that you supported the team when
you knew support was needed.
And as far as I know, the same was true on our part.
I think it's more about having sensors
about when support is needed that is not being delivered.
RICHARD EBERHARDT: I think when we're talking about them
in this way, I'm using it based on how
they describe their own relationship
and their own experience.
So that the teams who we are calling unsupported
are the ones who, in their postmortems,
called out and said, we didn't feel
like we got client support.
They got the same amount.
It was a different way, different quality,
different quantity.
SARA VERRILLI: The way to put it is the same amount of support
was there for them.
One other way to put it is they didn't know
how to reach out and get it.
They didn't use the resources that
were there because, at some level,
we weren't pushing them at them.
And a lot of MIT courses tend to encourage students to solve
all their problems themselves.
They don't want them reaching out.
They want them-- figure this out.
And so it may come as more of a second--
it may not come to them as first nature to go, hey,
we need to go and reach out for more help and grab that--
RICHARD EBERHARDT: And so that's actually the same thing.
So what we saw with the Snap team is
we wanted-- the feedback they were getting from us
was look at your front end, look at your UI.
We wanted to see a lot of changes on the UI.
Because they were working on these milestones,
they were really worried about the back end
so they were doing a lot of work in the back end.
They were getting two different messages,
and they chose client message rather than instructor message.
Totally OK and totally viable.
We took that into account.
But I think the main issues we had
with them is when they got our message,
it was, they heard it early on.
They didn't react to it until it was too late-- had they
done a little bit extra in the beginning.
So one thing I think we need to think about with the class
is thinking about how are we helping them understand
how to prioritize feedback?
Right?
What's more important?
Could be because here's something that's
actually going to get used.
It could also be here's a lesson we want to make sure they get.
I think they got the lesson.
They just got it in a harder way.
PHILIP TAN: Sometimes they get a lesson at a point of time
that they can't actually execute on it.
But hopefully they can do it for their next project after class.
Something I want to suggest, actually,
because we've been talking about how it's been difficult
for us to figure out what's going on inside the teams.
We can see the games and the testing,
but the team dynamics, we don't find out about that
until near the end of the class.
And something that we used to do outside of this class,
but in our research work, was to get all off
the scrum masters together on a regular basis
to just do an exchange of information with each other.
They don't even have to be necessarily reporting to us--
although we can moderate those sessions.
It's more about them sharing their experiences with each
other, and then it may make it more obvious that there are
resources out there that some teams are taking advantage
of and some teams aren't.
Of course, we can use those situations for teaching,
but also for teaching each other.
SARA VERRILLI: Creating some group, group support.
RICHARD EBERHARDT: We talked about that
before this semester started, but we couldn't figure out
a way to do that fairly.
I already feel producers bear the brunt
of the work in the class.
They're the ones we see all the time.
We don't see the students working on programming
or design as much.
Design a little bit more, but programming,
we just almost never see them.
SARA VERRILLI: And we let the teams use the producers
as their face-- so giving all the presentations,
giving all the reports, answering questions when
we ask questions in class.
Think about how we convince the teams to share out that work
a little bit more evenly.
PHILIP TAN: Well, perhaps, maybe, we
can reduce the extent that we require
these producers to do presentations in class,
and replace that with these more closed-off sessions
where they don't feel like they have to be on stage
and put on the best face for their team.
They can actually just talk about this is the problem
I'm having with my team.
RICHARD EBERHARDT: So one thing I also
wanted to make sure that we talked
about what was it like for you?
How much of your time did we demand?
Was it a lot compared to other projects?
Was it a little bit?
Was it about moderate?
PABLO SUAREZ: I want to start by saying one of the things
that I really appreciated was the time we spent--
both face-to-face and emailing and so
on-- before the beginning of the course,
to have understanding of what you and your students
can offer, what we need and can offer,
and to manage expectations, as you said.
I think that was very well done.
And it's not always the case.
So with the students and professors and Red Cross,
in general, we [INAUDIBLE], so thank you.
That was really appreciated.
I think that amount of time that I personally had to give
was about as much as I anticipated, about as much
as I think is reasonable to expect for the future.
We do have Jana, in particular, and a few other members
of the team-- of the Red Cross team--
who were available to give a hand.
And I think we needed to improve because I
did make it very clear that I couldn't
get five emails per day asking questions.
But to have some kind of ability for the right contact when
either engagement levels are too low or mega confusion
levels are too high or sense of purpose is diluted and so on.
So to refine that.
I think it would be better to think of times
when there can be slots of decided interaction
between each team, and someone from the client organization
that doesn't have to be during class.
Because I would always interfere with your plan.
You gave us one opportunity that I
thought was very useful, but a little bit late
in the development process and also very crammed.
By the time we said, OK, these are
the kind of things you can do, there was no time
for them to think and reflect on,
OK, we can do A or B or C. Show that to us,
and then us providing feedback.
So even if it were one more chance via Skype
but scheduled, planned, and if-- it is in the nature of our work
to have to travel, and it is in the nature of our work
that travel timing changes outside of our control.
So where there's a default date, and if things have
to be changed, it is changed.
So to know that there's going to be an instance of feedback that
may be too little, too late, still but to anticipate that.
RICHARD EBERHARDT: Yeah, part of that is the students,
we don't always know the student's schedules.
So it could possibly be planned in the beginning
of the semester.
But planning it before the semester
would be really difficult. Unless we just say it--
this is the time that you're going to have.
If you don't take advantage of it, you don't get it.
PABLO SUAREZ: Right.
It has to be we propose it's this week.
This is the default time slot.
If not, that time slot, agree on a different time slot.
And then it's up to the students.
RICHARD EBERHARDT: So you're actually
saying you would have talked to them a little bit more.
PABLO SUAREZ: If the problem was lack
of engagement or perception of lack of support.
RICHARD EBERHARDT: So it's really
that planning and scheduling.
So as long as that's in place, then the time
is there, because it's been planned.
OK.
PABLO SUAREZ: And also I think it
would have been nice for them to have access to, say, someone
from the Ghana Team to look at the color, someone
from the Heatwave Team in Buenos Aires-- to at least
to get a flavor, there's another human being there that
eventually can use their game.
And I didn't do enough to help them envision
the user-- either the Red Cross user
or the community level or policy or government user.
PHILIP TAN: I think we need to state that strongly
that it's not just the information that you're
providing.
It's the fact that there is a real person who can really
benefit from the work you're doing is much more
powerful than some sort of abstracted audience.
RICHARD EBERHARDT: So we provided
plenty of documentation, but that's not
the same as a human being.
PHILIP TAN: That's not the emotional drive
that you're going to get-- oh, someone is
counting on me doing good work.
RICHARD EBERHARDT: So maybe it might just
be something for us to talk to the Terrascope folks
to see how they do that.
PHILIP TAN: Well, they specifically
get people who can commit on regular meetings.
And the idea is-- my understanding
is that it's for the domain knowledge primarily.
But you get that side benefit of here
is a real human being who's excited about this problem,
or maybe really concerned about this problem,
and that can influence the motivation of a team.
PABLO SUAREZ: And also, I don't know to what extent
we can keep cramming what students have
to do-- I know they already have to do way too much--
but to look at how that kind of field
is currently being communicated in the game universe
and outside of that.
So PowerPoints or documents, they're, across the board,
generally boring and unsuccessful--
even if they have the right content.
So for them to get a sense that if someone really
needs to create awareness on that topic,
and this is the best they have in what
exists in the real world, how about trying to give something
that makes people want to do more through their game?
PHILIP TAN: I think saying, this is how low the bar is, can we
get it slightly better, is less motivating than saying,
what's the best you can do?
RICHARD EBERHARDT: But it's also choosing something
that's going to get used.
Why go in a direction that's-- if this is what those people
are thinking about, if you give them something completely not
like that at all, they're probably not going to know what
to do with it.
PHILIP TAN: But I mean, I can imagine things
like evaluating current methods of teaching
about a certain concept, to be part of that research process
that we've been talking about-- it's understanding the problem.
RICHARD EBERHARDT: Cool.
All right, so I think it's about time to let you go.
But thanks so much for coming in.
PHILIP TAN: We thank you.
Huge gratitude.