1
00:00:20,150 --> 00:00:23,600
PETER SZOLOVITS: So
today's topic is workflow,

2
00:00:23,600 --> 00:00:27,010
and this is something that--

3
00:00:27,010 --> 00:00:30,820
a topic that I didn't
realize existed

4
00:00:30,820 --> 00:00:33,970
when I started
working in this area,

5
00:00:33,970 --> 00:00:37,270
but I've had my nose
ground and ground into it

6
00:00:37,270 --> 00:00:39,030
for many decades.

7
00:00:39,030 --> 00:00:42,100
And so finally, it has
become obvious to me

8
00:00:42,100 --> 00:00:44,000
that it's something
to pay attention to.

9
00:00:49,100 --> 00:00:51,100
So here's an
interesting question.

10
00:00:51,100 --> 00:00:54,790
Suppose that your goal
in the kind of work

11
00:00:54,790 --> 00:00:59,230
that we're doing in this class
is to improve medical care--

12
00:00:59,230 --> 00:01:02,570
not an unreasonable goal.

13
00:01:02,570 --> 00:01:04,640
So how do you do it?

14
00:01:04,640 --> 00:01:08,390
Well, we had an idea
back in the 1970s

15
00:01:08,390 --> 00:01:10,880
when I was getting
started on this, which

16
00:01:10,880 --> 00:01:15,620
was that we wanted to
understand what the world's best

17
00:01:15,620 --> 00:01:19,640
experts did and to
create decision support

18
00:01:19,640 --> 00:01:24,560
systems by encapsulating
their knowledge about how

19
00:01:24,560 --> 00:01:27,680
to do diagnosis, how to
do prognosis and treatment

20
00:01:27,680 --> 00:01:32,330
selection, in order to
improve the performance

21
00:01:32,330 --> 00:01:35,480
of every other doctor
who was not a world class

22
00:01:35,480 --> 00:01:40,430
expert by allowing the world
class expertise captured

23
00:01:40,430 --> 00:01:44,060
in a computer system to
help people figure out

24
00:01:44,060 --> 00:01:45,350
how to do better--

25
00:01:45,350 --> 00:01:50,000
so to make them more accurate
diagnosticians, more efficient

26
00:01:50,000 --> 00:01:52,850
therapists, et cetera.

27
00:01:52,850 --> 00:01:56,030
And the goal here was
really to bring up

28
00:01:56,030 --> 00:01:58,490
the average performance
of everybody

29
00:01:58,490 --> 00:02:00,300
in the health care system.

30
00:02:00,300 --> 00:02:04,580
So we used to say things
like, bring everybody

31
00:02:04,580 --> 00:02:08,389
practicing medicine closer
to the level of practice

32
00:02:08,389 --> 00:02:09,679
of the world class experts.

33
00:02:13,690 --> 00:02:20,380
Now, that turned out not
to be what was important.

34
00:02:20,380 --> 00:02:24,610
And so there was another idea
that came along a little bit

35
00:02:24,610 --> 00:02:28,810
later that said, well,
it's not really so much

36
00:02:28,810 --> 00:02:32,890
the average performance
of doctors that's bad.

37
00:02:32,890 --> 00:02:37,340
It's the subaverage performance
that's really terrible.

38
00:02:37,340 --> 00:02:42,580
And so if you're subaverage
performance leads

39
00:02:42,580 --> 00:02:48,440
to your patients dying, but your
above average performance only

40
00:02:48,440 --> 00:02:53,510
makes a moderate difference
in their outcomes,

41
00:02:53,510 --> 00:02:58,150
then it's clearly more important
to focus on the people who

42
00:02:58,150 --> 00:03:03,590
are the worst doctors and to
get them to act in a better way.

43
00:03:03,590 --> 00:03:06,470
And thus, was born the
idea of a protocol that

44
00:03:06,470 --> 00:03:11,120
says, let's treat similar
patients in similar ways.

45
00:03:11,120 --> 00:03:15,200
And the value of that is
to reduce the variance--

46
00:03:15,200 --> 00:03:19,220
so improve average
versus reduce variance.

47
00:03:19,220 --> 00:03:21,780
So which of these is better?

48
00:03:21,780 --> 00:03:23,930
Well it depends on
your loss function.

49
00:03:23,930 --> 00:03:28,550
So as I was suggesting, if your
loss function is a symmetric so

50
00:03:28,550 --> 00:03:33,050
that doing badly or
doing below average

51
00:03:33,050 --> 00:03:36,080
is much worse than
doing above average

52
00:03:36,080 --> 00:03:43,040
is much better, than this
protocol idea of reducing

53
00:03:43,040 --> 00:03:46,620
variance is really important.

54
00:03:46,620 --> 00:03:51,050
And this is pretty much what
the medical system has adopted.

55
00:03:51,050 --> 00:03:54,140
So I wanted to try to
help you visualize this.

56
00:03:54,140 --> 00:03:59,780
Suppose that on some
arbitrary scale of 0 to 8,

57
00:03:59,780 --> 00:04:04,670
we have an usual, normal
distribution, of on the left

58
00:04:04,670 --> 00:04:07,080
the base behaviors--

59
00:04:07,080 --> 00:04:11,750
so this is how people, on
average, normally behave--

60
00:04:11,750 --> 00:04:15,210
we assume that there's something
like a normal distribution.

61
00:04:15,210 --> 00:04:18,940
So here is a world class
expert whose performance

62
00:04:18,940 --> 00:04:23,090
is up at 6 or 7
and here's the dud

63
00:04:23,090 --> 00:04:27,330
of a doctor whose performance
is down between 0 and 1.

64
00:04:27,330 --> 00:04:32,460
And the average doctor
is just shy of 4.

65
00:04:32,460 --> 00:04:34,230
So here are two scenarios.

66
00:04:34,230 --> 00:04:38,210
Scenario one is that we
improve these guys performance

67
00:04:38,210 --> 00:04:39,710
by just a little bit.

68
00:04:39,710 --> 00:04:45,530
So we improve it by
0.1 performance points,

69
00:04:45,530 --> 00:04:49,190
I think is what I've
done in this model.

70
00:04:49,190 --> 00:04:51,500
versus another
approach, which is

71
00:04:51,500 --> 00:04:54,050
suppose we could cut
down the variance

72
00:04:54,050 --> 00:04:58,400
dramatically so that this same
normal distribution becomes

73
00:04:58,400 --> 00:04:59,480
narrower.

74
00:04:59,480 --> 00:05:02,780
Its average is still in
exactly the same place,

75
00:05:02,780 --> 00:05:06,800
but now there are
no distant outliers.

76
00:05:06,800 --> 00:05:10,280
So there aren't doctors
who perform a lot worse,

77
00:05:10,280 --> 00:05:15,220
and there aren't doctors who
perform a lot better either.

78
00:05:15,220 --> 00:05:17,530
Well, what happens in that case?

79
00:05:17,530 --> 00:05:20,050
Well, you have to look
at the cost function.

80
00:05:20,050 --> 00:05:24,070
So if you have a cost
function like this that says,

81
00:05:24,070 --> 00:05:29,380
that somebody's performing at
the 0 level has a cost of 1.

82
00:05:29,380 --> 00:05:32,450
Whereas somebody
performing at the 8 level

83
00:05:32,450 --> 00:05:38,170
has a cost of almost 0, and
it's exponentially declining

84
00:05:38,170 --> 00:05:42,220
like this, so that the
average performance has

85
00:05:42,220 --> 00:05:47,080
a much lower cost than the
average between the worst

86
00:05:47,080 --> 00:05:50,920
performance and the
best performance.

87
00:05:50,920 --> 00:05:54,900
So this suggests that,
if you could bunch people

88
00:05:54,900 --> 00:05:58,950
into this region of performance,
that your overall costs

89
00:05:58,950 --> 00:06:00,420
would go down.

90
00:06:00,420 --> 00:06:04,710
And, in fact-- this is a
purely hypothetical model that

91
00:06:04,710 --> 00:06:05,830
I've built--

92
00:06:05,830 --> 00:06:08,130
but if you do the
calculations, you

93
00:06:08,130 --> 00:06:11,610
discover that for the
base distribution,

94
00:06:11,610 --> 00:06:14,460
here is the
distribution of costs.

95
00:06:14,460 --> 00:06:17,790
For the slightly
improved distribution,

96
00:06:17,790 --> 00:06:23,410
you get a cost, which is
1,694 versus 781, again,

97
00:06:23,410 --> 00:06:25,270
in arbitrary units.

98
00:06:25,270 --> 00:06:28,140
But if you manage to
narrow the distribution,

99
00:06:28,140 --> 00:06:31,380
you can get the total cost
down to less than what

100
00:06:31,380 --> 00:06:34,530
you do by improving the average.

101
00:06:34,530 --> 00:06:38,550
Now, this is not a proof,
but this is the right idea.

102
00:06:38,550 --> 00:06:41,160
The proof is
probably in the fact

103
00:06:41,160 --> 00:06:44,220
that medical systems
have adopted this,

104
00:06:44,220 --> 00:06:48,690
and have decided that getting
all doctors to behave more

105
00:06:48,690 --> 00:06:54,150
like the average doctor is the
best practical way of improving

106
00:06:54,150 --> 00:06:54,960
medical care.

107
00:06:59,780 --> 00:07:03,230
Well, how do we narrow the
performance distribution?

108
00:07:03,230 --> 00:07:08,680
So one way is by having
guidelines and protocols where

109
00:07:08,680 --> 00:07:12,880
you have some learned body who
prescribes appropriate methods

110
00:07:12,880 --> 00:07:15,680
to diagnose and treat patients.

111
00:07:15,680 --> 00:07:19,780
So what happens is, for example,
the article here from November

112
00:07:19,780 --> 00:07:24,610
of 2018, a report of the
American College of Cardiology,

113
00:07:24,610 --> 00:07:28,960
the American Heart Association
Task Force on Clinical Practice

114
00:07:28,960 --> 00:07:32,230
Guidelines, and this
has been adopted

115
00:07:32,230 --> 00:07:36,850
by this cornucopia of
three and four letter

116
00:07:36,850 --> 00:07:39,820
abbreviated organizations.

117
00:07:39,820 --> 00:07:42,450
And it's a guideline
on the management

118
00:07:42,450 --> 00:07:44,240
of blood cholesterol.

119
00:07:44,240 --> 00:07:49,150
So as you know, having high
cholesterol is dangerous.

120
00:07:49,150 --> 00:07:52,480
It can lead to heart
attacks and strokes,

121
00:07:52,480 --> 00:07:55,420
and so there is a
consensus that it would be

122
00:07:55,420 --> 00:07:58,160
good to lower that in people.

123
00:07:58,160 --> 00:08:03,100
So these guys went about
this by gathering together

124
00:08:03,100 --> 00:08:05,600
a bunch of world
experts and saying,

125
00:08:05,600 --> 00:08:07,170
well, how do we do this?

126
00:08:07,170 --> 00:08:11,620
What do we promulgate
as the appropriate way

127
00:08:11,620 --> 00:08:14,780
to care for patients
with this condition?

128
00:08:14,780 --> 00:08:16,280
And the first thing
they did is they

129
00:08:16,280 --> 00:08:22,820
came up with a
color coded notion

130
00:08:22,820 --> 00:08:27,440
of how strong the recommendation
a certain recommendation

131
00:08:27,440 --> 00:08:31,060
should be And
another color coded

132
00:08:31,060 --> 00:08:39,659
or shaded level of certainty
in that recommendation.

133
00:08:39,659 --> 00:08:43,940
So, for example, if you say
something is in class 1,

134
00:08:43,940 --> 00:08:47,150
so it's a strong
recommendation, then

135
00:08:47,150 --> 00:08:51,260
you use words like is
recommended, or is indicated,

136
00:08:51,260 --> 00:08:56,150
useful, effective, beneficial,
should be performed, et cetera.

137
00:08:56,150 --> 00:09:00,470
If it's in class 2,
where the benefit is

138
00:09:00,470 --> 00:09:02,900
much greater than
the risk, then you

139
00:09:02,900 --> 00:09:07,340
say things like it's reasonable,
it can be useful, et cetera.

140
00:09:07,340 --> 00:09:12,380
If the benefit is maybe equal
to or a little bit better

141
00:09:12,380 --> 00:09:15,800
than the risk, you say
waffle words, like might be

142
00:09:15,800 --> 00:09:18,950
reasonable, may be considered.

143
00:09:18,950 --> 00:09:22,500
If there is no benefit,
in other words,

144
00:09:22,500 --> 00:09:25,340
if it roughly equals
the risk, then you say,

145
00:09:25,340 --> 00:09:27,080
it's not recommended.

146
00:09:27,080 --> 00:09:29,630
And if the risk is
greater than the benefit,

147
00:09:29,630 --> 00:09:32,870
then you say things like
it's potentially harmful,

148
00:09:32,870 --> 00:09:34,820
causes harm, et cetera.

149
00:09:34,820 --> 00:09:37,400
So if you were giving
a recommendation

150
00:09:37,400 --> 00:09:42,110
on whether to spray
disinfectant down your lungs,

151
00:09:42,110 --> 00:09:47,390
you might put that in red and
say, this is not recommended.

152
00:09:47,390 --> 00:09:51,500
And then here,
this shading coding

153
00:09:51,500 --> 00:09:54,140
is basically how
good is the evidence

154
00:09:54,140 --> 00:09:56,060
for this recommendation.

155
00:09:56,060 --> 00:10:02,150
So the best evidence,
the level A,

156
00:10:02,150 --> 00:10:06,260
is high-quality evidence from
multiple randomized controlled

157
00:10:06,260 --> 00:10:09,740
clinical trials,
or a meta-analyses

158
00:10:09,740 --> 00:10:18,560
of a high-quality RCTs,
or RCTs corroborated

159
00:10:18,560 --> 00:10:21,020
by high-quality
registry studies.

160
00:10:21,020 --> 00:10:24,320
And then we go down
to level C, which

161
00:10:24,320 --> 00:10:29,000
is consensus of expert opinion
based on clinical experience,

162
00:10:29,000 --> 00:10:33,330
but without any sort
of formal analysis.

163
00:10:33,330 --> 00:10:36,350
So if you look at this
particular document

164
00:10:36,350 --> 00:10:40,280
on cholesterol it
says, well, here

165
00:10:40,280 --> 00:10:43,250
are the recommendations
on the measurement

166
00:10:43,250 --> 00:10:49,420
of LDL and non-HDL cholesterol.

167
00:10:49,420 --> 00:10:53,630
And they say here,
the confidence

168
00:10:53,630 --> 00:11:01,940
and the recommendation
is one, and it's based

169
00:11:01,940 --> 00:11:05,690
on B and our level of evidence.

170
00:11:05,690 --> 00:11:09,260
And it says, in adults who
are 20 years or older and not

171
00:11:09,260 --> 00:11:12,980
on lipid-lowering
therapy, measurements

172
00:11:12,980 --> 00:11:16,040
of either a fasting or a
non-fasting blood-- dot,

173
00:11:16,040 --> 00:11:17,000
dot, dot.

174
00:11:17,000 --> 00:11:20,120
So you could read this
in the notes later.

175
00:11:20,120 --> 00:11:25,820
But notice that there are
high force recommendations.

176
00:11:25,820 --> 00:11:28,490
There are lower force
recommendations,

177
00:11:28,490 --> 00:11:31,130
and each recommendation
is also shading

178
00:11:31,130 --> 00:11:34,040
coded to tell you what
the strength of evidence

179
00:11:34,040 --> 00:11:36,870
is for this kind
of recommendation.

180
00:11:36,870 --> 00:11:39,050
Here's just another example.

181
00:11:39,050 --> 00:11:43,700
This is secondary
atherosclerotic cardiovascular

182
00:11:43,700 --> 00:11:45,810
disease prevention.

183
00:11:45,810 --> 00:11:50,310
So this is for somebody
who's already ill,

184
00:11:50,310 --> 00:11:52,770
and it's a bunch
of recommendations.

185
00:11:52,770 --> 00:11:57,200
If you're over 75 years
of age, or younger

186
00:11:57,200 --> 00:12:03,020
with a clinical case of
coronary vascular disease,

187
00:12:03,020 --> 00:12:07,220
then high intensity
statin therapy

188
00:12:07,220 --> 00:12:10,820
should be initiated or
continued with the aim

189
00:12:10,820 --> 00:12:16,760
of achieving a 50% or greater
reduction in LDLC and et

190
00:12:16,760 --> 00:12:17,370
cetera.

191
00:12:17,370 --> 00:12:20,870
So again, a whole bunch of
different recommendations.

192
00:12:20,870 --> 00:12:24,460
Once again, the strength of the
recommendation-- by the way,

193
00:12:24,460 --> 00:12:28,100
this is just the first
page of a couple of pages--

194
00:12:28,100 --> 00:12:31,230
and the quality of
evidence for it.

195
00:12:31,230 --> 00:12:34,940
So this is very
much the way that

196
00:12:34,940 --> 00:12:37,820
learned societies are
now trying to influence

197
00:12:37,820 --> 00:12:42,380
the practice of medicine in
order to reduce the variance

198
00:12:42,380 --> 00:12:47,640
and get everybody to
behave in a normal way.

199
00:12:47,640 --> 00:12:52,070
You've probably seen articles
about Atul Gawande, who's

200
00:12:52,070 --> 00:12:56,600
a surgeon here in Boston, and
he's gotten publicly famous

201
00:12:56,600 --> 00:12:59,360
for advocating checklists.

202
00:12:59,360 --> 00:13:01,100
And he says, for
example, if you're

203
00:13:01,100 --> 00:13:04,820
a surgeon, you should act
like an airline pilot,

204
00:13:04,820 --> 00:13:07,490
that before you take
off in the airplane,

205
00:13:07,490 --> 00:13:09,950
you go through a
sanity checklist

206
00:13:09,950 --> 00:13:13,880
to make sure that all the
systems are working properly,

207
00:13:13,880 --> 00:13:16,730
that all the switches
are set correctly,

208
00:13:16,730 --> 00:13:18,950
which in a surgical
setting would be things

209
00:13:18,950 --> 00:13:23,720
like you have all the right
necessary equipment available,

210
00:13:23,720 --> 00:13:27,320
that you know what to do in
various potential emergencies,

211
00:13:27,320 --> 00:13:27,910
et cetera.

212
00:13:33,010 --> 00:13:37,770
So here are their take-home
messages, which makes sense.

213
00:13:37,770 --> 00:13:41,520
Here, I've abstracted
these from the paper

214
00:13:41,520 --> 00:13:44,010
that has all of these details.

215
00:13:44,010 --> 00:13:49,200
So number one, you
go, well, duh--

216
00:13:49,200 --> 00:13:53,400
in all individuals, emphasize
a heart healthy lifestyle

217
00:13:53,400 --> 00:13:54,810
across the life course.

218
00:13:54,810 --> 00:13:59,830
That seems not
terribly controversial,

219
00:13:59,830 --> 00:14:03,070
and in people who
are already diseased,

220
00:14:03,070 --> 00:14:08,235
reduce low-density lipoprotein
with high-intensity therapy

221
00:14:08,235 --> 00:14:10,290
by statins.

222
00:14:10,290 --> 00:14:16,590
And in very high risk
ASCVD, use a threshold

223
00:14:16,590 --> 00:14:19,710
of 70 milligrams per
deciliter, et cetera.

224
00:14:19,710 --> 00:14:22,890
So these are the
summary recommendations.

225
00:14:22,890 --> 00:14:27,480
And the hope is that doctors
reading these sorts of articles

226
00:14:27,480 --> 00:14:32,040
come away from them convinced
and will remember that they're

227
00:14:32,040 --> 00:14:34,530
supposed to act this way
when they're interacting

228
00:14:34,530 --> 00:14:36,910
with their patients.

229
00:14:36,910 --> 00:14:41,430
This is a flow chart,
again, abstracted

230
00:14:41,430 --> 00:14:47,340
from that paper by them
which says, everybody,

231
00:14:47,340 --> 00:14:51,210
you should emphasize
a healthy lifestyle.

232
00:14:51,210 --> 00:14:54,420
And then depending on
your age, depending

233
00:14:54,420 --> 00:15:01,380
on what your estimate
of lifetime risk is,

234
00:15:01,380 --> 00:15:04,230
you wind up in
different categories.

235
00:15:04,230 --> 00:15:09,000
And these different categories
have different recommendations

236
00:15:09,000 --> 00:15:12,530
for what you ought to
do with your patients.

237
00:15:12,530 --> 00:15:14,870
This is for
secondary prevention.

238
00:15:14,870 --> 00:15:19,170
So it's a similar flow chart for
people who are already diseased

239
00:15:19,170 --> 00:15:20,380
and not just at risk.

240
00:15:23,980 --> 00:15:30,050
And then for people at very
high risk for future events,

241
00:15:30,050 --> 00:15:33,910
which is defined
by these histories

242
00:15:33,910 --> 00:15:36,820
and these high-risk
conditions, these

243
00:15:36,820 --> 00:15:41,170
are the people who fall
into that second flow chart

244
00:15:41,170 --> 00:15:43,570
and should be treated that way.

245
00:15:43,570 --> 00:15:49,290
Now, by the way, I
didn't make a poll,

246
00:15:49,290 --> 00:15:50,850
so I'll give you the answer.

247
00:15:50,850 --> 00:15:53,870
But it's interesting to ask.

248
00:15:53,870 --> 00:15:56,960
So when papers like
this get published,

249
00:15:56,960 --> 00:16:00,870
how well do doctors
actually adhere to these?

250
00:16:00,870 --> 00:16:03,680
And the answer turns
out to be not very well,

251
00:16:03,680 --> 00:16:06,710
and it takes many, many
years before these kinds

252
00:16:06,710 --> 00:16:10,340
of recommendations are
taken up by the majority

253
00:16:10,340 --> 00:16:17,570
of the community, so even
very, very uncontroversial

254
00:16:17,570 --> 00:16:19,100
recommendations.

255
00:16:19,100 --> 00:16:22,280
For example, I think
20 years ago there

256
00:16:22,280 --> 00:16:26,090
was a recommendation that said
that anybody who's had a heart

257
00:16:26,090 --> 00:16:30,830
attack should be treated, even
if they're now asymptomatic,

258
00:16:30,830 --> 00:16:32,100
with beta blockers.

259
00:16:32,100 --> 00:16:35,750
Because in various
trials, they showed

260
00:16:35,750 --> 00:16:40,850
that there was a 35% reduction
in repeat heart attacks

261
00:16:40,850 --> 00:16:44,060
as a result of this treatment.

262
00:16:44,060 --> 00:16:51,140
It took, I think, over a dozen
years before most doctors

263
00:16:51,140 --> 00:16:54,440
were aware of this
and started making

264
00:16:54,440 --> 00:16:57,005
that kind of recommendation
to their patients.

265
00:17:02,320 --> 00:17:05,060
There's something
called the AHRQ,

266
00:17:05,060 --> 00:17:08,240
the Agency for Health
Research and Quality.

267
00:17:08,240 --> 00:17:11,630
And until the current
administration,

268
00:17:11,630 --> 00:17:15,260
they ran a national
guideline clearinghouse

269
00:17:15,260 --> 00:17:21,800
that contained myriad of
these guidelines, published

270
00:17:21,800 --> 00:17:25,069
by different authorities,
and was available for people

271
00:17:25,069 --> 00:17:29,000
to download and use.

272
00:17:29,000 --> 00:17:31,700
There's been an attempt
by Guideline Central

273
00:17:31,700 --> 00:17:36,740
to take over some of these roles
since the government shutdown

274
00:17:36,740 --> 00:17:42,380
the government run one, and
they have about 2,000 guidelines

275
00:17:42,380 --> 00:17:45,950
that are posted on their site.

276
00:17:45,950 --> 00:17:48,360
And these are some
of the examples.

277
00:17:48,360 --> 00:17:51,650
So risk reduction
of prostate cancer

278
00:17:51,650 --> 00:17:54,320
with drugs or
nutritional supplements,

279
00:17:54,320 --> 00:17:57,980
stem cell transplantation
in multiple myeloma,

280
00:17:57,980 --> 00:18:02,030
stem cell transplantation
in myelodysplastic syndromes

281
00:18:02,030 --> 00:18:06,770
and acute myeloid
leukemia, et cetera.

282
00:18:06,770 --> 00:18:10,340
And then they also publish
a bunch of risk calculators

283
00:18:10,340 --> 00:18:11,660
that say--

284
00:18:11,660 --> 00:18:15,950
I don't know what the 4T
score is for heparin-induced

285
00:18:15,950 --> 00:18:17,720
thrombocytopenia--

286
00:18:17,720 --> 00:18:20,280
but there are tons
of these as well.

287
00:18:20,280 --> 00:18:22,850
So there's a clearinghouse
of these things.

288
00:18:22,850 --> 00:18:25,970
And you, as a practicing
doctor, can go to these.

289
00:18:25,970 --> 00:18:28,610
Or your hospital can
decide that they're

290
00:18:28,610 --> 00:18:33,260
going to provide these
guidelines to their doctors,

291
00:18:33,260 --> 00:18:35,930
and either encourage,
or in some cases,

292
00:18:35,930 --> 00:18:39,320
coerce them to use the
guidelines in order

293
00:18:39,320 --> 00:18:43,220
to determine what
their activity is.

294
00:18:43,220 --> 00:18:47,150
Now, notice that this is a
very top-down kind of activity.

295
00:18:47,150 --> 00:18:51,650
So it's typically done by
these learned societies that

296
00:18:51,650 --> 00:18:54,560
bring together
experts to cogitate

297
00:18:54,560 --> 00:18:57,140
on what the right thing
to do is, and then they

298
00:18:57,140 --> 00:18:59,870
tell the rest of the
world how to do it.

299
00:18:59,870 --> 00:19:03,690
But there's also a kind
of bottom-up activity.

300
00:19:03,690 --> 00:19:09,250
So there is something
called a "care plan."

301
00:19:09,250 --> 00:19:14,100
Now, a care plan is
really a nursing term.

302
00:19:14,100 --> 00:19:16,850
So if you hang
out at a hospital,

303
00:19:16,850 --> 00:19:21,230
the thing you discover is that
the doctors are evanescent.

304
00:19:21,230 --> 00:19:23,430
They appear and disappear.

305
00:19:23,430 --> 00:19:27,800
They're like
elementary particles,

306
00:19:27,800 --> 00:19:30,110
and they're not
around all the time.

307
00:19:30,110 --> 00:19:32,900
The people who are
actually taking care of you

308
00:19:32,900 --> 00:19:34,610
are the nurses.

309
00:19:34,610 --> 00:19:39,440
And so the nurses have
developed a set of methodologies

310
00:19:39,440 --> 00:19:42,900
for how to ensure that
they take good care of you,

311
00:19:42,900 --> 00:19:44,780
and one of them
is the development

312
00:19:44,780 --> 00:19:46,580
of these care plans.

313
00:19:46,580 --> 00:19:48,740
And then what
clinical pathways are

314
00:19:48,740 --> 00:19:51,680
is an attempt to take the
care plans that nurses

315
00:19:51,680 --> 00:19:54,770
use in taking care
of individuals

316
00:19:54,770 --> 00:19:57,350
and to generalize from
those and say, well,

317
00:19:57,350 --> 00:20:02,150
what are the typical ways in
which we take care of patients

318
00:20:02,150 --> 00:20:04,070
in a particular cohort?

319
00:20:04,070 --> 00:20:06,180
So I'm going to talk a
little bit about that,

320
00:20:06,180 --> 00:20:09,650
and one of the papers I gave
you as an optional reading

321
00:20:09,650 --> 00:20:12,650
for today is about
cow paths, which

322
00:20:12,650 --> 00:20:19,310
are these attempts to build
generalizations of care plans.

323
00:20:19,310 --> 00:20:26,460
So this is a care plan from the
Michigan Center for Nursing,

324
00:20:26,460 --> 00:20:28,730
which is an educational
organization that

325
00:20:28,730 --> 00:20:32,660
tries to help nurses figure
out how to be good nurses.

326
00:20:32,660 --> 00:20:35,300
I was very amused when
I was looking for this.

327
00:20:35,300 --> 00:20:39,590
I ran across a video, which is
some experienced nurse talking

328
00:20:39,590 --> 00:20:42,650
about how you build
these care plans.

329
00:20:42,650 --> 00:20:46,520
And she sort of says, well,
when you're in nursing school,

330
00:20:46,520 --> 00:20:50,630
you learn how to build these
very elaborate carefully

331
00:20:50,630 --> 00:20:52,490
constructed care plans.

332
00:20:52,490 --> 00:20:54,680
When you're actually
practicing as a nurse,

333
00:20:54,680 --> 00:20:57,210
you'll never have
time to do this.

334
00:20:57,210 --> 00:21:00,510
And so you're going to do a
rough approximation to this.

335
00:21:00,510 --> 00:21:01,830
And don't worry about it.

336
00:21:01,830 --> 00:21:05,450
But for now, satisfy your
professors by doing these

337
00:21:05,450 --> 00:21:07,880
exercises correctly.

338
00:21:07,880 --> 00:21:10,500
So take a look at this.

339
00:21:10,500 --> 00:21:12,260
So there are a bunch of columns.

340
00:21:12,260 --> 00:21:14,430
The leftmost one
says assessment.

341
00:21:14,430 --> 00:21:17,330
So this is objective,
subjective,

342
00:21:17,330 --> 00:21:19,620
and medical diagnostic data.

343
00:21:19,620 --> 00:21:23,720
So the objective data is this
patient has gangrene-infected

344
00:21:23,720 --> 00:21:24,740
left foot--

345
00:21:24,740 --> 00:21:28,680
not a good thing, an open
wound, et cetera, et cetera.

346
00:21:28,680 --> 00:21:32,150
Subjective data, the patient
said the pain is worse

347
00:21:32,150 --> 00:21:34,490
when walking and turning.

348
00:21:34,490 --> 00:21:37,190
She dreads physical
therapy, and she

349
00:21:37,190 --> 00:21:41,220
wishes she did not have
to be in this situation--

350
00:21:41,220 --> 00:21:42,990
surprise.

351
00:21:42,990 --> 00:21:44,850
But that's definitely
subjective.

352
00:21:44,850 --> 00:21:48,360
You can't see external
evidence of that.

353
00:21:48,360 --> 00:21:51,510
The nursing diagnosis
is that this patient

354
00:21:51,510 --> 00:21:56,190
has impaired tissue
integrity in reference

355
00:21:56,190 --> 00:21:59,890
to the wound and the
presence of an infection.

356
00:21:59,890 --> 00:22:06,900
Now, that diagnosis actually
comes with a kind of guideline

357
00:22:06,900 --> 00:22:09,120
about how to make
that diagnosis.

358
00:22:09,120 --> 00:22:12,270
In other words, in order
to be able to put that down

359
00:22:12,270 --> 00:22:14,910
on the care plan,
she has to make sure

360
00:22:14,910 --> 00:22:16,920
that characteristics
of the patient

361
00:22:16,920 --> 00:22:21,060
satisfy certain criteria
which are the definition

362
00:22:21,060 --> 00:22:24,060
of that diagnosis.

363
00:22:24,060 --> 00:22:28,290
The patient outcomes--
so this is the goals

364
00:22:28,290 --> 00:22:30,660
that the nurse is
trying to achieve.

365
00:22:30,660 --> 00:22:33,220
And notice, there
are five goals here.

366
00:22:33,220 --> 00:22:36,750
One is that the patient will
report any altered sensation

367
00:22:36,750 --> 00:22:43,470
of pain at the tissue impairment
between January 23 and 24.

368
00:22:43,470 --> 00:22:46,770
So this is a very specific goal.

369
00:22:46,770 --> 00:22:49,410
It says, the
patient will tell me

370
00:22:49,410 --> 00:22:53,380
that they feel
better, that there's

371
00:22:53,380 --> 00:22:59,950
a change in their feeling
in their infected left foot.

372
00:22:59,950 --> 00:23:07,090
They will understand the plan to
heal tissue and prevent injury.

373
00:23:07,090 --> 00:23:09,870
So there's a patient
education component.

374
00:23:09,870 --> 00:23:13,590
They will describe measures to
protect and heal the tissue,

375
00:23:13,590 --> 00:23:16,230
including wound care by 124.

376
00:23:16,230 --> 00:23:19,770
So notice, this is the
patient describing to you

377
00:23:19,770 --> 00:23:22,440
what you are planning to do
for them, in other words,

378
00:23:22,440 --> 00:23:26,130
demonstrating an understanding
of what the plan is

379
00:23:26,130 --> 00:23:29,230
and what's likely
to happen with them.

380
00:23:29,230 --> 00:23:33,060
Experience a wound decrease
that decreases in size

381
00:23:33,060 --> 00:23:36,000
and has increased
granulation tissue,

382
00:23:36,000 --> 00:23:38,820
and achieve functional
pain goal of 0

383
00:23:38,820 --> 00:23:41,790
by 124 per the
patient's verbalization.

384
00:23:41,790 --> 00:23:44,070
So when they come
in and they ask you

385
00:23:44,070 --> 00:23:50,790
on that pain scale, are
you at a 0, or a 10,

386
00:23:50,790 --> 00:23:53,400
or somewhere in
between, the goal

387
00:23:53,400 --> 00:23:55,770
is that the patient
will say, I'm at a 0,

388
00:23:55,770 --> 00:23:58,380
in other words, no pain.

389
00:23:58,380 --> 00:24:00,210
Now, what are the interventions?

390
00:24:00,210 --> 00:24:02,580
Well, these are the
things that the nurse

391
00:24:02,580 --> 00:24:06,930
plans to do in order to
try to achieve those goals.

392
00:24:06,930 --> 00:24:09,720
And then the rationale
is an explanation

393
00:24:09,720 --> 00:24:14,640
of why it's reasonable to expect
those interventions to achieve

394
00:24:14,640 --> 00:24:16,110
those goals.

395
00:24:16,110 --> 00:24:18,480
And the evaluation
of outcomes says,

396
00:24:18,480 --> 00:24:25,860
what criteria or what are
the actual outcomes for what

397
00:24:25,860 --> 00:24:27,670
we're trying to achieve?

398
00:24:27,670 --> 00:24:30,480
So that gets filled
in later, obviously,

399
00:24:30,480 --> 00:24:33,300
then when the plan is made.

400
00:24:33,300 --> 00:24:35,790
So if you look at a
website like this,

401
00:24:35,790 --> 00:24:41,340
there are templated care plans
for many, many conditions.

402
00:24:41,340 --> 00:24:44,490
You can see that I'm
only up to C in an A to Z

403
00:24:44,490 --> 00:24:48,960
listing from this one website,
and there are plenty of others.

404
00:24:48,960 --> 00:24:53,520
But there is an admission care
plan, adult failure to thrive,

405
00:24:53,520 --> 00:24:59,790
alcohol withdrawal, runny
nose, altered cardiac output,

406
00:24:59,790 --> 00:25:01,260
amputation.

407
00:25:01,260 --> 00:25:03,900
I don't know what
an anasarca is--

408
00:25:03,900 --> 00:25:06,910
anemia, angina, anticoagulant
care, et cetera.

409
00:25:06,910 --> 00:25:09,750
So there are tons of
different conditions

410
00:25:09,750 --> 00:25:11,670
that different
patients fall into,

411
00:25:11,670 --> 00:25:17,910
and this is a way of trying to
list the template care plans.

412
00:25:17,910 --> 00:25:20,760
Now, this paper is
kind of interesting,

413
00:25:20,760 --> 00:25:24,390
by Yiye Zhang and colleagues.

414
00:25:24,390 --> 00:25:30,100
And what they did
is they said, well,

415
00:25:30,100 --> 00:25:33,090
let's take all these
care plans and let's

416
00:25:33,090 --> 00:25:36,660
try to build a machine learning
system that learns what

417
00:25:36,660 --> 00:25:40,230
are the typical patterns that
are embedded in those care

418
00:25:40,230 --> 00:25:41,490
plans.

419
00:25:41,490 --> 00:25:43,410
But they didn't
start with the plans.

420
00:25:43,410 --> 00:25:45,080
This is retrospective analysis.

421
00:25:45,080 --> 00:25:48,480
So what they started with
is the actual records

422
00:25:48,480 --> 00:25:51,450
of what was done
to each patient.

423
00:25:51,450 --> 00:25:54,480
And so the idea is that
you get treatment data

424
00:25:54,480 --> 00:25:57,240
from the electronic
health record.

425
00:25:57,240 --> 00:26:01,440
Then you identify patient
subgroups from that data,

426
00:26:01,440 --> 00:26:04,680
and then you mine for
common treatment patterns.

427
00:26:04,680 --> 00:26:07,770
And you have medical
experts evaluate these,

428
00:26:07,770 --> 00:26:10,680
and these then become
clinical pathways,

429
00:26:10,680 --> 00:26:13,730
which are this generalization
of the care plans

430
00:26:13,730 --> 00:26:17,680
to particular
subpopulations of patients.

431
00:26:17,680 --> 00:26:22,080
So the idea is that they
define a bunch of abstractions.

432
00:26:22,080 --> 00:26:25,080
So they say, look,
an event is a visit.

433
00:26:25,080 --> 00:26:27,510
So, for example,
for an outpatient,

434
00:26:27,510 --> 00:26:31,080
anything that happens to you
during one visit to a doctor

435
00:26:31,080 --> 00:26:33,840
or to a hospital.

436
00:26:33,840 --> 00:26:37,320
So it's a set of procedures,
a set of medications,

437
00:26:37,320 --> 00:26:38,440
a set of diagnoses.

438
00:26:41,910 --> 00:26:43,740
And by the way,
they were focusing

439
00:26:43,740 --> 00:26:50,580
on people with kidney disease
as the target population

440
00:26:50,580 --> 00:26:52,530
that they were looking at.

441
00:26:52,530 --> 00:26:56,970
So then they say,
OK, individual events

442
00:26:56,970 --> 00:27:01,230
are going to be abstracted into
these supernodes, which capture

443
00:27:01,230 --> 00:27:05,400
a unique combination of
associations of events

444
00:27:05,400 --> 00:27:08,070
associated with some visit.

445
00:27:08,070 --> 00:27:12,660
So you might worry that this
is going to be combinatorial,

446
00:27:12,660 --> 00:27:16,140
because there are many possible
combinations of things.

447
00:27:16,140 --> 00:27:18,330
And that is, in fact,
a bit of a problem,

448
00:27:18,330 --> 00:27:20,740
I think, in their analysis.

449
00:27:20,740 --> 00:27:23,070
So now, you have
these supernodes,

450
00:27:23,070 --> 00:27:26,580
and then each patient
has a visit sequence,

451
00:27:26,580 --> 00:27:29,970
which is a time-ordered
list of the supernodes.

452
00:27:29,970 --> 00:27:32,520
So every time you
go see your doctor,

453
00:27:32,520 --> 00:27:35,010
you have one new supernode.

454
00:27:35,010 --> 00:27:37,860
And so you have a
time series of these.

455
00:27:37,860 --> 00:27:39,790
And then they do
the following thing.

456
00:27:39,790 --> 00:27:44,580
They say, gee, when we talk
to our doctors and nurses,

457
00:27:44,580 --> 00:27:47,550
they tell us that
they care mostly

458
00:27:47,550 --> 00:27:52,080
about what happened at the last
visit that the patient had.

459
00:27:52,080 --> 00:27:54,660
But they also care
a little bit less,

460
00:27:54,660 --> 00:27:57,750
but they still care about what
happened at the visit previous

461
00:27:57,750 --> 00:28:03,120
to that, but not so much about
history going further back.

462
00:28:03,120 --> 00:28:06,210
And so they say, well,
in a Markov chain,

463
00:28:06,210 --> 00:28:11,520
we only have things depend on
the last node in the Markov

464
00:28:11,520 --> 00:28:12,610
chain.

465
00:28:12,610 --> 00:28:15,060
So let's change
the model here so

466
00:28:15,060 --> 00:28:17,430
that we will combine
pairs of visits

467
00:28:17,430 --> 00:28:22,110
into nodes so that each
node in the Markov chain

468
00:28:22,110 --> 00:28:27,300
will represent the last two
visits that the patient had.

469
00:28:27,300 --> 00:28:30,450
So this could, again, cause
some combinatorial problems.

470
00:28:30,450 --> 00:28:34,020
But here's the image
that they come up with.

471
00:28:34,020 --> 00:28:36,150
So there are individual items.

472
00:28:36,150 --> 00:28:39,240
Is it a hospital visit,
an office visit, a visit

473
00:28:39,240 --> 00:28:41,860
for the purpose of education?

474
00:28:41,860 --> 00:28:45,530
Are you in chronic kidney
disease stage four?

475
00:28:48,150 --> 00:28:50,730
Was an ultrasound done?

476
00:28:50,730 --> 00:28:54,630
Were you given ACE inhibitors?

477
00:28:54,630 --> 00:28:56,800
Were you given
diuretics, et cetera?

478
00:28:56,800 --> 00:28:59,340
So these are all the
data that we mentioned.

479
00:28:59,340 --> 00:29:02,160
They treat that as a bag.

480
00:29:02,160 --> 00:29:04,860
And then they say,
OK, we're going

481
00:29:04,860 --> 00:29:10,650
to identify all the bags that
have the same exact content.

482
00:29:10,650 --> 00:29:15,400
An asterisk, they didn't
look, for example,

483
00:29:15,400 --> 00:29:17,880
at the dose of medication
that you were given,

484
00:29:17,880 --> 00:29:20,250
only which medication it was.

485
00:29:20,250 --> 00:29:23,080
So there are some
collapsing that way.

486
00:29:23,080 --> 00:29:26,430
Then the supernodes
are these combinations

487
00:29:26,430 --> 00:29:31,230
where we say, OK, you
had a particular purpose,

488
00:29:31,230 --> 00:29:34,470
a particular diagnosis,
a particular set

489
00:29:34,470 --> 00:29:38,320
of interventions, a
particular set of procedures.

490
00:29:38,320 --> 00:29:41,940
And again, we list all
possible combinations of those,

491
00:29:41,940 --> 00:29:46,120
and then that sequence
represents your sequence.

492
00:29:46,120 --> 00:29:48,990
These are aggregated
into supernodes.

493
00:29:48,990 --> 00:29:51,540
That represents
your visit sequence,

494
00:29:51,540 --> 00:29:54,540
and then these super
pairs are this hack

495
00:29:54,540 --> 00:29:58,800
to let you look two steps
back in the Markov chain.

496
00:29:58,800 --> 00:30:03,270
And so they wind up with
about 3,500 different

497
00:30:03,270 --> 00:30:05,220
of these super pair nodes.

498
00:30:05,220 --> 00:30:07,860
So it is combinatorial,
but it's not terribly

499
00:30:07,860 --> 00:30:09,660
combinatorial in their data.

500
00:30:14,350 --> 00:30:18,570
They then compute the
maximum of the length

501
00:30:18,570 --> 00:30:22,740
of common subsequences between
each pair of visit sequences.

502
00:30:22,740 --> 00:30:25,350
So they're going to
cluster these sequences.

503
00:30:25,350 --> 00:30:28,620
They define a
distance function that

504
00:30:28,620 --> 00:30:34,140
says that the more they
share a common sequence,

505
00:30:34,140 --> 00:30:36,810
the less distant they
are from each other.

506
00:30:36,810 --> 00:30:38,610
And the particular
distance function

507
00:30:38,610 --> 00:30:43,140
they used is the length of
each sequence minus twice

508
00:30:43,140 --> 00:30:46,560
the length of the common
subsequence, the longest

509
00:30:46,560 --> 00:30:50,190
common subsequence, which
seems pretty reasonable.

510
00:30:50,190 --> 00:30:54,330
And then hierarchical clustering
into distinct subgroups,

511
00:30:54,330 --> 00:30:58,740
they came up with 31 groups
for this group of patients,

512
00:30:58,740 --> 00:31:01,350
and here they are.

513
00:31:01,350 --> 00:31:06,360
And what you see is
that some of them

514
00:31:06,360 --> 00:31:08,830
don't differ a whole
lot from each other.

515
00:31:08,830 --> 00:31:12,270
So, for example,
these two differ only

516
00:31:12,270 --> 00:31:17,220
in that the patient got some
medication and diuretics

517
00:31:17,220 --> 00:31:21,190
in one case and just that
medication in the other case.

518
00:31:21,190 --> 00:31:24,360
So these are-- it is a
hierarchical cluster,

519
00:31:24,360 --> 00:31:28,050
and the things lower
down in the clustering

520
00:31:28,050 --> 00:31:30,600
are probably fairly
close to each other.

521
00:31:30,600 --> 00:31:32,490
Nevertheless, what
they're able to do,

522
00:31:32,490 --> 00:31:35,910
then, is to estimate
a transition matrix

523
00:31:35,910 --> 00:31:42,180
among these supernode
pair states,

524
00:31:42,180 --> 00:31:45,180
and they can look at
different trajectories

525
00:31:45,180 --> 00:31:49,510
depending on the degree
of support for the data.

526
00:31:49,510 --> 00:31:52,020
So you can set
different thresholds

527
00:31:52,020 --> 00:31:57,390
on how many cases have to be
in a particular state in order

528
00:31:57,390 --> 00:32:02,910
for you to take transitions to
or from that state seriously.

529
00:32:02,910 --> 00:32:05,310
One of the critiques I
would make of the study

530
00:32:05,310 --> 00:32:10,410
is that they had way too little
data, and so many of the groups

531
00:32:10,410 --> 00:32:12,420
that they came up
with had relatively

532
00:32:12,420 --> 00:32:18,810
small numbers of patients in
them, which is unfortunate.

533
00:32:18,810 --> 00:32:21,990
Now, once you have these
transition matrices,

534
00:32:21,990 --> 00:32:26,280
then you can say, OK,
for cluster 29, which

535
00:32:26,280 --> 00:32:34,580
was this cluster, so
there were a grand total

536
00:32:34,580 --> 00:32:38,360
of 14 patients in this cluster.

537
00:32:38,360 --> 00:32:40,760
They were all at chronic
kidney disease stage

538
00:32:40,760 --> 00:32:42,770
4, so quite severe.

539
00:32:42,770 --> 00:32:44,300
They were all hypertensive.

540
00:32:44,300 --> 00:32:49,450
They were all on ACE
inhibitors and statins,

541
00:32:49,450 --> 00:32:54,200
and everybody in that group
had that categorization.

542
00:32:54,200 --> 00:32:57,720
So if you look there
then you can say, OK,

543
00:32:57,720 --> 00:33:01,020
for all the things we
know about that patient,

544
00:33:01,020 --> 00:33:06,650
what are the probabilistic
relationships between them?

545
00:33:06,650 --> 00:33:09,840
And what we find is that--

546
00:33:09,840 --> 00:33:11,990
man, I can't read these.

547
00:33:11,990 --> 00:33:16,340
So these nodes
imply other nodes,

548
00:33:16,340 --> 00:33:22,490
and the strength of the arrows
is proportional to their width.

549
00:33:22,490 --> 00:33:26,180
And so this is a
representation of everything

550
00:33:26,180 --> 00:33:28,640
that we've learned
about that cluster,

551
00:33:28,640 --> 00:33:31,490
but remember, only
from those 14 patients.

552
00:33:31,490 --> 00:33:34,490
So I'm not sure I would
take this to the bank

553
00:33:34,490 --> 00:33:37,340
and rely on it too intensely.

554
00:33:37,340 --> 00:33:40,640
But they then, by
hand, abstract it

555
00:33:40,640 --> 00:33:46,610
and say, well, let's look at
an interpretation of this.

556
00:33:46,610 --> 00:33:49,610
And so if they look
in typical patterns

557
00:33:49,610 --> 00:33:51,920
that they see in
that cluster, they

558
00:33:51,920 --> 00:33:58,130
say, hmm, we see an
office visit in which

559
00:33:58,130 --> 00:34:02,060
the patient is on
these medications

560
00:34:02,060 --> 00:34:04,070
and has these procedures.

561
00:34:04,070 --> 00:34:05,870
Then they're hospitalized.

562
00:34:05,870 --> 00:34:12,500
Then there's
another-- let's see.

563
00:34:12,500 --> 00:34:13,340
No, I'm sorry.

564
00:34:17,290 --> 00:34:19,139
Yeah, yellow node
is an office visit.

565
00:34:19,139 --> 00:34:21,230
So they're hospitalized.

566
00:34:21,230 --> 00:34:23,210
They then get an
education visit,

567
00:34:23,210 --> 00:34:26,540
so that's typically with the
nurse or nurse practitioner

568
00:34:26,540 --> 00:34:29,130
to explain to them what
they ought to be doing.

569
00:34:29,130 --> 00:34:30,679
They have another hospital--

570
00:34:30,679 --> 00:34:32,600
they have another office visit.

571
00:34:32,600 --> 00:34:34,280
They have a hospital visit.

572
00:34:34,280 --> 00:34:38,210
They have another hospital
visit, and then they die.

573
00:34:38,210 --> 00:34:43,130
So that, unfortunately,
is a not atypical pattern

574
00:34:43,130 --> 00:34:47,210
that you see in patients who
are at a pretty severe state

575
00:34:47,210 --> 00:34:49,639
of chronic kidney disease.

576
00:34:49,639 --> 00:34:52,219
And we don't know
from this diagram

577
00:34:52,219 --> 00:34:58,830
how long this process
takes to take place.

578
00:34:58,830 --> 00:35:01,520
So I have some questions.

579
00:35:01,520 --> 00:35:04,190
There are a lot of subgroups.

580
00:35:04,190 --> 00:35:08,220
Some of them were fairly
similar to others.

581
00:35:08,220 --> 00:35:13,810
They have between 10 and 158
patients in each subgroup.

582
00:35:13,810 --> 00:35:15,720
So I would feel
much better if they

583
00:35:15,720 --> 00:35:22,590
had between 1,000 and
15,000 or something

584
00:35:22,590 --> 00:35:27,220
patients in each group, or
150,000 patients in each group.

585
00:35:27,220 --> 00:35:32,850
I would feel much more
believing in the representations

586
00:35:32,850 --> 00:35:34,200
that they found.

587
00:35:34,200 --> 00:35:36,090
And the other
problem is that even

588
00:35:36,090 --> 00:35:38,190
within an individual
subgroup, you can

589
00:35:38,190 --> 00:35:40,360
find very different patterns.

590
00:35:40,360 --> 00:35:46,350
So, for example, here is a
pattern where, again, a person

591
00:35:46,350 --> 00:35:48,360
has a couple of office visits.

592
00:35:48,360 --> 00:35:50,160
They go to the hospital.

593
00:35:50,160 --> 00:36:01,515
Or they go to the hospital
twice with slightly different--

594
00:36:01,515 --> 00:36:02,015
yes.

595
00:36:02,015 --> 00:36:06,970
So this person at this point
is in acute kidney injury.

596
00:36:06,970 --> 00:36:10,170
So you can get there either
directly from the office visit

597
00:36:10,170 --> 00:36:12,960
or from an earlier
hospitalization,

598
00:36:12,960 --> 00:36:14,710
and then they die.

599
00:36:14,710 --> 00:36:17,440
And so this is part
of that pattern.

600
00:36:17,440 --> 00:36:22,380
But here's another pattern mined
from exactly the same subgroup.

601
00:36:22,380 --> 00:36:25,840
Now, this subgroup has
122 patients in it,

602
00:36:25,840 --> 00:36:28,290
so there's a little
bit more heterogeneity.

603
00:36:28,290 --> 00:36:30,570
But what you see here
is that a patient

604
00:36:30,570 --> 00:36:35,550
is going back and forth between
education visits and doctor's

605
00:36:35,550 --> 00:36:38,370
visits, back and
forth between doctors

606
00:36:38,370 --> 00:36:42,300
visits and hospitalizations,
then a hospitalization, then

607
00:36:42,300 --> 00:36:46,750
another hospitalization,
but they're surviving.

608
00:36:46,750 --> 00:36:52,360
So it's a little bit tricky,
but I think this is a good idea,

609
00:36:52,360 --> 00:36:53,970
but there are
probably improvements

610
00:36:53,970 --> 00:36:56,370
that are possible on
the technique that's

611
00:36:56,370 --> 00:36:57,720
being used here.

612
00:36:57,720 --> 00:37:00,780
And, of course, much more
data would be very helpful

613
00:37:00,780 --> 00:37:02,940
in order to really
delineate what's

614
00:37:02,940 --> 00:37:04,200
going on in these patients.

615
00:37:10,670 --> 00:37:14,570
Here's a similar idea
that I was involved.

616
00:37:14,570 --> 00:37:18,710
Jeff Klann did his
PhD at Regenstrief,

617
00:37:18,710 --> 00:37:22,550
which is a very well-known,
very early adopter

618
00:37:22,550 --> 00:37:27,240
of computerized information
systems in Indiana.

619
00:37:27,240 --> 00:37:31,712
And so what he started
off-- and he said, hmm.

620
00:37:34,310 --> 00:37:36,770
You know the Amazon
recommendation system

621
00:37:36,770 --> 00:37:42,230
that says you just
bought this camera lends,

622
00:37:42,230 --> 00:37:45,800
and other people who bought
this camera lens also

623
00:37:45,800 --> 00:37:49,460
bought a cleaning kit
and a battery that goes

624
00:37:49,460 --> 00:37:52,100
with that camera, and so on?

625
00:37:52,100 --> 00:37:54,650
So he said, why don't
we apply that same idea

626
00:37:54,650 --> 00:37:56,630
to medical orders?

627
00:37:56,630 --> 00:38:01,400
And so he took the record of
all the orders at Regenstrief,

628
00:38:01,400 --> 00:38:04,100
and he basically
built an approximation

629
00:38:04,100 --> 00:38:07,740
to the Amazon recommendation
system that said,

630
00:38:07,740 --> 00:38:11,300
hey, other doctors who have
ordered the following set

631
00:38:11,300 --> 00:38:14,750
of tests have also ordered
this additional test

632
00:38:14,750 --> 00:38:16,400
that you didn't order.

633
00:38:16,400 --> 00:38:19,130
Maybe you should
consider doing it.

634
00:38:19,130 --> 00:38:23,840
Or conversely, other doctors who
have ordered this set of tests

635
00:38:23,840 --> 00:38:28,400
have never ordered this
other one in addition.

636
00:38:28,400 --> 00:38:30,650
And so are you sure
you really need it?

637
00:38:30,650 --> 00:38:33,860
So that was the idea.

638
00:38:33,860 --> 00:38:37,430
And what he did was
he focused on four

639
00:38:37,430 --> 00:38:39,540
different clinical issues.

640
00:38:39,540 --> 00:38:42,470
So one of them was an
emergency department visit

641
00:38:42,470 --> 00:38:48,110
for back pain, pregnancy,
so labor and delivery,

642
00:38:48,110 --> 00:38:50,790
hypertension in the
urgent visit clinic--

643
00:38:50,790 --> 00:38:54,870
so the urgent visit clinic
is one of these lower-level

644
00:38:54,870 --> 00:38:59,180
non-emergency department,
cheaper, lower level of care,

645
00:38:59,180 --> 00:39:03,140
but still urgent care kinds
of clinics that many hospitals

646
00:39:03,140 --> 00:39:06,890
have established in order to try
to keep people who are not that

647
00:39:06,890 --> 00:39:10,430
sick out of the emergency
department and in this

648
00:39:10,430 --> 00:39:13,700
lower-intensity clinic--

649
00:39:13,700 --> 00:39:16,250
and hypertension, and
high blood pressure,

650
00:39:16,250 --> 00:39:19,340
and then altered mental state
in the intensive care unit.

651
00:39:19,340 --> 00:39:23,270
So people in the ICU
are often medicated,

652
00:39:23,270 --> 00:39:26,930
and they become wacko,
and so this is trying

653
00:39:26,930 --> 00:39:28,970
to take care of such patients.

654
00:39:28,970 --> 00:39:31,250
They used three years
of encountered data

655
00:39:31,250 --> 00:39:33,440
from Regenstrief.

656
00:39:33,440 --> 00:39:37,670
And for each domain,
they limited themselves

657
00:39:37,670 --> 00:39:42,710
to the 40 most frequent orders,
and, again, low granularity.

658
00:39:42,710 --> 00:39:45,050
So, for example,
drug, but not the dose

659
00:39:45,050 --> 00:39:49,400
of the drug for
medications, and the 10 most

660
00:39:49,400 --> 00:39:56,240
frequent comorbidities or
co-occurring diagnoses.

661
00:39:56,240 --> 00:40:00,770
So this is an example of wisdom
of the crowd kind of approach

662
00:40:00,770 --> 00:40:04,970
that says, well, what
your colleagues do

663
00:40:04,970 --> 00:40:07,490
is probably a good
representation of what

664
00:40:07,490 --> 00:40:09,470
you ought to be doing.

665
00:40:09,470 --> 00:40:13,745
Now, what's an obvious
pitfall of this approach?

666
00:40:16,586 --> 00:40:18,810
I'm just checking to
see if you're awake.

667
00:40:18,810 --> 00:40:20,540
Yeah?

668
00:40:20,540 --> 00:40:23,517
AUDIENCE: Just reinforce
whatever's [INAUDIBLE]..

669
00:40:23,517 --> 00:40:25,350
PETER SZOLOVITS: Yeah,
if they're all bozos,

670
00:40:25,350 --> 00:40:27,460
they're going to train
you to be a bozo too.

671
00:40:29,970 --> 00:40:32,220
And there's a lot
of stuff in medicine

672
00:40:32,220 --> 00:40:35,310
that is not very
well-supported by evidence,

673
00:40:35,310 --> 00:40:38,640
where, in fact, people
have developed traditions

674
00:40:38,640 --> 00:40:41,400
of doing things a certain way
that may not be the right way

675
00:40:41,400 --> 00:40:42,460
to do it.

676
00:40:42,460 --> 00:40:45,290
And this just reinforces that.

677
00:40:45,290 --> 00:40:48,500
On the other hand, it
probably does reduce variance

678
00:40:48,500 --> 00:40:51,380
in the sense that we talked
about at the beginning.

679
00:40:51,380 --> 00:40:54,650
And so, as a result, it may
be a reasonable approach,

680
00:40:54,650 --> 00:40:58,010
if you're willing to
tolerate some exceptions.

681
00:40:58,010 --> 00:41:02,180
My favorite story is
Semmelweiss figured out

682
00:41:02,180 --> 00:41:07,580
that having a baby in
a hospital in Vienna

683
00:41:07,580 --> 00:41:12,020
was extremely dangerous
for the mother,

684
00:41:12,020 --> 00:41:13,730
because they would
die of what was

685
00:41:13,730 --> 00:41:19,070
called "child bed fever," which
was basically an infection.

686
00:41:19,070 --> 00:41:22,160
And Semmelweiss figured
out that maybe there

687
00:41:22,160 --> 00:41:24,440
was-- this was before Pasteur.

688
00:41:24,440 --> 00:41:26,180
But he figured out
that maybe there

689
00:41:26,180 --> 00:41:29,180
was something that was being
transmitted from one woman

690
00:41:29,180 --> 00:41:33,560
to the next that was causing
this child bed fever,

691
00:41:33,560 --> 00:41:35,030
and, of course, he was right.

692
00:41:35,030 --> 00:41:39,620
And he did an experiment,
where on his maternity ward,

693
00:41:39,620 --> 00:41:43,370
he had all of the
younger doctors

694
00:41:43,370 --> 00:41:47,450
wash their hands with some
sort of alcohol or something

695
00:41:47,450 --> 00:41:50,930
to kill whatever they
were transmitting.

696
00:41:50,930 --> 00:41:55,220
And their death rate
from this child bed fever

697
00:41:55,220 --> 00:41:57,730
dropped to almost 0.

698
00:41:57,730 --> 00:42:01,800
And he went to his colleagues
and he said, hey, guys, we

699
00:42:01,800 --> 00:42:04,650
could really make the
world a better place

700
00:42:04,650 --> 00:42:06,720
and stop killing women.

701
00:42:06,720 --> 00:42:10,200
And they looked at
him, and they said,

702
00:42:10,200 --> 00:42:15,900
you know, these hands
heal, they don't kill.

703
00:42:15,900 --> 00:42:20,490
Many of them were upper
class or noblemen who

704
00:42:20,490 --> 00:42:22,500
had gone into this profession.

705
00:42:22,500 --> 00:42:26,460
The idea that somehow they were
responsible for transmitting

706
00:42:26,460 --> 00:42:29,520
what turns out to
be bacteria was just

707
00:42:29,520 --> 00:42:31,380
a non-starter for them.

708
00:42:31,380 --> 00:42:33,930
And Semmelweiss wound
up ending his days

709
00:42:33,930 --> 00:42:37,630
in a mental institution,
because he went nuts.

710
00:42:37,630 --> 00:42:40,680
He was unable to
change practice even

711
00:42:40,680 --> 00:42:45,810
though he had done an experiment
to demonstrate that it worked.

712
00:42:45,810 --> 00:42:48,780
So this is a case where
the wisdom of the crowd

713
00:42:48,780 --> 00:42:52,560
was not so good and
led to bad outcomes.

714
00:42:55,230 --> 00:42:59,040
So like Amazon's
recommendation system,

715
00:42:59,040 --> 00:43:02,670
it automates the learning
of decision support rules.

716
00:43:02,670 --> 00:43:07,440
And what's attractive about
this is that because it's

717
00:43:07,440 --> 00:43:12,810
induced from real data, it tends
to deal with more complex cases

718
00:43:12,810 --> 00:43:17,010
than the sort of simple,
stereotypical cases

719
00:43:17,010 --> 00:43:19,620
for which people can
develop guidelines,

720
00:43:19,620 --> 00:43:22,110
for example, where
they can anticipate

721
00:43:22,110 --> 00:43:24,960
what's going to happen
in various circumstances.

722
00:43:24,960 --> 00:43:28,320
So he used the Bayesian
networking model

723
00:43:28,320 --> 00:43:32,760
that used diagnoses possible
orders and evidence, which

724
00:43:32,760 --> 00:43:36,790
is the results from orders
that were already completed.

725
00:43:36,790 --> 00:43:39,840
There's a system out of
University of Pittsburgh,

726
00:43:39,840 --> 00:43:43,950
called Tetrad, that implements
a nice version of something

727
00:43:43,950 --> 00:43:46,470
called Greedy
Equivalent Search, which

728
00:43:46,470 --> 00:43:51,190
is a faster way of
searching through the space

729
00:43:51,190 --> 00:43:55,290
of Bayesian networks for
an appropriate network that

730
00:43:55,290 --> 00:43:57,420
represents your data.

731
00:43:57,420 --> 00:44:01,800
So it's a highly
combinatorial problem,

732
00:44:01,800 --> 00:44:05,460
and the cleverness in this
is that it figures out

733
00:44:05,460 --> 00:44:09,630
classes of Bayesian networks
that, by definition, would

734
00:44:09,630 --> 00:44:11,790
fit the data equally well.

735
00:44:11,790 --> 00:44:15,960
And it does it by class rather
than by individual network,

736
00:44:15,960 --> 00:44:20,110
and so it gets a nice
combinatorial reduction.

737
00:44:20,110 --> 00:44:26,250
And what Jeff found is, for
example, in the pregnancy

738
00:44:26,250 --> 00:44:29,670
network, these
are the nodes that

739
00:44:29,670 --> 00:44:33,150
correspond to
various interventions

740
00:44:33,150 --> 00:44:35,710
and various conditions.

741
00:44:35,710 --> 00:44:41,070
And this is the Bayesian network
that best fits that data.

742
00:44:41,070 --> 00:44:44,220
It's reasonably complicated.

743
00:44:44,220 --> 00:44:46,230
Here are some others.

744
00:44:46,230 --> 00:44:50,230
This is for the emergency
department case.

745
00:44:50,230 --> 00:44:54,660
So you see that you have things
like chest pain and abdominal

746
00:44:54,660 --> 00:44:58,140
pain presenting
diagnoses, and then

747
00:44:58,140 --> 00:45:00,750
you have various
procedures, like an abdomen

748
00:45:00,750 --> 00:45:06,060
CT, or a pelvic CT, or
a chest CT, or a head

749
00:45:06,060 --> 00:45:10,080
CT, or a basic metabolic
panel, et cetera,

750
00:45:10,080 --> 00:45:12,630
and this gives you the
probabilistic relationships

751
00:45:12,630 --> 00:45:14,560
between them.

752
00:45:14,560 --> 00:45:21,240
And so what they were able to
do is to take this Bayesian

753
00:45:21,240 --> 00:45:24,510
network representation,
and then if you

754
00:45:24,510 --> 00:45:29,700
lay a particular patient's
data on that representation,

755
00:45:29,700 --> 00:45:33,850
that corresponds to fixing
the value of certain nodes.

756
00:45:33,850 --> 00:45:36,720
And then you do Bayesian
inference to figure out

757
00:45:36,720 --> 00:45:39,780
the probabilities of
the unobserved nodes,

758
00:45:39,780 --> 00:45:43,440
and you recommend the highest
probability interventions

759
00:45:43,440 --> 00:45:46,330
that have not yet been done.

760
00:45:46,330 --> 00:45:48,240
So it's a little bit
like, if you remember,

761
00:45:48,240 --> 00:45:51,100
we talked about
sequential diagnosis.

762
00:45:51,100 --> 00:45:53,020
This is a little
bit in that spirit,

763
00:45:53,020 --> 00:45:57,030
but it's a much more complicated
Bayesian network model rather

764
00:45:57,030 --> 00:46:00,490
than a naive-based model.

765
00:46:00,490 --> 00:46:03,550
And so the interface
looks like this.

766
00:46:03,550 --> 00:46:07,680
You have-- it's called the
Iterative Treatment Suggestions

767
00:46:07,680 --> 00:46:11,400
algorithm, and it
shows the doctor

768
00:46:11,400 --> 00:46:15,050
that these are the
problems of the patient,

769
00:46:15,050 --> 00:46:18,000
and the current orders, and
the probability that you

770
00:46:18,000 --> 00:46:23,580
might ask to have any
one of these orders done.

771
00:46:23,580 --> 00:46:30,690
And what they're able to show is
that this does reasonably well.

772
00:46:30,690 --> 00:46:33,300
Obviously, it wouldn't
have been published if they

773
00:46:33,300 --> 00:46:35,680
hadn't been able to show that.

774
00:46:35,680 --> 00:46:42,300
And so what you see is that, for
example, the next order that's

775
00:46:42,300 --> 00:46:47,010
done in an inpatient pregnancy
using this Bayesian network

776
00:46:47,010 --> 00:46:53,070
formalism has a position of
about fourth on the list.

777
00:46:53,070 --> 00:46:57,220
So their criterion for
judging this algorithm

778
00:46:57,220 --> 00:47:01,270
is, is it raising the
things that people actually

779
00:47:01,270 --> 00:47:04,420
do too high on the
list of the recommended

780
00:47:04,420 --> 00:47:07,510
list, on the recommended
set of actions

781
00:47:07,510 --> 00:47:09,250
that you consider doing?

782
00:47:09,250 --> 00:47:12,400
And you see that it's
fourth, on average,

783
00:47:12,400 --> 00:47:16,420
in inpatient pregnancy,
about sixth in the ICU,

784
00:47:16,420 --> 00:47:19,330
about sixth in the
emergency department,

785
00:47:19,330 --> 00:47:22,630
and about fifth in the
urgent care clinic.

786
00:47:22,630 --> 00:47:24,370
So that's pretty
good, because that

787
00:47:24,370 --> 00:47:27,520
means that even if you're
looking at an iPhone,

788
00:47:27,520 --> 00:47:30,730
there's enough screen
real estate that it'll

789
00:47:30,730 --> 00:47:34,540
be on the so-called first
page of Google hits,

790
00:47:34,540 --> 00:47:39,280
which is the only thing
people ever pay attention to.

791
00:47:39,280 --> 00:47:43,390
And, in fact, they can
show that the average list

792
00:47:43,390 --> 00:47:48,320
position corresponds to the
order rank by frequency,

793
00:47:48,320 --> 00:47:53,770
but that their model does a
reasonably good job of keeping

794
00:47:53,770 --> 00:48:01,510
you within the first 10 or
so for much of this range.

795
00:48:05,750 --> 00:48:08,300
I'm going to shift gears again.

796
00:48:08,300 --> 00:48:10,610
So Adam Right, you've met.

797
00:48:10,610 --> 00:48:14,570
He was discussant in one
of our earlier classes.

798
00:48:14,570 --> 00:48:18,290
And Adam's been very active
in trying to deploy decision

799
00:48:18,290 --> 00:48:19,880
support systems.

800
00:48:19,880 --> 00:48:25,420
And he had an interesting
episode back in--

801
00:48:25,420 --> 00:48:28,140
when was this--

802
00:48:28,140 --> 00:48:29,100
2016.

803
00:48:29,100 --> 00:48:32,040
So it must have been
a little before 2016.

804
00:48:32,040 --> 00:48:36,000
He went to demonstrate this
great decision support system

805
00:48:36,000 --> 00:48:39,780
that they had implemented
at the Brigham,

806
00:48:39,780 --> 00:48:45,030
and he put in a fake case where
an alert should have gone off

807
00:48:45,030 --> 00:48:51,960
for a patient who has been on
a particular drug for more than

808
00:48:51,960 --> 00:48:58,260
a year and needs to have their
thyroid stimulating hormone

809
00:48:58,260 --> 00:49:01,590
measured in order to check
for a potential side effect

810
00:49:01,590 --> 00:49:07,120
of long-term use of amiodarone,
as well as to have their--

811
00:49:07,120 --> 00:49:11,100
ALT is a liver test,
liver enzyme test.

812
00:49:11,100 --> 00:49:13,410
So they needed both
of those tests.

813
00:49:13,410 --> 00:49:16,710
He was demonstrating
this wonderful system.

814
00:49:16,710 --> 00:49:20,340
He put in a fake patient
who had these conditions,

815
00:49:20,340 --> 00:49:22,980
and the alert didn't go off.

816
00:49:22,980 --> 00:49:27,870
So he goes, hmm,
what's going on?

817
00:49:27,870 --> 00:49:30,930
And they went back,
and they discovered

818
00:49:30,930 --> 00:49:39,240
that in 2009 the system's
internal code for amiodarone

819
00:49:39,240 --> 00:49:43,040
had been changed
from 40 to 70-99.

820
00:49:43,040 --> 00:49:45,160
Who knows why?

821
00:49:45,160 --> 00:49:47,340
But the rule logic in
the system was never

822
00:49:47,340 --> 00:49:49,810
updated to reflect this change.

823
00:49:49,810 --> 00:49:55,230
And so, in fact, if
you look at the history

824
00:49:55,230 --> 00:49:57,907
of the use of amiodarone--

825
00:49:57,907 --> 00:49:59,490
by the way, it's an
interesting graph.

826
00:49:59,490 --> 00:50:03,780
The blue dots are weekdays, and
the black dots are weekends.

827
00:50:03,780 --> 00:50:07,950
So not a lot goes on in the
hospital during the weekend.

828
00:50:07,950 --> 00:50:11,250
But what you see is that--

829
00:50:11,250 --> 00:50:15,060
I don't know what happened
before about the end of 2009.

830
00:50:15,060 --> 00:50:18,240
They probably weren't running
that rule or something.

831
00:50:18,240 --> 00:50:21,210
But what you see is sort
of a gradual increase

832
00:50:21,210 --> 00:50:26,010
in the use of this rule, and
then you see a long decrease

833
00:50:26,010 --> 00:50:32,640
from 2010 up through 2013 when
they discovered this problem.

834
00:50:32,640 --> 00:50:34,350
Now, why a decrease?

835
00:50:34,350 --> 00:50:36,420
I mean, it's not a
sudden jump to 0.

836
00:50:39,030 --> 00:50:43,290
And the reason was
that this came about--

837
00:50:43,290 --> 00:50:45,750
first of all, it
came about gradually,

838
00:50:45,750 --> 00:50:50,700
because the people who had had
this drug before that change

839
00:50:50,700 --> 00:50:55,290
in the software had
gotten the old code, which

840
00:50:55,290 --> 00:50:57,630
was still triggering the rule.

841
00:50:57,630 --> 00:51:01,050
It's just that as time went
on, more and more people

842
00:51:01,050 --> 00:51:06,810
who needed the test had gotten
the drug with its new code.

843
00:51:06,810 --> 00:51:12,360
And with that new code, it was
no longer triggering the rule.

844
00:51:12,360 --> 00:51:15,900
And then this is the point at
which they discovered the bug,

845
00:51:15,900 --> 00:51:17,040
and then they fixed it.

846
00:51:17,040 --> 00:51:18,840
Of course, it came
right back up again.

847
00:51:27,280 --> 00:51:28,270
Oh.

848
00:51:28,270 --> 00:51:31,550
Well, I'll talk about some
of the others as well.

849
00:51:31,550 --> 00:51:36,670
So this was the amiodarone case.

850
00:51:36,670 --> 00:51:40,510
So it fell suddenly, as
some patients were taken off

851
00:51:40,510 --> 00:51:44,530
the drug and others were started
with this new internal code.

852
00:51:47,760 --> 00:51:52,270
And as I said, the alert
logic was fixed back in 2013.

853
00:51:52,270 --> 00:51:52,770
Yeah?

854
00:51:52,770 --> 00:51:55,062
AUDIENCE: So I don't know
how hospital IT systems work,

855
00:51:55,062 --> 00:51:56,705
and it might vary
from place to place.

856
00:51:56,705 --> 00:51:59,020
But is there ever a notion
of like this computer needs

857
00:51:59,020 --> 00:52:00,890
to be updated for the
software, but that one already

858
00:52:00,890 --> 00:52:01,416
got updated?

859
00:52:01,416 --> 00:52:03,915
Or are they all synced
up so that they all

860
00:52:03,915 --> 00:52:05,380
get updated at the same time?

861
00:52:05,380 --> 00:52:06,922
PETER SZOLOVITS:
They tend to all get

862
00:52:06,922 --> 00:52:08,680
updated at the same time.

863
00:52:08,680 --> 00:52:11,650
There are disasters
that have happened

864
00:52:11,650 --> 00:52:13,990
in that updating process.

865
00:52:13,990 --> 00:52:18,220
Famously, the Beth Israel was
down for about three days.

866
00:52:18,220 --> 00:52:21,430
Their computer
system just crashed.

867
00:52:21,430 --> 00:52:24,760
And what they discovered is that
they had this very complicated

868
00:52:24,760 --> 00:52:31,330
network in which there were
cyclic dependencies in order

869
00:52:31,330 --> 00:52:33,580
to boot up different systems.

870
00:52:33,580 --> 00:52:35,830
So some system had
to be up in order

871
00:52:35,830 --> 00:52:38,440
to let some other
system be up, which

872
00:52:38,440 --> 00:52:42,040
had to be up in order to
let the first system be up.

873
00:52:42,040 --> 00:52:45,310
And, of course, in
normal operation,

874
00:52:45,310 --> 00:52:47,270
they never take down
the whole system,

875
00:52:47,270 --> 00:52:50,860
and so nobody had discovered
this until there was--

876
00:52:50,860 --> 00:52:53,380
Cisco screwed them.

877
00:52:53,380 --> 00:52:58,360
There was some fix in the
routers that caused everything

878
00:52:58,360 --> 00:53:01,700
to crash, and then they
couldn't bring it back up again.

879
00:53:01,700 --> 00:53:04,330
And so that was a big panic.

880
00:53:04,330 --> 00:53:06,900
John Halamka, who's
the CIO there,

881
00:53:06,900 --> 00:53:08,590
is a former student of mine.

882
00:53:08,590 --> 00:53:13,450
And after this all
played out, I asked John,

883
00:53:13,450 --> 00:53:16,610
so what's the first thing
you did when this happened?

884
00:53:16,610 --> 00:53:19,690
And he said, I sent a
couple of panel trucks

885
00:53:19,690 --> 00:53:28,180
down to the Staples warehouse
to buy pads of paper,

886
00:53:28,180 --> 00:53:29,230
which is pretty smart.

887
00:53:33,190 --> 00:53:34,630
So here's another example.

888
00:53:34,630 --> 00:53:36,980
This is lead screening.

889
00:53:36,980 --> 00:53:40,720
And so this was a case where
there is a lead screening

890
00:53:40,720 --> 00:53:42,010
rule for two-year-olds.

891
00:53:42,010 --> 00:53:45,970
There is also one for one-,
three-, and four-year-olds.

892
00:53:45,970 --> 00:53:49,180
And there was no change in
screening for one-, three-,

893
00:53:49,180 --> 00:53:53,260
and four-year-olds, but the
screening for two-year-olds

894
00:53:53,260 --> 00:54:01,000
went from 300 or 400 a day down
to 0 for several years before

895
00:54:01,000 --> 00:54:06,850
they noticed it, and then went
back up to the previous level.

896
00:54:06,850 --> 00:54:10,960
And they never did quite
figure out what happened here,

897
00:54:10,960 --> 00:54:17,140
but something added two
incomplete clauses to the rule

898
00:54:17,140 --> 00:54:20,920
having to do with gender
and smoking status.

899
00:54:20,920 --> 00:54:25,510
But the clauses were incomplete,
and so they were actually

900
00:54:25,510 --> 00:54:31,150
looking for the case of neither
the gender nor the smoking

901
00:54:31,150 --> 00:54:33,880
status having been specified.

902
00:54:33,880 --> 00:54:36,160
So smoking status
for a two-year-old,

903
00:54:36,160 --> 00:54:39,580
you could imagine, is
not often specified,

904
00:54:39,580 --> 00:54:42,550
but gender typically is.

905
00:54:42,550 --> 00:54:47,170
And so the rule never
fired because of that,

906
00:54:47,170 --> 00:54:50,780
and they have no idea how
these changes were made.

907
00:54:50,780 --> 00:54:53,560
There's a complicated
logging system

908
00:54:53,560 --> 00:54:58,100
that logs all the changes, and
it crashed and lost its logging

909
00:54:58,100 --> 00:54:58,600
data.

910
00:54:58,600 --> 00:55:03,830
And it's a just so story.

911
00:55:03,830 --> 00:55:09,570
Chlamydia screen--
this was human error.

912
00:55:09,570 --> 00:55:13,430
And so they wound
up-- they found

913
00:55:13,430 --> 00:55:18,230
this very quickly, because they
had a two-month-old boy who

914
00:55:18,230 --> 00:55:21,440
had numerous duplicate
reminders, including

915
00:55:21,440 --> 00:55:25,490
suggestions for
mammograms, pap smears,

916
00:55:25,490 --> 00:55:28,670
pneumococcal vaccination,
and cholesterol

917
00:55:28,670 --> 00:55:32,090
screening, and a
suggestion to start

918
00:55:32,090 --> 00:55:34,190
the patient on various meds.

919
00:55:34,190 --> 00:55:38,330
So this was just a human
error in revising the rule,

920
00:55:38,330 --> 00:55:40,640
and that one they
found pretty quickly.

921
00:55:40,640 --> 00:55:44,120
So that's amusing.

922
00:55:44,120 --> 00:55:48,110
But what's interesting is these
guys went on to say, well, how

923
00:55:48,110 --> 00:55:52,100
could we monitor for this
in some ongoing fashion?

924
00:55:52,100 --> 00:55:54,530
And so they said, well,
there's this notion

925
00:55:54,530 --> 00:55:59,150
of change point detection,
which is an interesting machine

926
00:55:59,150 --> 00:56:01,280
learning problem, again.

927
00:56:01,280 --> 00:56:06,110
And so they said, well, suppose
we built a dynamic linear model

928
00:56:06,110 --> 00:56:08,480
that includes seasonality,
because we have

929
00:56:08,480 --> 00:56:11,420
to deal with the fact
that a lot of stuff

930
00:56:11,420 --> 00:56:14,030
happens Monday through
Friday and nothing happens

931
00:56:14,030 --> 00:56:15,510
on weekends?

932
00:56:15,510 --> 00:56:19,370
And so they created
a model that says

933
00:56:19,370 --> 00:56:24,590
that your output is some
function, f, of your inputs,

934
00:56:24,590 --> 00:56:27,540
plus some noise.

935
00:56:27,540 --> 00:56:31,760
The noise is Gaussian with
some variance, capital V,

936
00:56:31,760 --> 00:56:36,440
and that x evolves according
to some evolution that

937
00:56:36,440 --> 00:56:40,310
says it depends on the
previous value of x,

938
00:56:40,310 --> 00:56:43,800
plus some other noise,
which is also Gaussian.

939
00:56:43,800 --> 00:56:48,080
So that's the general sort of
time series modeling approach

940
00:56:48,080 --> 00:56:49,940
that people often take.

941
00:56:49,940 --> 00:56:53,120
And then they said, well, we
have to deal with seasonality.

942
00:56:53,120 --> 00:56:57,200
So what we're going to do is
define a period, namely a week,

943
00:56:57,200 --> 00:56:59,540
and then we're going
to separate out

944
00:56:59,540 --> 00:57:03,200
the states on different
days of the week

945
00:57:03,200 --> 00:57:07,850
in order to give us the ability
to model that seasonality.

946
00:57:07,850 --> 00:57:09,980
I worked on a different
project having

947
00:57:09,980 --> 00:57:15,280
to do with outbreak detection
for infectious diseases,

948
00:57:15,280 --> 00:57:18,140
and there the
periodicity was a year,

949
00:57:18,140 --> 00:57:21,260
because things like the
flu come in yearly cycles

950
00:57:21,260 --> 00:57:23,360
rather than in weekly cycles.

951
00:57:23,360 --> 00:57:26,930
And so that idea
is pretty common.

952
00:57:26,930 --> 00:57:29,810
And then they built this
multiprocess dynamic linear

953
00:57:29,810 --> 00:57:34,580
model that says,
basically, imagine

954
00:57:34,580 --> 00:57:38,240
that our data is
being generated by one

955
00:57:38,240 --> 00:57:43,380
of a set of these
dynamic linear models.

956
00:57:43,380 --> 00:57:45,770
And so we have an
additional state variable

957
00:57:45,770 --> 00:57:48,610
at each time that says
which of the models

958
00:57:48,610 --> 00:57:54,110
is in control to generate
the data at this point.

959
00:57:54,110 --> 00:58:00,110
And so if you have the set of
observations up to some time,

960
00:58:00,110 --> 00:58:03,650
t, then you can
compute the probability

961
00:58:03,650 --> 00:58:08,940
that model i is driving the
generator at this point.

962
00:58:08,940 --> 00:58:11,480
And so you can have
three basic models.

963
00:58:11,480 --> 00:58:14,030
You can have a
model that says it's

964
00:58:14,030 --> 00:58:16,190
a stable model, in
other words, what

965
00:58:16,190 --> 00:58:18,770
you expect is the steady state.

966
00:58:18,770 --> 00:58:22,550
So that would be the normal
weekly variation in volume

967
00:58:22,550 --> 00:58:24,500
for any of these alerts.

968
00:58:24,500 --> 00:58:27,710
You can have a model which
is an additive outlier.

969
00:58:27,710 --> 00:58:30,950
So that's something that says,
all of a sudden, something

970
00:58:30,950 --> 00:58:33,350
happened, like that
chlamydia screen

971
00:58:33,350 --> 00:58:37,610
or one of the other things
that had a very quick blip.

972
00:58:37,610 --> 00:58:40,010
Or you can have a
level shift change,

973
00:58:40,010 --> 00:58:45,530
like the change that
happened when the screening

974
00:58:45,530 --> 00:58:48,590
rules or the alert
rule for amiodarone

975
00:58:48,590 --> 00:58:51,740
stopped firing, because
it went from one level

976
00:58:51,740 --> 00:58:56,120
to a very different level
over a period of a relatively

977
00:58:56,120 --> 00:58:58,110
short period of time.

978
00:58:58,110 --> 00:59:02,120
And then what you can do is
calculate the probability

979
00:59:02,120 --> 00:59:08,040
of any of these models being
in control at the next time,

980
00:59:08,040 --> 00:59:10,160
and that's called the
change point score.

981
00:59:10,160 --> 00:59:14,750
And you can calculate this from
the data that you're given.

982
00:59:14,750 --> 00:59:17,030
And of course, they
have tons of data.

983
00:59:17,030 --> 00:59:20,970
It's a big hospital and
lots of these alerts go on.

984
00:59:20,970 --> 00:59:26,670
And if you plot this, there's
the data for a time series.

985
00:59:26,670 --> 00:59:29,690
So you see the weekly variation.

986
00:59:29,690 --> 00:59:32,480
But what you see is
that the probability

987
00:59:32,480 --> 00:59:40,730
of the steady behavior is quite
high except at certain points

988
00:59:40,730 --> 00:59:43,130
where it all of a sudden dips.

989
00:59:43,130 --> 00:59:45,980
And so those are places where
you suspect that something

990
00:59:45,980 --> 00:59:47,960
interesting is going on.

991
00:59:47,960 --> 00:59:53,960
And similarly, the probability
of a temporary offset

992
00:59:53,960 --> 01:00:00,110
goes up at these various points,
and the probability of a level

993
01:00:00,110 --> 01:00:02,740
shift goes up at this point.

994
01:00:02,740 --> 01:00:04,230
And you can see
that, indeed, there

995
01:00:04,230 --> 01:00:07,440
is a level shift
from essentially 0 up

996
01:00:07,440 --> 01:00:13,440
to this periodic behavior in
the original data sequence.

997
01:00:13,440 --> 01:00:16,740
And so they actually implemented
this in the hospital,

998
01:00:16,740 --> 01:00:19,320
and so now you get
not just alerts,

999
01:00:19,320 --> 01:00:24,090
but you get
meta-alerts that say,

1000
01:00:24,090 --> 01:00:27,510
this kid ought to be screened
for their lead levels,

1001
01:00:27,510 --> 01:00:30,480
but also the lead
level screening rule

1002
01:00:30,480 --> 01:00:33,570
hasn't fired as often as
we expected it to fire.

1003
01:00:41,790 --> 01:00:43,680
Yeah, so there are
a lot of details

1004
01:00:43,680 --> 01:00:46,710
in the paper that you can
look up, if you're interested.

1005
01:00:46,710 --> 01:00:50,250
And what they find
is that, if you

1006
01:00:50,250 --> 01:00:54,480
look at the area under the
delay false positive rate curve,

1007
01:00:54,480 --> 01:00:58,410
so you're trading off how
long it takes to be certain

1008
01:00:58,410 --> 01:01:02,670
that one of these conditions has
occurred versus how often you

1009
01:01:02,670 --> 01:01:09,180
cry wolf, and you see
that their algorithm does

1010
01:01:09,180 --> 01:01:11,580
much better than a
bunch of other things

1011
01:01:11,580 --> 01:01:15,030
that they tried it against,
which are earlier attempts

1012
01:01:15,030 --> 01:01:16,680
to do this.

1013
01:01:16,680 --> 01:01:19,690
And these are all highly
statistically significant,

1014
01:01:19,690 --> 01:01:22,770
so they got a nice
paper out of it.

1015
01:01:25,710 --> 01:01:28,440
In the remaining
time, I wanted to talk

1016
01:01:28,440 --> 01:01:32,550
about a number of other
issues that really

1017
01:01:32,550 --> 01:01:34,290
have to do with workflow.

1018
01:01:34,290 --> 01:01:37,680
So we've talked about
alerting, but there

1019
01:01:37,680 --> 01:01:39,840
are an interesting
set of studies

1020
01:01:39,840 --> 01:01:43,060
about how these alerting
systems actually work.

1021
01:01:43,060 --> 01:01:47,040
So there was a cool idea from
the Beth Israel Deaconess

1022
01:01:47,040 --> 01:01:52,440
Hospital here in Boston
where they said, well,

1023
01:01:52,440 --> 01:01:56,580
what we really need to
do is to escalate alerts.

1024
01:01:56,580 --> 01:02:00,780
So, for example, it's
quite typical in a hospital

1025
01:02:00,780 --> 01:02:04,950
that, if you're a doctor and
you have a patient who you have

1026
01:02:04,950 --> 01:02:07,620
just sent their
blood to the lab,

1027
01:02:07,620 --> 01:02:15,330
and let's say there serum
potassium comes back as 7 or 8,

1028
01:02:15,330 --> 01:02:18,990
that patient is at high risk of
going into cardiac arrhythmia

1029
01:02:18,990 --> 01:02:20,340
and dying.

1030
01:02:20,340 --> 01:02:23,640
And so your pager, in
those days, goes off,

1031
01:02:23,640 --> 01:02:25,800
and you read this text
message that says,

1032
01:02:25,800 --> 01:02:28,920
Mr. Jones has a
serum potassium of 8.

1033
01:02:28,920 --> 01:02:32,580
You'd better look in on him.

1034
01:02:32,580 --> 01:02:34,660
So what they did
was very clever.

1035
01:02:34,660 --> 01:02:38,430
They said, well, the problem is
busy doctors might ignore this.

1036
01:02:38,430 --> 01:02:42,030
And so we'll then start
a countdown timer.

1037
01:02:42,030 --> 01:02:47,100
And we'll say, did Dr.
Smith actually come and look

1038
01:02:47,100 --> 01:02:52,060
at Mr. Jones within 20 minutes?

1039
01:02:52,060 --> 01:02:54,060
And if the answer
is no, then they

1040
01:02:54,060 --> 01:02:58,420
send the page to the
doctor's boss that says,

1041
01:02:58,420 --> 01:03:01,980
hey, we sent this guy a
page, and within 20 minutes

1042
01:03:01,980 --> 01:03:05,460
he didn't look in
on the patient.

1043
01:03:05,460 --> 01:03:08,340
And then they start
another timer.

1044
01:03:08,340 --> 01:03:14,850
And they say, if that boss
doesn't respond within an hour,

1045
01:03:14,850 --> 01:03:18,900
then they send a page to
the head of the hospital

1046
01:03:18,900 --> 01:03:22,440
saying, you're her
infectious disease people

1047
01:03:22,440 --> 01:03:25,120
are doing a lousy job,
because they're not--

1048
01:03:25,120 --> 01:03:28,680
or in this case, you're
endocrine people, or whatever,

1049
01:03:28,680 --> 01:03:31,080
are doing a lousy job,
because they're not

1050
01:03:31,080 --> 01:03:33,820
responding to these alerts.

1051
01:03:33,820 --> 01:03:38,470
Now, how do you think
the doctors liked this?

1052
01:03:38,470 --> 01:03:40,320
Not much.

1053
01:03:40,320 --> 01:03:44,430
And there is a real
problem with overalerting.

1054
01:03:44,430 --> 01:03:46,890
And there is no
general rule that

1055
01:03:46,890 --> 01:03:51,300
says, how often can you bug
the head of the hospital

1056
01:03:51,300 --> 01:03:55,810
with an alert like this before
he or she just says, well,

1057
01:03:55,810 --> 01:03:59,070
turn off the damn thing,
I don't want to see these?

1058
01:03:59,070 --> 01:04:02,130
And clearly, if you set the
thresholds at different places,

1059
01:04:02,130 --> 01:04:04,090
you get different results.

1060
01:04:04,090 --> 01:04:07,650
So, for example, I remember
Tufts implemented a system

1061
01:04:07,650 --> 01:04:12,715
like this back in the 1980s,
but they would send a page

1062
01:04:12,715 --> 01:04:19,240
on every order where any of
the lab results were abnormal,

1063
01:04:19,240 --> 01:04:21,240
and that was way too much.

1064
01:04:21,240 --> 01:04:27,060
Because a lot of these
tests generate 20 results.

1065
01:04:27,060 --> 01:04:31,650
Normal is defined as the
95% confidence interval.

1066
01:04:31,650 --> 01:04:34,110
What are the chances
that out of 20 tests,

1067
01:04:34,110 --> 01:04:38,310
which aren't really independent,
but if they were, one of them

1068
01:04:38,310 --> 01:04:40,740
would be pretty guaranteed
to be out of range

1069
01:04:40,740 --> 01:04:42,790
for most of the patients?

1070
01:04:42,790 --> 01:04:46,890
And so basically every
test generated an alert

1071
01:04:46,890 --> 01:04:48,120
to the doctor.

1072
01:04:48,120 --> 01:04:50,250
And the doctors did
threaten to kill

1073
01:04:50,250 --> 01:04:52,800
the people who had
implemented the system,

1074
01:04:52,800 --> 01:04:55,110
and it got turned off.

1075
01:04:55,110 --> 01:04:58,050
A system like this, if
you set the threshold

1076
01:04:58,050 --> 01:05:03,180
to be not abnormal, but
life-threateningly abnormal,

1077
01:05:03,180 --> 01:05:08,730
and if you set the
rate and the time

1078
01:05:08,730 --> 01:05:11,970
durations such that it's
reasonable for people

1079
01:05:11,970 --> 01:05:16,320
to respond to it, then
maybe it can be acceptable.

1080
01:05:16,320 --> 01:05:21,420
When we did this project on
looking at how an emergency

1081
01:05:21,420 --> 01:05:24,810
department could anticipate
a flood of patients

1082
01:05:24,810 --> 01:05:27,480
because it looked like
flu season was starting,

1083
01:05:27,480 --> 01:05:31,200
for example, the
question we asked is,

1084
01:05:31,200 --> 01:05:36,390
how many false alarms a
month can you guys tolerate?

1085
01:05:36,390 --> 01:05:37,890
And they thought about it.

1086
01:05:37,890 --> 01:05:42,720
And the ED docs got together
and said, three times a month

1087
01:05:42,720 --> 01:05:46,370
you can cry wolf,
because we really

1088
01:05:46,370 --> 01:05:49,530
want to know when
it actually happens.

1089
01:05:49,530 --> 01:05:53,990
And we'd rather be prepared,
and we can tolerate a 10% error

1090
01:05:53,990 --> 01:05:56,690
rate on this prediction.

1091
01:05:56,690 --> 01:05:59,270
But I don't know what
it is in this domain.

1092
01:06:04,270 --> 01:06:06,700
Another interesting study was--

1093
01:06:06,700 --> 01:06:08,490
it's become quite popular.

1094
01:06:08,490 --> 01:06:10,570
I got a bunch of
emails from my doctor

1095
01:06:10,570 --> 01:06:15,010
today, because I had ordered
a refill on some prescription,

1096
01:06:15,010 --> 01:06:17,980
and he wanted to know how it's
going, and blah, blah, blah.

1097
01:06:17,980 --> 01:06:24,100
So the BI asked the question,
what fraction of those messages

1098
01:06:24,100 --> 01:06:27,400
are never read by the
patients that they're sent to?

1099
01:06:27,400 --> 01:06:29,620
Which is an important
question, because if you're

1100
01:06:29,620 --> 01:06:32,650
relying on that mode
of communication

1101
01:06:32,650 --> 01:06:36,160
as part of your workflow,
you'd like it to be 0.

1102
01:06:36,160 --> 01:06:40,490
It turned out only to be 3%,
which is remarkably good.

1103
01:06:40,490 --> 01:06:43,630
That means that most people
are actually paying attention

1104
01:06:43,630 --> 01:06:44,800
to those kinds of messages.

1105
01:06:47,350 --> 01:06:50,680
Then I wanted to say a few
words about the importance

1106
01:06:50,680 --> 01:06:54,490
of communication
and then finish up

1107
01:06:54,490 --> 01:06:58,150
by mentioning some so
far failed attempts

1108
01:06:58,150 --> 01:07:03,260
at really good integration of
all different data sources.

1109
01:07:03,260 --> 01:07:08,590
So as I said, the
BI started in 1994

1110
01:07:08,590 --> 01:07:11,380
with a system that
said, if you're

1111
01:07:11,380 --> 01:07:15,160
taking a renally-excreted
or a nephrotoxic drug,

1112
01:07:15,160 --> 01:07:18,280
then we're going to
warn people if there

1113
01:07:18,280 --> 01:07:20,350
is a rising creatinine
level, which

1114
01:07:20,350 --> 01:07:22,930
is an indication that
your kidneys are not

1115
01:07:22,930 --> 01:07:24,820
functioning so well.

1116
01:07:24,820 --> 01:07:28,090
Because, of course, if the
drug is renally excreted,

1117
01:07:28,090 --> 01:07:31,060
that means that if your kidneys
are not excreting things

1118
01:07:31,060 --> 01:07:33,130
at the rate they're
supposed to, you're

1119
01:07:33,130 --> 01:07:36,760
going to wind up building up
the amount of drug in your body,

1120
01:07:36,760 --> 01:07:39,340
and that can become toxic.

1121
01:07:39,340 --> 01:07:42,880
So they saw a 21-hour,
so almost a full day,

1122
01:07:42,880 --> 01:07:48,160
reduction in response time
from the medical staff

1123
01:07:48,160 --> 01:07:51,610
given these alerts versus
what happened before.

1124
01:07:51,610 --> 01:07:52,970
That's remarkable.

1125
01:07:52,970 --> 01:07:55,690
I mean, saving a
day in responding

1126
01:07:55,690 --> 01:07:58,780
to a condition like
this is really quite

1127
01:07:58,780 --> 01:08:01,540
an impressive
result. And they also

1128
01:08:01,540 --> 01:08:04,150
saw, in terms of
clinical outcome,

1129
01:08:04,150 --> 01:08:07,180
that the risk of
renal impairment

1130
01:08:07,180 --> 01:08:11,410
was reduced to about half of
the preintervention level.

1131
01:08:11,410 --> 01:08:15,130
So that earlier
response actually

1132
01:08:15,130 --> 01:08:17,620
was saving people's
kidney function

1133
01:08:17,620 --> 01:08:21,910
by getting people to
intervene earlier.

1134
01:08:21,910 --> 01:08:24,939
I found it interesting
they said 44% of doctors

1135
01:08:24,939 --> 01:08:29,950
found these alerts helpful,
28% found them annoying,

1136
01:08:29,950 --> 01:08:33,580
but 65% of them
wanted them continued

1137
01:08:33,580 --> 01:08:36,114
to be used in a survey.

1138
01:08:40,819 --> 01:08:43,180
Enrico Carrera is
one of my heroes.

1139
01:08:43,180 --> 01:08:45,609
He used to be in the UK.

1140
01:08:45,609 --> 01:08:47,720
He's now in Australia.

1141
01:08:47,720 --> 01:08:53,240
And he had this very deep
insight back in the 1980s.

1142
01:08:53,240 --> 01:08:56,890
He said, you know,
all you computer guys

1143
01:08:56,890 --> 01:09:01,060
who are treading on
this medical field

1144
01:09:01,060 --> 01:09:06,590
think that all of the action
is about decision-making,

1145
01:09:06,590 --> 01:09:08,240
but it's not.

1146
01:09:08,240 --> 01:09:11,740
All of the action is
really about communication,

1147
01:09:11,740 --> 01:09:16,130
that health care is
basically a team sport.

1148
01:09:16,130 --> 01:09:19,069
And unless we spend
much more time

1149
01:09:19,069 --> 01:09:21,990
studying what goes
on in communication,

1150
01:09:21,990 --> 01:09:23,689
we're going to miss the boat.

1151
01:09:23,689 --> 01:09:30,080
And then mostly, we didn't
pay any attention to him,

1152
01:09:30,080 --> 01:09:31,430
but he's kept at it.

1153
01:09:31,430 --> 01:09:35,180
So he said, well, how big
is the communication space?

1154
01:09:35,180 --> 01:09:44,630
So he cited a 1985 study
that said that about 50%

1155
01:09:44,630 --> 01:09:50,210
of requests for information
are ones that people

1156
01:09:50,210 --> 01:09:54,020
ask their colleague
for versus 26%

1157
01:09:54,020 --> 01:09:57,180
that they look up
in their own notes.

1158
01:09:57,180 --> 01:10:05,120
So if a doctor is on rounds,
walks into a patient's room

1159
01:10:05,120 --> 01:10:09,200
and says, I want to know has
this guy's temperature been

1160
01:10:09,200 --> 01:10:15,230
going up or down, a quarter of
the time he'll look at notes.

1161
01:10:15,230 --> 01:10:18,560
And half the time, he'll
turn to the nurse and say,

1162
01:10:18,560 --> 01:10:24,860
is this patient's
temperature going up or down?

1163
01:10:24,860 --> 01:10:27,860
So he says that's interesting.

1164
01:10:27,860 --> 01:10:31,430
Paul Tang did a study
in the '90s that

1165
01:10:31,430 --> 01:10:35,390
said that in a clinic,
about 60% of the time

1166
01:10:35,390 --> 01:10:41,450
is spent talking among the
staff, not doing anything else.

1167
01:10:41,450 --> 01:10:49,180
Enrico and one of his
colleagues said that almost 100%

1168
01:10:49,180 --> 01:10:51,890
of non-patient
record information,

1169
01:10:51,890 --> 01:10:54,730
in other words, the thing
that's not in the written health

1170
01:10:54,730 --> 01:10:58,480
record, is done by talking.

1171
01:10:58,480 --> 01:11:02,980
That's almost tautological,
because where else would you

1172
01:11:02,980 --> 01:11:04,420
get it?

1173
01:11:04,420 --> 01:11:10,060
And then Charlie Saffron at the
BI did a time and motion study

1174
01:11:10,060 --> 01:11:13,090
and was looking at, I think,
nursing behavior, and saying

1175
01:11:13,090 --> 01:11:17,200
that about half their time was
face-to-face communication,

1176
01:11:17,200 --> 01:11:21,070
about 10% with electronic
medical records,

1177
01:11:21,070 --> 01:11:25,570
and also a lot of email,
and voicemail, and paper

1178
01:11:25,570 --> 01:11:30,410
reminders as ways of
communicating among people.

1179
01:11:30,410 --> 01:11:37,120
So this was a study looking at--

1180
01:11:37,120 --> 01:11:41,920
this is that 1998 study
by Colera and Tombs.

1181
01:11:41,920 --> 01:11:44,410
And they're looking at
a consultant, the house

1182
01:11:44,410 --> 01:11:46,480
officer, another consultant.

1183
01:11:46,480 --> 01:11:48,190
These are British
titles, because this

1184
01:11:48,190 --> 01:11:49,920
was done in Australia--

1185
01:11:49,920 --> 01:11:51,700
a nurse, et cetera.

1186
01:11:51,700 --> 01:11:56,770
And they say, OK,
among hospital staff--

1187
01:11:56,770 --> 01:12:04,180
I think this was in
one shift, I believe,

1188
01:12:04,180 --> 01:12:07,450
I should have had
that on the slide--

1189
01:12:07,450 --> 01:12:11,180
this is the number of pages
that they sent and received.

1190
01:12:11,180 --> 01:12:14,260
So they range from
0 up to about 4.

1191
01:12:14,260 --> 01:12:17,410
The number of telephone
calls made and received--

1192
01:12:17,410 --> 01:12:20,890
this ranges from 0 up to 13.

1193
01:12:20,890 --> 01:12:22,700
Oh, here's the length
of observation.

1194
01:12:22,700 --> 01:12:25,600
So this was over a period
of about three hours

1195
01:12:25,600 --> 01:12:27,950
for each of these patients.

1196
01:12:27,950 --> 01:12:30,250
And this is the total
number of events.

1197
01:12:30,250 --> 01:12:31,180
So think about it.

1198
01:12:31,180 --> 01:12:35,380
In 3 and 1/2 hours, the
senior house officer

1199
01:12:35,380 --> 01:12:41,060
had 24 distinct communication
events happen to that person.

1200
01:12:41,060 --> 01:12:46,940
So that means, what,
that's like 7--

1201
01:12:46,940 --> 01:12:50,030
yeah, like 7 an hour.

1202
01:12:50,030 --> 01:12:57,550
So that's like 1 every
10 minutes, roughly.

1203
01:12:57,550 --> 01:13:01,307
So it's an interrupt-driven
kind of environment.

1204
01:13:05,210 --> 01:13:09,230
Here's one particular
subject that they looked at,

1205
01:13:09,230 --> 01:13:11,900
three and a quarter
hours of observation.

1206
01:13:11,900 --> 01:13:15,420
This person spent 86%
of their time talking.

1207
01:13:15,420 --> 01:13:20,060
31% were taken up
with 28 interruptions.

1208
01:13:20,060 --> 01:13:24,910
So even the interruptions
were being interrupted.

1209
01:13:24,910 --> 01:13:29,990
25% were multitasking with
two or more conversations.

1210
01:13:29,990 --> 01:13:34,290
87%, face-to-face or
on a phone or a pager.

1211
01:13:34,290 --> 01:13:36,890
So most of that is talk time.

1212
01:13:36,890 --> 01:13:40,970
And 13% dealing with
computers and patient notes.

1213
01:13:40,970 --> 01:13:44,810
So the communication
function is really important.

1214
01:13:44,810 --> 01:13:49,730
And I don't have anything
profound to say about it other

1215
01:13:49,730 --> 01:13:53,580
than I'll put up a pointer
to some of these papers.

1216
01:13:53,580 --> 01:13:56,490
But the kinds of things
they're considering

1217
01:13:56,490 --> 01:13:58,850
are, well, we could
introduce new channels,

1218
01:13:58,850 --> 01:14:02,900
or new types of messages, or
new communication policies

1219
01:14:02,900 --> 01:14:05,580
that say, you know
you may not interrupt

1220
01:14:05,580 --> 01:14:08,660
the person who's
taking care of patients

1221
01:14:08,660 --> 01:14:11,910
while they're doing it,
or something like that.

1222
01:14:11,910 --> 01:14:16,160
And then moving from synchronous
to asynchronous methods,

1223
01:14:16,160 --> 01:14:20,300
like voicemail, or
email, or Slack,

1224
01:14:20,300 --> 01:14:24,590
or some modern
communication mechanism.

1225
01:14:27,380 --> 01:14:28,790
Let me skip by these.

1226
01:14:34,510 --> 01:14:36,790
Next to the last
topic, quickly, how

1227
01:14:36,790 --> 01:14:38,710
do you keep from
dropping the ball?

1228
01:14:38,710 --> 01:14:40,540
So there are a lot
of analyses that

1229
01:14:40,540 --> 01:14:44,920
say that the biggest
mistakes in health care

1230
01:14:44,920 --> 01:14:47,770
are made not because somebody
makes the wrong decision,

1231
01:14:47,770 --> 01:14:50,920
but it's because somebody
fails to make a decision.

1232
01:14:50,920 --> 01:14:52,960
They just forget
about something.

1233
01:14:52,960 --> 01:14:55,795
They don't follow-up on
something that they ought to.

1234
01:14:55,795 --> 01:15:00,670
The patient is going along,
and you think everything's OK,

1235
01:15:00,670 --> 01:15:02,470
and you don't deal with it.

1236
01:15:02,470 --> 01:15:08,290
So inspired partly by
that escalation of pagers

1237
01:15:08,290 --> 01:15:11,110
that I read about
at the Beth Israel,

1238
01:15:11,110 --> 01:15:13,810
I said, well, this sounds
like what we really need

1239
01:15:13,810 --> 01:15:17,380
is a workflow engine that's
approximately a discrete event

1240
01:15:17,380 --> 01:15:18,640
simulator.

1241
01:15:18,640 --> 01:15:23,810
So has anybody built a discrete
events simulator in this class?

1242
01:15:23,810 --> 01:15:26,320
It's a fairly standard sort
of programming problem,

1243
01:15:26,320 --> 01:15:29,500
and it's useful in simulating
all kinds of things

1244
01:15:29,500 --> 01:15:32,980
that involve discrete events.

1245
01:15:32,980 --> 01:15:37,120
And the idea is that
you have a timeline,

1246
01:15:37,120 --> 01:15:39,730
and you run down the
timeline, and you

1247
01:15:39,730 --> 01:15:43,840
execute the next
activity that comes up.

1248
01:15:43,840 --> 01:15:47,260
And that activity
does something.

1249
01:15:47,260 --> 01:15:52,510
It sends an email, or it shoots
a rocket, or whatever field

1250
01:15:52,510 --> 01:15:54,460
you're doing the simulation in.

1251
01:15:54,460 --> 01:15:57,370
But most importantly, what
it does is-- the last thing

1252
01:15:57,370 --> 01:16:01,090
it does is it schedules
something else to happen later

1253
01:16:01,090 --> 01:16:02,740
in the timeline.

1254
01:16:02,740 --> 01:16:06,610
So, for example, for something
that happens once a day, when

1255
01:16:06,610 --> 01:16:09,880
it happens, the task
that runs schedules it

1256
01:16:09,880 --> 01:16:12,290
to happen again the next day.

1257
01:16:12,290 --> 01:16:15,130
And that means that it's going
to be continually operating

1258
01:16:15,130 --> 01:16:16,480
all the time.

1259
01:16:16,480 --> 01:16:20,770
So the idea I had was that what
you'd like to do is to say,

1260
01:16:20,770 --> 01:16:24,100
if at some time, t,
I have a task that

1261
01:16:24,100 --> 01:16:28,330
says do x or asks
z to do y, or both,

1262
01:16:28,330 --> 01:16:30,970
then the last thing
should be at some time

1263
01:16:30,970 --> 01:16:36,910
in the future schedule another
task that says, is y done?

1264
01:16:36,910 --> 01:16:41,860
And if not, then go notify
somebody or go remind somebody.

1265
01:16:41,860 --> 01:16:44,230
And as far as I
know, no hospital

1266
01:16:44,230 --> 01:16:48,580
and no electronic record system
has any capability like this,

1267
01:16:48,580 --> 01:16:52,270
but I still think
it's a terrific idea.

1268
01:16:52,270 --> 01:16:56,620
And then I wanted to
finish with a pointer

1269
01:16:56,620 --> 01:17:01,790
to a problem that is
still very much with us.

1270
01:17:01,790 --> 01:17:06,310
So in 1994, some
colleagues and I

1271
01:17:06,310 --> 01:17:11,020
wrote this thing we called
"The Guardian Angel Manifesto."

1272
01:17:11,020 --> 01:17:14,320
And the idea was that we
should engage patients

1273
01:17:14,320 --> 01:17:16,900
more in their own
care, because they

1274
01:17:16,900 --> 01:17:18,940
can keep track of
a lot of the things

1275
01:17:18,940 --> 01:17:23,140
that systems didn't do a very
good job of keeping track of.

1276
01:17:23,140 --> 01:17:27,370
And the idea was that you would
have a computational process

1277
01:17:27,370 --> 01:17:31,390
that would start off at the
time your parents conceived you

1278
01:17:31,390 --> 01:17:36,430
and run until your
autopsy after you died.

1279
01:17:36,430 --> 01:17:38,650
And during this
time, it would be

1280
01:17:38,650 --> 01:17:42,760
responsible for collecting
all the relevant health care

1281
01:17:42,760 --> 01:17:43,750
data about you.

1282
01:17:43,750 --> 01:17:46,660
So it would be your
electronic medical record,

1283
01:17:46,660 --> 01:17:48,190
but it would also be active.

1284
01:17:48,190 --> 01:17:51,340
So it would help you
communicate with your providers.

1285
01:17:51,340 --> 01:17:54,310
It would help educate you
about any conditions you have.

1286
01:17:54,310 --> 01:17:56,440
It would remind
you about things.

1287
01:17:56,440 --> 01:17:58,970
It would schedule stuff
for you, et cetera.

1288
01:17:58,970 --> 01:18:02,050
So this was a nice
science fiction vision.

1289
01:18:02,050 --> 01:18:06,100
And in the mid-2000s,
Adam Bosworth,

1290
01:18:06,100 --> 01:18:09,012
who was a VP of
Google, came to me.

1291
01:18:09,012 --> 01:18:10,720
And he said, you know,
I read your thing.

1292
01:18:10,720 --> 01:18:11,580
It's a good idea.

1293
01:18:11,580 --> 01:18:13,960
I'm going to do it.

1294
01:18:13,960 --> 01:18:18,610
So Google started up this thing
called Google Health, which

1295
01:18:18,610 --> 01:18:21,460
was more focused on being
at least the personal health

1296
01:18:21,460 --> 01:18:22,690
record.

1297
01:18:22,690 --> 01:18:26,590
They did a pilot with 1,600
people at Cleveland Clinic,

1298
01:18:26,590 --> 01:18:30,730
and then they went
public as a beta.

1299
01:18:30,730 --> 01:18:33,360
And three years
later, they killed it.

1300
01:18:36,340 --> 01:18:37,790
And they had a
bunch of partners.

1301
01:18:37,790 --> 01:18:42,460
So they had Allscripts,
and Beth Israel,

1302
01:18:42,460 --> 01:18:45,760
and Blue Cross of Massachusetts,
and the Cleveland Clinic,

1303
01:18:45,760 --> 01:18:47,660
and CVS, and so on.

1304
01:18:47,660 --> 01:18:49,960
So they did their job
of trying to connect

1305
01:18:49,960 --> 01:18:52,330
to a bunch of important players.

1306
01:18:52,330 --> 01:18:54,760
But, of course, they
didn't have everybody.

1307
01:18:54,760 --> 01:18:56,740
And so, for example,
I, of course,

1308
01:18:56,740 --> 01:19:00,130
immediately signed
up for an account,

1309
01:19:00,130 --> 01:19:06,040
and the only company that I had
ever dealt with out of that set

1310
01:19:06,040 --> 01:19:08,650
was Walgreens, where
I had bought a skin

1311
01:19:08,650 --> 01:19:11,440
cream one time for a skin rash.

1312
01:19:11,440 --> 01:19:14,470
And so my total medical
record consisted

1313
01:19:14,470 --> 01:19:16,390
of a skin rash
and a cream that I

1314
01:19:16,390 --> 01:19:19,120
had bought to take care of it--

1315
01:19:19,120 --> 01:19:22,250
not very helpful.

1316
01:19:22,250 --> 01:19:25,660
And so nobody, other
than these partners,

1317
01:19:25,660 --> 01:19:28,270
could enter data
automatically, which

1318
01:19:28,270 --> 01:19:31,720
meant that you had to be even
more anal compulsive than I

1319
01:19:31,720 --> 01:19:34,450
am in order to
sit there and type

1320
01:19:34,450 --> 01:19:38,410
in my entire medical
history into the system,

1321
01:19:38,410 --> 01:19:45,070
especially, because if I did so,
nobody would ever look at it.

1322
01:19:45,070 --> 01:19:48,100
Because if I go to
my doctor and say,

1323
01:19:48,100 --> 01:19:52,300
hey, Doc, here's the Google
URL for my medical record,

1324
01:19:52,300 --> 01:19:55,623
and here's the password by
which you can access it,

1325
01:19:55,623 --> 01:19:57,790
what do you think are the
odds that they're actually

1326
01:19:57,790 --> 01:19:59,274
going to look?

1327
01:19:59,274 --> 01:20:00,160
AUDIENCE: 0.

1328
01:20:00,160 --> 01:20:02,510
PETER SZOLOVITS: 0.

1329
01:20:02,510 --> 01:20:06,650
So the thing was an
absolute abject failure.

1330
01:20:06,650 --> 01:20:08,700
And people keep trying it.

1331
01:20:08,700 --> 01:20:11,360
And so far, nobody has
figured out how to do it,

1332
01:20:11,360 --> 01:20:13,520
but it's still a good idea.

1333
01:20:13,520 --> 01:20:16,840
With that, we'll
stop on workflow.