Everyone good
afternoon, good evening, good morning from the chat and
from the familiar names that I see, as well as the discussion
about people's locations. I can see that we have
folks from a wide range of destinations that it is
exciting to welcome you here with us to a session on applied
AI and health care, challenges, and opportunities. We are grateful to have this
collaboration between HBS Health Care Alumni Association,
of which I am a representative. The Harvard T.H. Chan School
of Public Health, and also the Harvard Chan School
Alumni Association. To begin the session
I will run us through some of the logistics. We invite you folks to submit
questions to our speakers as the session progresses. And you can do that by adding
your questions to the Q&A box.
As you may have noticed, the
session is being recorded. So if you submit a question
there is a good chance that if your question
is used in the session, that you will be called out,
your name will be named. So please be cognizant of that. Once again, it is a joint
program between our alumni associations and Harvard T.H.
Chan School of Public Health.
And I am grateful to
our collaboration, grateful to the schools,
grateful to my colleagues Jo Montrose, Grady Close,
to Trishan Panch, and all our outstanding panelists. David Rogers and the
team from Harvard T.H. Chan School
of Public Health. This program is a record
number of attendees that we're seeing here
both in the program, and a number of people that
are still supposed to join us. Obviously, an exciting topic. And we're conducting
this program with an eye to future further collaborations
between the Harvard HBS Health Care Alumni Association
and Harvard T.H. Chan School of Public Health. I'm excited to introduce Trishan
Panch who is a primary care physician and President of
the Harvard Chan School Alumni Association. He is a lecturer in health
policy and management at HSBH. He co-founded and
led Wolfram, which was sold to health edge owned
by Blackstone, in 2021 late last year. And served on the board
since its inception. Trishan, over to you. Thank you. Thanks Boris,
thank you everyone. Thank you for joining
us wherever in the world you'll be. Yeah, I mean it's pretty
mind blowing actually, there are 700 of
either have signed up to this across the world.
And then also this
is open invite, so there's people who have
signed up who aren't alone. I guess what we have to say
to those of you is, welcome and maybe we need to have a
discussion about your choice of education as well. But anyway no that's all,
of course, I'm joking there. But this is a really,
really interesting field. And it's incredibly broad. I think just to set it up. I just want to give
you a few thoughts.
So the first thing
is we are here within the envelope of the
respective visions of Srikant Datar, the Dean of
Harvard Business School, and Michelle Williams, the Dean
of the School of Public Health. And both of them have
stated the importance of design and user-centric
thinking and data science within all aspects of business. And on the public health
side these five frontiers of public health,
massive, structural issues such as climate change,
aging, health and nutrition, and chronic disease
management, addressing the effects of
structural violence that have come to prominence. And one that you might be
aware of is a risk pandemics. Now, there's no reason
that any of these issues are arbitrarily medical issues,
or public health issues, or just business issues. Of course, they're issues
that affect all of us. And that's what makes
them so interesting. And they require not
just the problems being understood but
solutions being developed, organizations being developed
to scale those solutions.
And all of that being
done rigorously, and in a sustainable
manner, that ensures a fair
distribution of things. So there's a lot of– is a very fertile
ground, this intersection of public health and business. And also this huge
megatrend in terms of the development of
machine learning technologies that we're going to go into. And we believe that there is
a common focus of the schools, of the broader community
that's represented here, all of you in making sure
we can get what works in health to the
greatest number of people and ensure a fair and
sustainable distribution of those gains. That brings us all together. And we're going to discuss
one technology that is rapidly growing,
emerging, I think you could argue it's
emerged technology in this area of machine
learning and AI. Just before we go
into that discussion, I just have a few kind of
set up things in addition to the ground rules that
Boris identified there.
So the first, of course,
is that we appreciated this is a huge field and
the people represented here, and you're going
to hear from them in person about the
way they're doing go across the whole spectrum. But each one of the people
that are represented here is our storied and
recognized leaders in this field and their field
of expertise itself is huge. And we can have an entire
week or month of events just in each one of those areas. But we're not going to do that. We're going to give them
a few minutes and ask. And so it's going to
pose a lot of questions, as well as hopefully
answer some for you. And that's intentional. The second thing is
that there's a ton of– there is some
technical foundation that would make this
material more interesting and would challenge more. And unfortunately,
we're not going to have the time to go into
those things on this call. Which we apologize for,
but practically speaking there was no real way
of achieving that.
Except to say, we can
point you in the direction either at the end
or in the follow up that will be sharing
with everyone who's registered on resources that you
can look into to find out more. And the last one in terms
of caveats or considerations before we go
straight into it, is that we also accept that
it's very early innings. These whatever you hear
kind of in the hype machine. We are much closer
to the start than we are to terminator
or benevolent robots taking over all of our jobs. And making kind of
humankind redundant. And there is a ton
of work in R&D, there's a ton of
work in basic science R&D, in business models, in
technology infrastructure.
And we're right in the
middle of all of that now, which makes it so interesting. But we accept, we're going
to refrain on this panel from talking about
the challenges as well as the opportunities. And we're not going
to overplay how mature these technologies are. But give you a sense
of hope and belief that this is a really
foundational set of trends. And all of you,
whichever aspects of health care or business
you're working at the moment, there's some way that
you can contribute here in health care AI. So I just like to kind
of open the panel up. We're going to go
through a few questions, and then we'll have
some time for you guys to post some questions as well.
When it does come to
posing of those questions, if you could basically turn your
camera on and ask the questions yourself, and we'll call
you from the questions that are put forward in Zoom. Cool. OK. So the first one and
Ben if you don't mind, I'll start with you. Is if you could just share
with our 700 new friends, what is that you're working on? And also if there was kind
of, what your path into health care AI was and if there was
a particular light bulb moment that you'd be willing to share. And I believe you
have a slide as well. That's right, yeah. Thank you Justin and thanks
everyone, for being here. So Ben Zeskind
educational background, I did a PhD at MIT in
bioengineering, and then an MBA at HBS. So hello to all my fellow
alum who are tuned in. And right after
graduating from HBS I started engineering, which has
grown and had quite a journey, and is now a publicly
traded company. So one of the fun
parts about being the CEO of a publicly
traded company is that my lawyer makes
me show a slide like this before speaking publicly.
So I'll just remind
you all that I'll be making forward-looking
statements today. So please see our
public disclosures for more information. Now that we've gotten
that out of the way, just to kind of give you a little,
a bit about my background. I won't read the whole slide. But the company is
really 13 years old. And our goal was always
to use computation to figure out what was happening
in patients who were responding well to cancer therapy? So that we can make that
happen in more people. So that we could
achieve broad activity. And we really took a
translational bioinformatics approach. Where we analyze transcriptome
and genomic data. And really build the company
by partnering with pharma, where we had the opportunity to
work on many great medicines, including ibrutinib, ipilimumab,
daratumumab, and others.
And as we did that
we built a platform. And really, one
of the nice things about having grown
the company this way is that every project
that we worked on had a pharma statistician, and
an experimentalist who really encouraged us to
generate results that passed statistical muster
and could be experimentally validated. And that was really the crucible
in which our platform grew. The platform has
several components that let us do novel
biology, novel chemistry, and translational planning. And particularly AI has
been quite relevant to us on the novel
chemistry side, where we have an AI based
drug screening technology that we developed. Well, I'll talk more
about that as it makes sense for the panel. But ultimately, all of
this computational work enabled us to develop a lead
product candidate in IMM 104 that we're really excited about. It's for Ras and Ras
mutant solid tumors, which are some of the
most common mutations among solid tumors. And the thing that's so unique
about our product candidate relative to others,
is in the animal data that we see, we have
very broad activity that's really independent of
what particular mutation is driving the tumor.
So that's exciting. And it's really the
computation and the platform that enabled us
to get these kind of counterintuitive insights,
that ultimately led to 104. And we have a broader
pipeline behind that. But I'll pause
here, I think that's a sufficient introduction. Thank you so much Ben. Yeah, very much look
forward to diving into that. OK cool. Next up, Heather Mattie. Thanks, Trishan. Hi, everyone. Thanks for joining. I'm Heather Mattie. I am a lecturer on biostatistics
in the biostatistics department at the Harvard Chan School. I'm also the co-director
of the health data science master's program that
we launched in 2017, at the school inside of the
department of biostatistics. I got my PhD in biostatistics
from the Harvard Chan School so I've been here a while. I do a lot of research in
kind of the intersection of network science,
data science, and health disparities, research. Algorithmic bias is a huge
part of my research now. And I teach anywhere between
four to five data science and biostatistics
courses a year. Mainly aimed at the health
data science master's students but also
clinicians looking to get a little bit of a
foundation in data science and coding.
In terms of my
light bulb moment. I don't think
there was a moment, I think it's been at least two
or three years in the making. Where during my dissertation
research in network science, I learned about machine
learning and utilized machine learning in that. And then I got to work with
clinicians like Leo and Trishan on the call, and realized that
somebody with a very technical background had to learn how
to communicate with those without a technical background. And so communication
between the two and bridging kind of that
gap and working together, really sparked my interest in
kind of clinical AI and health care. And then seeing
the inequities that were perpetuated by machine
learning in health care really made me kind of focus on
algorithmic bias and fairness.
And I think that's it. That's wonderful Heather. I think you can
all, well, hopefully we'll have time to
reflect on the end. There's some general themes
kind of coming out here already from what Ben and
Heather have said. And we're going to revisit
those during the conversation. But let's go forward. So Javier, Javier
Tordable from Google. Welcome. Hi, everyone. And thank you for
inviting me Trishan. So my name is Javier Tordable. I am a technical
director at Google.
I'm part of the city
office in Google Cloud, we're a small team of
senior technologists and former CEOs
that report directly to Thomas Kurian, the
CEO of Google Cloud. And our mission is to
support strategic partners and customers. So I meet with executives to
understand that technology and organizational goals, listen
to feedback about our product, and then share a little
bit about our strategy.
And especially
find opportunities to collaborate and co-innovate. I've been involved
in health care and particularly in
AI for health care, for about two years. I think the light
bulb moment for me was the original
announcement of alcohol. In the sense that it showed
how a purely technical progress or development coming
from the world of AI could make a difference, within
health and life sciences. Coming from people who had
very similar backgrounds to me, from the software engineering
side of the world, instead of clinicians
or people that had a long history in pharma. And previously I was working
in media entertainment. And before that I
spent another 10 years, 14 years total at Google,
building all sorts of systems. Mostly on the big data and AI
for a wide variety of things. Cool. Thanks, Javier. Welcome, of course. I'm very much looking
forward to the discussion.
So last but never least,
good friend and collaborator Leo Celi. Hi, good evening, good
morning, good night. I represent two big
communities actually. The first one is, am
I the critical data? And the other community is
the plus digital health. Which is a journal that
just launched this month. And there's a lot of
intersection between the two communities. The goal of is high
learning, which is learning from each other
and learning together. This is done across
expertise across countries, across lived experiences. Our group at MIT
is behind a number of publicly available data sets
from electronic health records. There's Mimic, there's EICU,
RCRG, mimic chest X-ray. As regards what was my path
into health care and AI? I think it's the realization
as a practicing clinician that the medical
knowledge system that informs the way we deliver
care is flawed in so many ways.
Research is coming from
a few rich countries and clearly not inclusive. And as Heather pointed out
earlier, the issue of health disparities. I think to a certain extent,
the schools of public health and the business schools
have created the problem, have perpetuated the problem. And it's time for
us to own this, because I don't think
we can move forward unless we say that we are
a part of this problem and somehow we need to
change the way we're addressing these issues. Cool. OK. Yeah, that spicy way to
finish their, I love it. So I think if we just
before we kind of go into the next question. I think each one of
the speakers there is exactly as we kind of said
it would be at the beginning. They're in very different
areas if you look technically. But there's some
core themes I believe that apply to all of the
areas that they're brought up. So Ben has brought
up the importance of enterprise
partnerships in kind of getting these
computational learnings, into actually
creating like business and organizational results and
getting in front of patients.
So working with the incumbents
as more data science and machine learning field
companies and organizations. Heather's brought up just the
necessity of multidisciplinary. And Ben brought this up as well. The necessity of
multidisciplinary teams in this area. And therefore exactly
what that describes people from very different backgrounds
to the very different languages, learning to
work together to make these technologies work. Because it needs that
entire spectrum of skills. Which is why public health
community we believe these are public health
issues, of course. And Javier talked
about essentially productizing the future. And I think also
implied, I think it's a very important
constituency here. The huge academic
and open source contributions of the large tech
companies in the development of this field. But also how one
of those things, there are foundational
advances that we're all kind of standing on
top of at the moment.
And the whole kind of
machine learning movement goes back to what? Goes back a long way. But this kind of era of it,
with the start of deep learning and image recognition. And then most recently,
foundationally with what Javier described as alpha-fold. Which we'll go into a little
bit more with Ben also. And then Leo, lastly in
terms of actually realizing these gains at the
bedside, how early we are. And the importance
of even once that's done given the fragility
of these approaches, how open data and open
data collaborations are going to be a
core part of getting these technologies to scale. So with that in mind,
I think this sets up kind of our next question.
Well, obviously,
not by coincidence. Which is that
yeah, there's a lot of kind of reasons to be
hopeful in this field. And there's also
realistically huge concerns around the actual real
clinical results delivered at the bedside
outside of studies. Or in the research capacity,
in the clinical capacity as research capacity. But then also concerns
regarding privacy, and bias, and sustainability that Heather
and Leo brought up earlier. So I guess I would
like to again, start with Ben if
you don't mind.
What excites you most
about this field in 2022? And if you have any either
from specifically engineering or more generally in the
field of drug discovery, if you would hazard to come
look into your crystal ball and tell us a little bit
about, any predictions of where you see this area going
over the next couple of years. Sure, happy to. And look, I think one
of the things that's most fundamentally exciting
about AI broadly defined in health care, is
really the ability to provide counterintuitive
results that are rooted in data. Something that you kind of
wouldn't have thought of. And for us in the
area that we work in, which is developing
cancer medicines, that's really important. Because for all the medical
progress that there's been, it's still the case
that millions of people die of cancer every year. And if we keep doing
the intuitive thing we're going to keep
getting the same results.
So that's why counterintuitive
insights are so important. But you can't just sort of
come up with random ideas, willy-nilly. They need to be rooted in data. And so I think that's
kind of the beauty of AI and computation more generally
in the space that I'm in. And that's what
excites us so much, is the idea that you can come
up with a very counterintuitive insight. Whether it's in our case, the
biological mechanisms that underlie our lead program,
which is sort of a different way to target cancer cells
while sparing healthy cells. Whether it's the novel chemistry
that we've come up with, using our AI platform for some
of the earlier stage programs. But in each case, it was
something counterintuitive. We wouldn't have come up with
by the traditional methods. But that ultimately
proved robust because it was rooted in data. And I think that sort of
ultimately proving robust is critical.
Because at the end
of the day, you have to take these
predictions and validate them in the real world. In our case, validate
them experimentally. And the more rooted in
data the predictions are, the more likely they are
to validate experimentally. So for me at a
high level, that's what's so exciting
about this field. Is just the ability to generate
counterintuitive insights that can really help to
change the status quo.
And I think Ben, I mean it's
a very, very important point that you bring up. Which is that I mean, of course,
that's the scientific method. And essentially,
we have a new set of tools in the toolbox
around the scientific method. Super, super exciting. OK, Heather. So if you wouldn't mind
if I moved on to you. And I'll ask you essentially
a spin on the question if you don't mind. So the folks that
are doing this work, that Ben is hiring
within Immuneering, you are training those folks. And I think you introduced
what you're doing before. But I think we
should kind of make even more of a point on it.
It was the first
dedicated health data science mastered program in
a school of public health anywhere. And so I guess from
that point of view, in terms of training
the current generation and the next generation
of data scientists, they're doing this kind
of work using these tools within the spectrum of
organizations represented here in other ways. Yeah, what is it
that most excites you about AI, even from a
technical point of view, or from a consideration
of some of these much broader, structural
issues, regarding bias and inclusion in the field? Yeah. So I'm excited
about a few things. But what I think
most excited for is more work in
defining, detecting, and mitigating algorithmic
bias, and making algorithms more fair. I also love that we have kind
of from our health data science program, more have sprung up
around all over the country. And schools of public health
but also other institutions. And so we're kind of building
this new kind of workforce that hopefully can communicate with
clinicians and health care professionals, in a
really cohesive way.
So that these AI
breakthroughs that we have, if we can get them into
use at the bedside, they are going to be
kind of seamlessly worked into the workflow there. And hopefully, my hope in the
next couple of years or so, is to figure out how to kind
of alleviate clinician fatigue. Get them away from the screen
by having AI kind of look at a lot of data for
them, so they don't have to sit through it themselves. And kind of organize
all of that for them. So they can spend more time
with the patient, and less with interacting with
the screen, and things.
Excellent. Good stuff. Thank you very much. Javier, I guess
the same question. So I guess maybe let's
just kind of start back where your last
answer finished actually, if you don't mind. So if we start with
like alpha-fold, these kind of core R&D
investments product, housing them, getting some
of these things to scale. Yeah, what are the things that
most excite you in those areas? Well, there's a long list
of technical achievements and where we see the
field progressing, I think we're all interested. But I want to focus
on one specific idea. In the world of machine
learning there's always been this tension between
understanding at a deep level how methods work,
or how systems work. And this notion that if
a Black box just works, then doesn't really matter
exactly what it does. If you have a machine
learning model that can find the perfect
drug for a specific patient, what does it matter
that you don't really understand how it works? And that has been a
problem traditionally, I think for the
application of AI health.
Because doctors
and scientists they want to understand
how things work. It's really hard to
rely on a system that is going to have an
impact on somebody's life. And I think one of the things
that happened with alpha-fold, was not just the
technical achievements of being able to
predict protein folding. But also this
acceptance that if we had a system that
works and works just as well as experimental
methods, then we can use it. And it's OK to use
these kinds of systems. So I think we're going to see a
broad usage of machine learning and in scientific
problems, even in cases in which we don't have
100% understanding of how these methods work. So I think that's a
very controversial idea. But at some point we may be
using critical decision support systems for example, that
they're not based on rules. They're based on
machine learning methods for which we
may not understand exactly how they work.
But they may work significantly
better than a human, they make less mistakes,
they may not get tired and so on and so forth. So I think that mindset
shift is really interesting. Good. Excellent. Thank you so much. I mean, thank you so much
for bringing that up. Leo I've got a number
of thoughts here, but I'd like to
hear yours first. I think let's go into a bit
of a discussion on this area.
OK so just to kind of set it up. I mean, Javier has
brought up, yeah, these machine learning
models of Black boxes. Not in terms of it's
impossible to see what's going on
inside, which I guess is what the metaphor suggests. It's more that it's
imperfectly transparent, but it's incredibly
complex, so it's kind of not human intelligible. So how does some input into
the model create the output? Yes, that's perfectly described,
but it's not intelligible. Which that may be acceptable in
some situations and it may not be acceptable in others.
And I guess Leo what I'd
like to ask you is, the areas where there's a higher
level of proof required. I.e. where we know
these approaches are probabilistic so
therefore they can fail. And where there's a
human cost to that. I.e. like what are the challenges
of applying machine learning methods in clinical medicine? And yeah, I'd love to know what
your thoughts are in that area. Yeah, before I
answer that question I prepared some responses
to earlier questions. And it's expanding on
what Heather mentioned, as what's most exciting about
AI in health care in 2022. That this is happening
at a time when there is heightened awareness
of diversity, equity, and inclusion.
If this happened
even a decade ago, I am certain that we
would have failed. Which brings me to the
question about predictions for 2022 and 2023. I hate to be the
doomsayer in the group. But I think that
we will not make any significant
advances in AI in health care for a number of reasons. The first is the use
of real world data to develop and
validate algorithms. As you know, real world data
is derived with disparities. And if we're using
accuracy to assess them, then there is no hope for us
to address these disparities. And then the other
reason is that those who build algorithms,
that's us, do not represent the perspectives of
those who are most vulnerable.
We cannot chart the course
of health care with the same mental models that created the
problems in the first place. And then moving on to your
question about explainability. To me the most important
aspect of explainability is making sure that the
algorithms are making decisions, not based
on sensitive attributes such as race, ethnicity, or
gender, or other demographics. I don't need to know exactly
how those decisions are made. More than half of
the medications that I prescribe,
we have no clue how they work but we still prescribe
them, because they work. But from an
algorithm what I need to know is that those
features that are not relevant to the prediction
or classification, are not being used in the
same way that humans do. And I would like to
mention it again, the real world data
that we're using is full of subjectivity on
the part of the providers. And if we're going to
predict the outcome and based that on
the ground truth, then we're going
to continue to have poorer outcomes in the
most vulnerable population.
And I think this is the
reason why we're not going to make significant headway. But maybe we're going to
pivot in 2022 and 2023 on how we're building
and assessing algorithms. But no major movements
in terms of improvement in population health. OK. So that's what my
follow up question was going to be actually. I think we should qualify
these segments a little bit. I understand where
you're coming from. I think I just want to make
sure that everyone else does as well.
So what you're saying
is not that there won't be like scientific
advances or the field won't move forward. I think what you're or that
actually there will be– I guess I'm asking
you, is it that you're saying that the impact
that you have in mind is about addressing health
inequalities or inequities? And that there won't be
any significant movement in the short term in terms of
AI improving those disparities. Because essentially it's going
into a system that structurally biased being trained
on data, that is created by that system
and broader society. And being developed by the
people who are currently administering the
current set up. Is that roughly
what you're saying? Yes, the purpose of
scientific advancement is to improve population health.
And the cohort of the
people who carry the biggest burden of disease,
I don't think it's going to be impacted by AI. And for that reason, the
scientific advancements are irrelevant, are useless. In the short run. So my question to you would
be, what about five years out or 10 years out? I mean, if we were talking
about cancer because of the advancements
in drugs in the 1990s, there's a lot of optimism too. But we could say that
there are advances in cancer in rich countries. There's minimal advances in
cancer in most of the world. And can we say that
there has been a success? I would disagree. So to me the purpose
of all this health care technology is to as I said,
improve everyone's health, not the health of a small
segment of the population. OK. All right. So I mean, this
is very much from the macroscopic global
point of view, and very much thinking through a long term. OK, understood. So as we see folks, there's a
large spectrum of opportunities and large spectrum
of challenges here. And I think our intention
is to kind of give you a flavor of what
these issues are.
And I think all of
you out there will have some opinions on
which side of this debate you want to go on and
that's totally fine. I think all of these– it takes everyone. And I think as we
kind of go back to what we said before, the
reason we believe that there's a common focus here
is that we're all trying to focus on
that long range goal that Leo has in mind. And there's different methods
towards getting there. And it's a really
hard problem to solve. And it's not a problem that's– I think, one of
the key issues here that I would like to kind
of throw into the mix, is that this isn't a specific
issue of machine learning.
That sets up the layered thought
you could apply to literally any health technology. And kind of economic
system more broadly. So these are large scale,
persistent, structural issues in human societies. And we should realize
that and qualify thinking about machine learning. These aren't uniquely
machine learning problems but they are problems
that we should be cognizant of when we're
developing any new technology. Such as machine learning. So I thought I'd
give the rest of you, if you had any kind of follow
up to what Leo was saying.
We have a few minutes. Or if not, I might just go
forward to the next question. Because I think this could
be an interesting one to engage the broader
audience around. So if anyone's got
something like burning they want to say now then
please kind of let me know and I'll call on you. Or if not we'll move
on to, Javier please.
Yeah, just to follow
up on that maybe from a slightly
different perspective. I think there is a real problem
as well, from the perspective of a technology provider. For a while, some of us have
been trying to democratize AI and to make it available. I completely agree that this
is going to be a problem. Where we have a situation where
a few technology companies are well-funded,
institutions have access to this type of
technology, and can spend the budgets, and
their compute power, and have the budgets
to hire the people that can do these kinds of things. Whereas the technology is just
not available to most players. I don't have a solution
for it [LAUGHS] but I think this is also a
problem from the perspective of big tech. Yep, got it. Sorry I'm just being
given time warning. So I was just a little
bit distracted there. OK so let's go into that. I hope there'll be some
interesting questions from the audience and I can
curate them and ask them.
But before we get there we
just got a few minutes left. So I think just something to
leave the broad audience with. What is it that you
know now, kind of been in the trenches, so to
speak with this set of problems and seeing this field
hugely blossom as well. That you wish you'd known
back when you started. Or that you didn't know
when you were started.
And maybe someone who's new
and coming into this field, it wouldn't necessarily be
obvious just from the things that they hear in
the media and read. So yeah, what is it that you've
learned about health care AI, that you wish you'd known
when you started off on this journey? So maybe we'll start
off Javier with you. I would say for me, I wish
I had known or understood how hard it is to drive
organizational changes. I guess like you Trishan. I see the technology
and I think, well, this is a problem that
can be solved with computers. But then you try to apply these
kinds of things in real life and it takes forever. And people are not
excited about change, and it's very hard
to modify systems, and to adapt incentives.
But I wish I had known how hard
it was to deal with change. I mean, and is it fair to say
like especially in health care. I mean, it's difficult
in the industry. I mean, health care must
be the only industry where people still use faxes. [LAUGHTER] Yeah, that's a waste. That's the iconic example
of poor technical progress. OK, understood. Ben. Yeah, I mean, I think for me
one of the important things to really come to appreciate
is just how much hype there is out there right now about AI. And to really just separate
that from the reality on the ground,
which is there are some very interesting
and important advances. And I think people need to
understand the difference. And look, artificial
intelligence is not a new concept. 20 years ago as an
undergrad at MIT I took an artificial
intelligence class.
My lab partner joked that
I was the artificial, she was the intelligent. But just to show you that it's
been around for a long time. And it wasn't a new field
20 years ago either. And to be sure, there have
been some incredible advances driven by computing power,
image processing and the like. But I think part of why
there's such a focus on this is because of the progress. But part of it is also
just kind of the hype that's been generated. And there's different
reasons for that. And I think there's been
certain companies that have let the hype get way,
way, way ahead of the reality. I won't name names
but I think that's why it's really important to ground
the outcomes of AI in truth. And in the field that
I work in, the truth is in the experiments. The truth is in if you
can make a prediction and then go test it
experimentally and see that it works. So I think that there's
such incredible promise for AI and computation. And I think the people that
sort of take the hype too far, they do it a disservice.
Because then people
start thinking, humans are going to be replaced
and all of these things. And reality is very useful
in certain situations. But I think that's to really
appreciate the true areas where it can be helpful. And I think it's
important to distinguish between that and
kind of the hype that gets out of control sometimes. Yeah, it's a very
important point. I mean, I guess the
kind of obvious counter is that yes, but
then also like we have utterly superhuman
things being achieved. We've got generative
speech from machines that humans, most of
us can't distinguish. We have superhuman performance
in certain diagnostic tasks, in protein folding, It's like
core, scientific challenge in game playing. And I think they're probably
also is some value of I guess, painting the
impossible future as a way of it's
inspiring people to move directionally there. But then exactly as
you say, it kind of makes it seem like, well
that's an inevitability. But there's a ton of technical
challenges and realities to get there.
And I think if people
portray that exciting future as an exciting future,
then it's totally fine, and it's a good way to sort
of show society direction. But when people portray
something as being here, now, that's not
really here, now. I think that's where it starts
to get a little bit confusing for people. But yeah, I mean certainly
there have been advances. I think again, just bring
it back to the field that I work on. I don't think there have been
such dramatic contributions yet from AI. Because for us, the North
Star is always the patients. And millions of people
are still dying of cancer. So there's still a lot
that needs to be done. But I think there
have been as you say, really exciting progress
from AI in a number of fields.
So as long as people
portray that accurately, I think it's all totally good. Excellent. Thank you sir. OK next up, Heather. I was thinking the same thing. How difficult it is to
actually implement anything that we come up with. I've dipped my toes into the
industry field with nutrition, I won't frame. But I'm mostly coming from
the academia standpoint. Where we get research grants and
we work on these what we think are amazing algorithms
or breakthroughs in machine learning. And then we can't
implement them. Or it takes years to even
try to get it to a stage where it can be implemented.
There's exceptions to that rule. Dana Farber is actually
really good at that, because they're kind
of all in-house. But yeah, I did not know
how difficult it would be going into this to do that. And then I also think, and I'll
be a broken record because it's what I'm most passionate about. But how impactful
algorithms can be, if they are biased,
and used, and are implemented when
they shouldn't be.
How detrimental that can be. I didn't think about
it when I jumped in. That's interesting. So like a technical training,
yeah, that's interesting. Cool. Heather, we actually
have some questions from the audience around this. So I'll go to Leo and then
I'll come back to that if you don't mind. So Leo, yeah, over to you.
I guess actually if you wouldn't
mind in your answer Leo. I think one of the things
broadly that people don't understand, that I think
people understand very well on the technical side
of the community is just the amazing
contribution of mimic. And what that is and the scale
of it, and what it's started. So if you wouldn't mind, I
know you're kind of probably too humble to go into
this directly yourself. But just if you could also
talk about that open data movement and some of
the impacts of that in developing machine learning
capacity around the world.
Yeah, sure. So as I mentioned, we
have been in the business of making available electronic
health record data sets. And we were hoping that by
2022, that there will be more of us who are in this business. But unfortunately,
that hasn't happened. But I am optimistic that
we're heading that way with new initiatives
from the NIH, new requirements from
funding organizations. That we will not give you money
unless you share your data set and that this is going to become
a resource for the research community.
And not just focus on some
vertical clinical question. But I wanted to answer your
question of what did I not appreciate when I started this. And I think that
is what we refer to as digital
determinants of health, which is kind of different from
social determinants of health. And I think this is a very
important topic for this group.
So I wanted to
expand a little bit. And there are really
three categories of digital
determinants of health. The first one pertains to
digital health tools that leave behind certain populations. So those without adequate
connectivity, those with cognitive dysfunction
such as those with dementia or developmentally delayed,
those with other disabilities, those with mental
health issues, those with behavioral
disorders, the hearing impaired, the visually impaired. Especially with Telehealth
becoming a norm and standard now. And then the second
category pertains to digital health tools that
were designed around the White population. Such as those that leverage
infrared technology. So we've heard about
pulse asymmetries that seem to be not as accurate
in the non-white individuals. Or technology that
doesn't work for those who do not have a normal habitus.
So some sensors
don't really work for the morbidly
obese individuals, and how many diagnoses
are we going to be missing because the EC are not capturing
the electrical activity of the heart for those
who are morbidly obese. And finally, the third category
of digital determinants of health is coming
from the AI community. Algorithms that perpetuate
or magnify health disparities because they were
trained on real world data. So I wish that I had
known this earlier, I wouldn't have wasted well,
I wasn't wasting a lot of time in just understanding
the technology. But having that mindset
from the very beginning would really have helped
me to make research more efficient and more effective. Interesting. All right. So we have a number
of questions. And they're great questions. I think I'm just going to have
to pick a few out I'm afraid. So to those that I don't pick
out, I apologize in advance.
I'm going to pick the ones that
I think are kind of broadly go across what the different
people can talk about. But I'd like to thank you
all for your questions. And I think as you
can see here, just by the spectrum of questions
if you want to have a look, there's a lot we could go
into further with this. And we want to do a
series of these events, and we would invite
you, and hope that you can join us for that. So the first one that I'd
like to go into, sorry, I just lost the thread
of where it's at, is a question from Lawrence. And this is around
the bias issue, around the Black box issue,
those kind of things. So explainability
excuse me, so how do you determine if a
counterintuitive finding is a reflection of reality
and not a result of algorithmic or other biases? Such biases their
magnitude and the impact may differ between
for example, use of AI approach in protein
folding or image analysis.
Compared for example
in use of real world data from electronic health
records and other health records. So an algorithm that
is trained on data from the real world, that
produces a result that one doesn't expect. Is that result and insight? Is that result a poor
technical performance? Is it algorithmic bias? That's a very interesting
conceptual question there. I guess maybe, Heather, would
you mind if I start with you? I don't want to put you
on the spot too much but we can share
it around as well. I was hoping you wouldn't. I thought Ben would– [LAUGHS] Yeah. I'll go next. Man this is yeah. It's a hard question.
So Lawrence we want to
thank you for the question. It's a hard one, it doesn't
have a perfect answer. But it is a very important
structural question. Yeah, so I don't have a
straightforward answer for this. I'm just kind of thinking
off the top of my head. But if your data is biased
and you're putting it into the model, it's going to
create biased outcomes usually.
I know Leo is going to have
more to say about this, he's raising his hand. [LAUGHS] So separating if you're
using real world data, that is already biased. You're just going
to kind of usually perpetuate that
through the model and then have biased outcomes. So I think that will
be algorithmic bias. Is it a real insight? Probably. Go back to kind of
the judicial system data set about parole,
and everything, and sentence lengths,
and racist data led to racist kind of outcomes. And so it does reflect the world
that's being used to train it. So there's a more
eloquent way to say that. [INTERPOSING VOICES] So you wouldn't mind in
the order of these things. Ben, I suspect you've
got some thoughts, would you mind if I
called on you next? Sure, yeah. And I'll just answer it quickly
and then we can go to Leo. Because I did mention
counterintuitive in my introduction. And thanks Lawrence
for the question.
Look, at the molecular
level it's easy. So the kind of counterintuitive
findings we talk about are sort of the mechanisms of
how cancer cells are working relative to healthy cells. And so the beauty of that is
you can test it experimentally very easily. And that's a very quick way
to get it at ground truth. And evaluate the
counterintuitive findings that we talk about in our
work to develop medicines. Obviously, it's a much
more complex issue for the kind of population data
that Heather and Leo focus on. So maybe I can hand it over– [INTERPOSING VOICES] So Ben, a critical point here
is that in kind of society, in there. So if you look at
health outcomes, they're incredibly
causally dense. And some of those things are
biological or biochemical, some of those things are
social factor, some of them are environment. Some of them are effectively
roll of the dice charts. Things that pandemics come
up, or you have earthquakes, or these kind of things. So then figuring out
whether an inference is the result of biased
data or it's picked up on other things in the
distribution of possible events that we're not seeing,
is a really hard problem.
And like in biology,
according to our mechanistic
understanding of it, you have some ground truth that
you can compare against and use the scientific method. But it's very hard in population
medicine or population health. That's been a very good point. OK, cool. Leo then Javier,
if you don't mind. So Leo. Yeah, no but my term for this
counterintuitive findings is actually spurious
associations. And spurious associations
are almost always a result of sampling
selection bias. And I'm going to plead
to those people who are creating and validating
algorithms out there. That before you even do any
exploratory data analysis, you need to figure out how
people got into your data sets.
So before you even go into
mimic, you need to find out are there people who don't
even reach the ICU because they die at home? Are there people who
are systematically excluded from being admitted
to the ICU for so many reasons? And you need to know the
baseline disparities. So don't just go
in there and start dumping all the covariates into
some R-encoder and transformer. You need to understand
where are the disparities and what are the
drivers of disparities? So knowing that disparities
exist is not enough. You need to tease out
what are the factors that contribute to the disparities.
Is it access to health care? Is it subjectivity on the
part of the providers? Or is it something else? Is it a technology that was
designed around the White individual? So maybe the bias
is because it's not picking up the right
oxygen saturation of a patient of color. So having this understanding
before you even do any histograms,
I think is required. You have to do your due
diligence or your homework. And not just start
creating the models. Because I think you're going to
be wasting so much time if you just dive deep into the data. Yeah, interesting. So I think you've also
kind of reinforced some things that were brought
up by Ben, and Javier, and Heather earlier. It's like the importance
of working in teams. These multidisciplinary
teams who have knowledge of the
clinical process, as well as like how to build and
test the algorithms. But then equally, also the
importance of understanding of the context. And the context defined
really broadly I mean, you don't hear
people who typically talk about open data talking
about the kind of things that you're talking about, well
certainly not in my experience.
So that's a very critical point. So to kind of I guess,
you would broadly say, and this is an established
idea in machine learning. That you just need to know about
the data generating process. I mean, that's effectively
what you're saying. But in this case, it's a
very complex data generating process that isn't
just technically within the hospital,
it's about who's even in hospital in the first place? And how the hospital operates. Which is very important point. Thank you, Javier. Yeah, I would say just
a couple of ideas. I think it's
important to remember that a lot of these methods
are probabilistic in nature. So if something is
99% accurate, it means that in 1% of
the cases effects. So nothing related to
machine learning is perfect. I think the best that
we can do is likely as said, to make sure that
the process used to build those systems it's appropriate. That there's enough
data with enough variety for the population,
that is going to affect and so on and so forth.
And maybe when we read a
paper about an AI system that is built for
a specific problem, we should rate it as if it
was a scientific paper that shows the activity of
a drug or whatever. And we'll read the
methods section and make sure that the way
that the data was obtained, is the same way that
you would select people for a clinical trial or
something along those lines.
Yeah very important. I think there's a ton on
this area actually, sorry I won't go into it. Because we're going to
go down the rabbit hole, we probably need to
wrap up a little bit. So I'm going to try and
get one more question in from the audience. Maybe it'll requires
a little bit more of a short form response. I've been trying to pick a
question that lends itself to a short form response. And it's a really good question. But these are kind of not
going to blend themselves to– OK. Let's go to from
anonymous attendee. Where do you see the readiness
and adoption on machine learning AI models by clinicians
and clinical practices? So I think really let
me generalize that one. So we non-tech health
care organizations, I think we could
probably put maybe– I'm afraid to put
large pharma in there. But let's do it for
the sake of argument.
Large pharma, payers,
providers, national governments. Where are they in your
opinion on actually using these technologies as part of
the mainstream of what they do? So Javier maybe
in reverse order. I would say years
away in general. Even technology that has been
out there for a long time, it's really hard to find. Using voice to communicate
with your health plan. To ask details about
an insurance claim. I think it will still
take a few years until we see this industry. Yeah, obviously
it's going to mean that's close to home
as the kind of work you and I do together there. But it's– Ben. Yeah, I mean I think in pharma
for certain specific problems I'd say it's already
starting to be used.
And particularly in areas where
you can validate the result. Where you can run an
experiment to test it. I can't speak to
more the applications in clinical practice. But obviously for
all the challenges we've discussed today, that's
probably a little further out. Yeah, thanks so much, Ben. Leo. I agree with Ben and Javier. I think as long
as the people who are in the pay scale
of making the decisions don't understand
what the issues are, we're not going to
see any headway. We're not going to
move the needle. They need to understand
what are the limitations, they need to understand
where the problems are. Yeah. And actually, sorry,
just quick shameless plug before they cut us off. We have, of course, all
of us together here, actually Ben would love
it if you could join us this year in fact. Applied AI for health care,
where we try and bridge that gap. People running health
care organizations, trying to make decisions
for you to be literate, in what these issues are so you
can make more informed ones. Heather, you have
the final word. Boy, I'm more of the data
scientist behind the computer person.
But yeah, I would absolutely
agree with everyone. It's years away, we have
to get higher level people to get on board, and really
understand what's going on. And that's going to need
better communication, better results, and better
integration tactics I think. So a ways away. I do think there's like I
mentioned before, Dana Farber is doing a really great job. I'm sure there's other
places that are doing really well in this area.
But I haven't heard any. Excellent. Well, Heather that
sets it up perfectly. So firstly I'd like to thank the
respective Alumni Association who helped us organize
this, the HBS Alumni Association from whom the
seed of this idea comes from. The health care alumni,
I'd like to thank them, I'd like to thank you. 700 new friends wherever
you might be in the world, be it in the morning,
the afternoon, evening, or in the night
time, especially those of you in the night time.
Of course, and we would like
to do subsequent events. Basically, probably what
I'm going to propose is a deep dive in different
areas of health care AI, because it's incredibly broad. Each one of these panelists
represents a huge field of study. And I think we could do
sessions just on those. And these views from the field
gets an HBD alarm, HSBH alarm, and some senior
technical specialists. And then also probably
some on like other emerging technologies if
there's interest. So what is the metaverse? What's the relevance of it? Web theory, cryptocurrency,
these other technological movements that are much
further behind and much earlier than the machine learning. But we've gone over time,
we've overstayed our welcome. We'd like to on behalf
of the audience. We'd like to thank you, Javier,
thanks Ben, thanks Leo, thanks Heather.
Your wisdom was
incredibly well received. I've been taking notes I'm sure,
lots of people in the audience have. And then for you all
who spend your time to sit with us, thanks
for your time and thanks for taking part in that. It's what kind of makes
these live events special. Just one last thing
even though we're over, if people want to find
out about your work online or however else,
where should they go? Heather, would you mind
if I start with you? You can Google, Heather Mattie,
Harvard, [LAUGHS] my faculty.– [INTERPOSING VOICES] I do, yeah. Yeah, excellent. OK cool.
So that's that, Leo. That was shameless plugging
of Google, Heather. I'm just going to call you out. [LAUGHTER] I might
be critical data. We're trying to
update our website. Excellent. And more like I mean,
Google make a hell of a lot of useful products. But one of the Google Scholar,
you can see the research work. Most of which is open access
from these two as well. Ben. Immuneering.com. Excellent. Good stuff. And Javier. Anything LinkedIn, Twitter,
just Google my name, and you'll find a
way to contact me. Good stuff. All right. Well, thank you everyone. Thanks so much for your time. Thank you. Thanks everyone. Bye bye..
Free Coupon for Discounts on Pharmacy Medications
