Welcome back to Research Ethics Reimagined. I'm Catherine Batsford.
Dan McLean:And I'm Dan McLean.
Catherine Batsford:Alex John London is a K & L Gates Professor of Ethics and Computational Technologies at Carnegie Mellon University, where he directs the Center for Ethics and Policy. He has also served as a chief ethicist at the Block Center for Technology and Society at CMU. As an elected fellow of the Hastings Center, his work sits at the intersection of ethics, medicine, biotechnology, and artificial intelligence. Professor London's book, For the Philosophical Foundations of Research Ethics, was published by Oxford University Press in 2022. Professor London is the author of more than a 100 papers and book chapters in journals, including Science, JAMA, and The Lancet, and is the co editor of Ethical Issues in Modern Medicine, one of the most widely used textbooks in the field.
Catherine Batsford:His work in AI ethics centers on the structural obstacles to building safe and effective technologies and on creating mechanisms for social trust and accountability. As a member of the WHO expert group on ethics and governance for AI for health, Professor London helped write guidance, which published in 2024. He has served on two National Academy of Medicine committees, shaping frameworks for emerging health technologies. Throughout his career, Professor London has helped to shape key ethical guidelines for oversight of research with human participants. We are also proud to say Professor London serves as one of PRIM&R's board members.
Catherine Batsford:Thank you for joining us today to explore the elements of the rapidly evolving field of AI deployment and its use in medicine.
Alex John London, PhD:Thanks for having me. Once again, thanks
Dan McLean:for being here. We do like to start these conversations off exploring a little bit how you found yourself in this field. And I do realize that you've been a professor at CMU for more than twenty five years focusing on ethics, philosophy, and policy, but I was hoping you could shed some light on how you got there.
Alex John London, PhD:Yeah. I started out in graduate school at the University of Virginia at a time, probably in some ways, heyday of bioethics at the University of Virginia. John Arris was there was hired while I was a graduate student. And when I was went into graduate school, I didn't really have that much interest in bioethics and knew nothing about research ethics. I was doing ancient philosophy and and ethics.
Alex John London, PhD:And they said, would you wanna be John Eris' research assistant? And I was like, no. Why why would I wanna do that? I wanna do real philosophy. And they said, you you will pay you a thousand dollars more in your stipend.
Alex John London, PhD:And I was like, well, I I don't wanna do real philosophy that bad. So and but John is was an incredible teacher and an incredible man. And sitting in and and TA ing for his bioethics classes, I soon realized that was genuine philosophy, real issues, but that we're dealing with things that sort of were that mattered, were happening today. And then John loved Research Ethics, we would talk about real deep philosophical problems in research ethics. John's view was, yeah, those are real problems.
Alex John London, PhD:Don't know why, both practical and conceptual problems, I don't know why more people aren't more interested in them. And I certainly was interested in them. And so although I kept doing my traditional philosophy, I developed very early this sort of second parallel track in bioethics, but also in research ethics. And so I I've been doing it ever since.
Dan McLean:Your book, "For the Common Good: Philosophical Foundations of Research Ethics," which you published a few years back in 2022, why did you want to to write this book?
Alex John London, PhD:I mean, that's why did you write that book? No one reads books. So that's one of what one of my very good colleagues says. And I had said, yes. I I won't ever write a book like no one reads books.
Alex John London, PhD:But the the for me, part of the issue was I had many, many papers that were ostensibly about different topics in research ethics, but they were all guided by a single underlying set of ideas and real concerns about the field. And so I said, I I need to put them all together and bring out the underlying concerns that motivate them. And and to be honest with you, I think those concerns are more relevant than they were when I wrote the book given what's happening today.
Catherine Batsford:So what's the core argument that you want readers to walk away with and that could reflect on today?
Alex John London, PhD:Yeah. So the one of the the main ideas of the book is that the foundations of research ethics are are riven with these tensions. And at the time I was writing, when Alan Wertheimer had just published his book, and a lot of people in the field were very critical of that book because that book was very critical of the field. And part of what I I actually was one of the referees for Oxford University Press for that book, and I gave him lots and lots and lots of comments. So I'm actually the anonymous referee that he, you see, thanked in some of the footnotes.
Alex John London, PhD:But I thought that book was a challenge because it was taking pieces of research ethics that everyone accepts and repurposing them for for ends that a lot of people in research ethics didn't agree with. And to me, that just illustrated one of the vulnerabilities of the field, which is that it had become very concerned, very practical, very concerned with a particular set of institutions, a particular set of guidelines, a particular incarnation for a set of concerns without really thinking more deeply about what are the things that gave rise to that motivate, what are the larger goals that we're trying to accomplish with these structures? And so the book was basically to try to fill that gap. And the main gap being if you read the Belmont Report, justice is listed as one of the main principles. But if you read the way that it's explicated, it doesn't really say anything.
Alex John London, PhD:It says it could be this. It could be that. And so the goal of the book is to try to say, actually, I think justice is the foundational first principle. Because like the philosopher John Rawls said, justice is the first virtue of social institutions. And I'm treating research ethics as a very important social institution.
Alex John London, PhD:It has to be grounded in justice, connected to other social institutions. And then I unravel everything else from there. So I do try to provide a foundation for the things that people on IRBs would recognize as very familiar practices and principles, but provide that foundation in a larger context of an approach to justice.
Catherine Batsford:So when reviewing, almost consider justice first in some ways.
Alex John London, PhD:Yes. And and consider it though in a way so I think part of what has been a problem for research ethics is it's tried to think about justice in a way that is unmoored from social institutions. So it primarily thinks about justice as fairness in the distribution of benefits and burdens and subject selection. And so one of the one of the things I show in the book is that the other you don't need justice to care about those things, that you can have beneficence and respect for autonomy and and generate from those that concern about fairness in study selection. And in fact, then that's part of Wertheimer's critique draws on exactly this, on not having research ethics practices grounded in a set of institutional commitments and a broader theory theory of justice.
Alex John London, PhD:And I think why is this relevant to to to today? Well, changes that are happening, changes that are afoot where institutions and structures that sort of defined the field are gradually being eroded away and deemphasized, then we're at a crossroads where partly what we need to do is be rethinking what is it that what's baby and what's bathwater here? What were institutional structures and that had a purely bureaucratic aspect that we can get rid of? And what were the fundamental things that we're doing and how can we do those fundamental things in other structures or in other incarnations? And so I think that's the conversation that we have to have if we're going to go forward and eventually if there's going to be a rethinking and a rebuilding, then it's not clear to me that what we need to do is rebuild what we had before.
Alex John London, PhD:We need to be thinking deeply about what are we trying to do in order to make research ensure that there's social value to the research that gets done, to make sure that it's respectful of the people who participate in that research, that it adds value to the communities in which the research is carried out. And there might be other structures and other ways of doing that, especially since social value, there's a lot of people right now who would recognize there's a lot of research that gets done that doesn't have adequate social value. And it was very difficult to try to do that using traditional IRBs because the IRB structure really wasn't set up for that. And the IRB review happens after a lot of those decisions have already been made.
Catherine Batsford:So someone said intersection as research concepts are incorporated. Are they considering the bigger picture before they even get to the IRB?
Dan McLean:Yeah. And you mentioned social value just now, and that reminds me of part of what you wrote. For example, you wrote, Appeals to the common good as a ground for social imperative to carry out research are now rare and likely to be greeted with skepticism. Can you elaborate a little bit more on that?
Alex John London, PhD:Yeah, so I think if you go back historically into the era right before the common rule, there was significant debate about So you had proponents of research arguing for the importance of research precisely because of the progressive ideals, that it generates information that helps a lot of people, that will make the world a better place. It's just that some of those people also thought, and that's so important that it can ab it can justify abrogating the rights of study participants. And then you had people like Canzionas and and others who wanted to blunt the potential, the utilitarian potential for medical research to kind of run roughshod over the interests of study participants. And they did that partly by disconnecting research from that underlying social imperative. And so, like, the Jonas's essay was one of the first essays I encountered as a as a student.
Alex John London, PhD:And I thought there was something very brilliant about it and something deeply wrong about it. Because it it saves respect for individual interests at the expense of basically saying sickness and disease don't threaten the common good. They threaten individuals. Mhmm. And so carrying out medical research is a kind of optional individual project, that there's not a social imperative to do that.
Alex John London, PhD:And so part of the goal of the book is to say there is a social imperative to carry out research, but that social imperative doesn't justify running roughshod over the rights of study participants. I can show you how to reconcile those two things.
Dan McLean:And in the preface, you do mention COVID-19. You wrote this as the pandemic was ongoing, essentially. So how does that emphasize that line of thought?
Alex John London, PhD:There was so much about the pandemic that was surreal, but part of the special unreality for me was that I spent a lot of the pandemic finishing this manuscript. And and so watching mistakes play out that I was writing about in the book. Now to be fair, I had been at the WHO when one of the flu pandemics was declared. And at a meeting where we were talking about what is it that we need to do in order to make sure that medical research is prepared to hit the ground running in the face of an epidemic or a pandemic. And we had been thinking about that, and this isn't just me.
Alex John London, PhD:I mean, are lots of people who are involved in projects that are like this, and it seems like you can go back to a lot of these cases and look at what were written, the sort of retrospectives, lessons learned. We seem to never learn these lessons. And so as the pandemic was happening and the earliest research that was done, research in scare quotes with hydroxychloroquine, poor controls, poor study design, misleading signal that then led to a lot use of something that was ineffective and probably unsafe made it much more difficult to carry out research on things that were more promising and interventions that ultimately proved to be more effective. And part of that, the reason for that some of the researchers gave for very early poor quality studies was that they were concerned about the absence of equipoise in the face of a pandemic. And so exactly one of the things if there's a potentially fatal disease, how can you randomize?
Alex John London, PhD:That's one of the questions I've spent my life writing about and that others have spent a lot of their time writing about. If you think that there's just a fundamental conflict in medical research between the interest of the individual and the research enterprise, that the research enterprise is necessarily this kind of utilitarian thing that is inconsistent with the interest of study participants, then you have that attitude that we saw at the very beginning of the pandemic. If you think, no, the point of equipoise is to make sure that everybody is randomized to something that would be recommended for them by at least some reasonable minority of experts, then in the face of a pandemic where we don't what works, randomizing people to what we think is our best guess and including standard of care is not only more permissible, but it respects their interests, and it advances the goods of the community. So to me, what happened early in that pandemic was shooting ourselves in the foot and just the idea too that high that high quality research is optional in an emergency as though confounding will go, oh, you know what? It's an emergency, so that's okay.
Alex John London, PhD:There's no more confounding. The procedures that you need in order to get a clear signal about what works and what doesn't, they're needed even more they're all the more important in a context where you have a pandemic raging because the information that research generates is used by important social institutions to try to meet people's needs. And the slower we go and the more we fumble the ball, the less effective we are at meeting their needs and the more people who die. And so in that sense, it illustrated the idea why I think there is a social imperative to carry out this research because we rely on health institutions to safeguard our interests. And in the pandemic, everybody was saying, what do we need to do?
Alex John London, PhD:What should we do? And the idea was, well, it's a novel virus, so we're not sure. And so what I was arguing then was what we need to do is do the research as quickly as possible, high quality research so that we can answer those questions. And then the institutions that we rely on can help us preserve our well-being, and the people who get infected will know how best to care for them.
Catherine Batsford:There's a section in your book stating study participation is not a prisoner's dilemma, referencing the famous prisoner dilemma game theory. I found that argument that you are making there intriguing. Can you share more about what you mean, and can you explain the prisoner's dilemma and how you see it apply to potential participants in human subjects research?
Alex John London, PhD:Yeah. So several people have made the argument that research is the form of a prisoner's dilemma. So being at Carnegie Mellon, this is a place where decision and game theory woven into just about everything. Alan Wertheimer is actually one of the people who makes this argument in his book. And you can look up it's good to have visuals to understand the prisoner's dilemma.
Alex John London, PhD:But, basically, the idea is no matter what you're faced with in a prisoner's dilemma because it's it's the original story is two criminals are arrested, and they're being interviewed separately. And so if you admit to the crime, then you get a certain sentence. And if you defect and say, no. It was the other guy, then you get a certain sentence. And because of the way the payoffs are structured in the game, the idea is you're always better off blaming the other guy, but you both do worse than you would in these other circumstances.
Alex John London, PhD:That's why it's kind of this dilemma or a tragic case. And so why does it relate to medical research? Well, what Wertheimer had said was nobody wants to participate in research. Everybody wants somebody else to do it so that we can then get the information and use it. So I thought there were two things going on there.
Alex John London, PhD:One, this is a this idea that research is a risky enterprise is a vestige of this idea, the same idea we were talking about, that research is this kind of scary utilitarian thing that will run roughshod over the interests of people. But it's also dressing this idea up in a formal result from decision theory economics that is a really powerful result. And so if you can say that something like this result shows that that's true, then it further embeds the idea that research is a scary undertaking. And so I actually just looked at it very carefully, and the one problem is that game theory is very complicated. There's lots and lots of different games, lots of different structures to coordination.
Alex John London, PhD:But for people who are less familiar with it, game theory is kinda synonymous with the prisoner's dilemma. And I think that was really what was what was going on here. And so I just try to show that it actually has a different structure, that it's the structure of the game is a stag hunt, and that you can look up what the stag hunt is on your own time if you want to. The the bottom line is that it does make sense to cooperate and to participate so long as enough other people are going to cooperate that you're gonna produce the good that you want. So it's a very different game from the prisoner's dilemma.
Alex John London, PhD:What it shows though, to me, this is one of the most interesting parts of the book because what it shows is it is possible mathematically to reconcile the interests of study participants and the people who will ultimately benefit from the research that's generated so that you're not necessarily worse off by being a study participant. So that it isn't the case that rationality says, don't participate in medical research. All that rationality says is don't participate in medical research if you think that no one else is gonna participate in medical research. And that makes sense because if a study doesn't recruit, don't get generalizable knowledge. And now there is a real problem that we have a lot of studies that don't recruit.
Alex John London, PhD:This gets back to that question about social value. Right? And so improving the social value of our science, the research that we carry out is actually important for helping to reconcile individual interests and social interests.
Dan McLean:Taking a step back for a second, we talked about social value and research adding value to the community, but we also have a lot of for profit companies involved in drug development. So is there a tension of some kind with social value and responsibility to stockholders?
Alex John London, PhD:That's a great question, and I think it gets back to that concern I started out with about parochialism, right, that research ethics has become so narrowed and so focused on a few institutions that it has a more difficult time answering this question. Because for me, what I wanna say and what I argue in the book is we have to understand research as this collaborative activity between a wide range of stakeholders. So some of those stakeholders are academics. Academics have their own parochial interests. They have to publish papers.
Alex John London, PhD:They have to get grants. They need to get promoted. Right? Everybody needs to eat. Researchers who work for pharmaceutical companies, they've gotta generate evidence.
Alex John London, PhD:They've gotta get drugs to market. Right? They're under the patent clock. Universities who house these academics and or facilitate this research, They've got their own set of interests. The book goes through all the different stakeholders.
Alex John London, PhD:Study participants often have their own parochial interests, right? Some of them are desperate for a cure. Some of them very much want to help science and help people in the future who will have the disease that they have do better than they are doing. Some people wanna get paid. Mhmm.
Alex John London, PhD:So in all these cases, with each of these stakeholders, there can be a complex mixture of motives. And my view is one of the goals of research ethics is to try to align, create incentives and institutional structures that will align the interests of these parties as much as possible around producing high quality scientific evidence. And so, the FDA approval, the gateway, having to show safety and efficacy, that's an example where it provides a strong incentive to pharmaceutical companies to quickly and clearly demonstrate safety and efficacy. Every stakeholder wants the drug discovery process to go quickly. And so would you wanna incentivize that?
Alex John London, PhD:It's also true, that's not the only thing we care about. We want a broader bandwidth of information than we currently get, because you can demonstrate efficacy in a very narrow patient population, far narrower than what the population who's treated is going to look like. So the question for me then is absolutely true, how do we create incentives so that we distribute fairly the burden of carrying out the additional research that's necessary to answer those questions? And if we don't think about it that way, if we don't think about it at a societal level, and if we don't think about who are these different stakeholders and what are their interests, then we will not be the group that will be part of
Dan McLean:the
Alex John London, PhD:conversation about which policies or institutional structures do a better or a worse job of aligning the incentive of these parties to achieve the goals that we wanna achieve. So I gave a talk to one of the regulatory science groups at a top university, and I was trying to tell them, you're doing research ethics. And they were like, wait. No. We're not.
Alex John London, PhD:Like, we're doing regulatory science. And I because my view is the IRB is a single element in a much larger set of organizations and institutions that are all trying to do basically the same thing. Make sure that research is respectful of study participants, that it has social value, that it's relevant to the needs of the people in the communities where it's carried out, so that health care institutions can function safely and more effectively and more equitably. So the in a certain sense, the book is a recipe for how should we think about that. There's a ton of things that are going on right now.
Alex John London, PhD:I would just say that the one opportunity that they might present is can we take that perspective and try to do better as we rebuild and reconfigure the way that we think about human subjects research and the institutions that regulate human subjects research.
Catherine Batsford:Do you see a more global collaborative research enterprise in the future? You
Alex John London, PhD:know, the first thing I got involved in actually was thinking about global research because at the time, offshoring basically meant externalizing risk to internalize benefits. And to me, this was a tragedy because there are many unmet health needs around the globe where we know the answer and it's an issue of political will and scarcity of resources. That's true. But there are also a lot of needs around the world where we don't know how to meet them, and research is necessary to generate that that evidence and that information. And in some cases, those needs are unique to people in low or middle income countries or particular climates or geographic location.
Alex John London, PhD:But very often, they're shared. The globe is shrinking. And so part of the book you know, part of that early work that was the foundations for the book was to try to articulate a framework within which you could say, how can these people separated by different languages, different traditions, socioeconomic status, still be able collaborate as equals to generate information that would be relevant to both of them. Now, whether we're going to do better at that or not, I mean, geopolitically, I think there increasing trends to stop the sharing of science Mhmm. Because it's viewed as a kind of strategic asset.
Alex John London, PhD:Mhmm. You know, part of when we start talking about AI and the concerns that AI and medicine the bright sunny side is AI and medicine is going to unlock the ability to treat more diseases and cure more diseases. The scary side is it's going to unlock the ability of rogue or bad actors to make worse bioweapons. And the latter is motivating people to want to stop the sharing of large models, access to biological information, and the increasing commercialization. So that was one of the things during COVID.
Alex John London, PhD:How quickly can people get can scientists get access to samples? If it's a public good, it should be widely shared. Mhmm. If it's an if it's an asset that you're monetizing, then you wanna keep it private. Right?
Alex John London, PhD:Mhmm. And that's exactly one of those things where it's like, well, this is an incentive problem. How do we incentivize these stakeholders to share this information? So personally, I think it's always better when scientists are know each other.
Catherine Batsford:Mhmm.
Alex John London, PhD:They're in a network. Mhmm. They are critiquing each other's work, but they're also building on each other's work so that we're not reduplicating.
Catherine Batsford:Mhmm.
Alex John London, PhD:Poor quality research affects everybody everywhere.
Catherine Batsford:Yes.
Alex John London, PhD:So improving the quality of research everywhere is very likely to have widespread benefits. I still am optimistic about that. Whether we'll be able to move in that direction and achieve that soon, I think that's one of the things we have to press for. Mhmm.
Dan McLean:And if we could, this is a good opportunity to shift over to talk a little bit specifically about AI. I know you've done a lot of work on this as well with the WHO. As a first question to that, and this is something you brought up at the beginning of our conversation, was about the Belmont Report and the role of justice as perhaps the first principle. So as you think about AI data used to train it, implementation and health uses, how are those principles applied starting off with justice?
Alex John London, PhD:Yeah, so it's an interesting thing because I think the AI community has been very sensitized to issues of bias and unfairness, not just in data, but across a bunch of decision points that can affect how AI systems function. And so I think that's a really good and salutary way in which that community is, in a certain sense, ahead of the curve. It's an interdisciplinary interaction. Mhmm. You know, when I talk to biostatisticians, it's a that's a really challenging area to be in because you have to be an expert in statistics, but you have to know a lot about the medical areas where you're working in order for your advice to sort of link up with meaningful science.
Alex John London, PhD:And we're in very, very early days with the machine learning community and AI community when it comes to medicine. And so, look back seven years ago to, you know, the pronouncements of one of the quote unquote, godfathers of artificial intelligence who stood in front of an audience, I think it was in Montreal, and said, in five years, there won't be any radiology. Like, it and he said, it is just as as obvious to anybody who has eyes that AI is gonna do radiology, close down radiology departments, don't go to school as a radiologist. And there are more radiologists today than there were when he said that, and that he said that for five years from now, and that was at least seven years ago. And I think part of what it shows is there's a huge knowledge gap between what it is to know about AI and and the special challenges that arise in medicine.
Alex John London, PhD:Mhmm. So, you know, so not being a physician, right, being a philosopher, the thing that has always interested me about medicine is the uncertainty. Mhmm. There's so much uncertainty. It's never the case that you walk into a building and and the architect would go, I don't actually know how this building stands up.
Catherine Batsford:Mhmm.
Alex John London, PhD:Right? We know exactly where every beam in a building is, and we can give you the equations for what would happen if you every counterfactual. What if what if I lengthened it or shortened it? But it's not uncommon at all. You could say, what what why does this drug work to modulate a person's affect?
Alex John London, PhD:And a doctor will be like, I don't know. It does. Like, we we have really clear evidence that it does, but we don't know the mechanism. We don't know why it works. And most of the thing most of the drugs that we trial don't work.
Alex John London, PhD:Mhmm. But nobody spends a billion dollars to bring a drug to a phase three trial because they think it's gonna fail. Mhmm. And so that that just goes to show that we're still in early days in medicine. And even though we learn more and more and more, and it's not to be a pessimist to say that we're we're not going to eventually get to precision medicine, it's just to say we're not there yet.
Alex John London, PhD:And so I think part of the early days of AI is building that mutual knowledge where people on the medical side don't think that AI is a wizard that can just do everything. And the people on the AI side understand the degree of uncertainty that exists in medicine, especially since that matters for the data on which these AI systems are built. So there are parts of medicine that we will in the future look back on and say, oh, we were in those days, in 2026, our views on that topic were the equivalent of saying, like, oh, blood, bile, black bile, phlegm.
Catherine Batsford:Right.
Alex John London, PhD:Right? And and if you try to train a machine learning model on blood, bile, black bile, and phlegm, it's not gonna be very good. And and so we'll look back in the future and say, yeah, part of the problem at this day and age wasn't that our models weren't good. It was that we we didn't know enough to have the right kind of data Data. To use those models the way that we needed to.
Catherine Batsford:And eventually, it's a tool that can help accelerate if the data is good and we work together in combination, AI and medicine.
Alex John London, PhD:Absolutely. I think so what we're struggling to find out figure out now is is where can AI give us genuine uplift? Mhmm. And I know a lot of smart people who disagree about a lot of things in this space, and I have my I have my views. I think one thing that I think is is safe to say, though, is that we're we're going part of what is happening in the shakeout right now is finding practices, the procedures, whether it's on the backside of medicine as a business or in the clinical facing side, where AI can perform the kinds of things that we need in order to add some type of value.
Catherine Batsford:Mhmm.
Alex John London, PhD:And as we figure that out, we'll be able to identify what the low hanging fruit is. Mhmm. And then we'll be able to use AI more broadly to sort of pick that low hanging fruit. And we'll have a clearer sense of, ah, these are the things that require more research. These are the moonshots.
Alex John London, PhD:And so personally, I think a lot of people are confused about where that boundary is, right? And some of the things that they think are going to be easy to do now probably fall on the other side of that boundary. But we're figuring that out. And I think the sooner we do, more benefit that we'll get from this technology.
Dan McLean:So there appears to be mixture between anxiety and optimism with the deployment of AI across the board. In terms of AI and medicine, then there's a lot of different use cases in this report. Where are you on that? I don't know if that's a a linear spectrum between anxiety and optimism, but I don't know where you are on that one. Is this the best thing ever?
Dan McLean:Are you nervous about it? Is it overblown?
Alex John London, PhD:So I I think well, let me say two things. One is I'm a certain kind of nerd. Like, I like technology, and so I like to play with these systems and see what they can do. And in a certain sense, I feel a little bit robbed because I would like to be out there geeking out about these things and being like, look at how amazing it is that we have models that can do things that ten years ago we we thought were not possible. But I live in a time where people are saying things that are so overblown, where the hype is so far away from what I think the actual technical capabilities are that I wind up mostly trying to be talking people back.
Alex John London, PhD:Right? And giving but in a certain sense, if you have been in bioethics or research ethics long enough, you've seen this sort of thing before. I mean, when we sequenced the genome.
Catherine Batsford:The genome.
Alex John London, PhD:Mhmm. There was all this stuff like, we're gonna have precision medicine. We are not gonna It's need instantaneous. Yep. Because exactly, we won't need to run clinical trials.
Alex John London, PhD:Right? Just, we'll get a sample from you. We'll create a medicine that's tailored to you.
Catherine Batsford:One of one.
Alex John London, PhD:Yeah. Yeah, exactly. And ideally, is that where we wanna go? Absolutely. Do I hope we get there at some point in the future?
Alex John London, PhD:Absolutely. Did sequencing the genome open that door? No. Did it help us take some of the steps we might need to get there? Well, absolutely.
Alex John London, PhD:The electronic medical record. There's another thing, like medicine's gonna enter the digital age. Think about all the wonderful things that are gonna happen when medicine enters the digital age. And well, I mean, are we being more efficient and are there benefits that are coming from that? Sure, absolutely.
Alex John London, PhD:Did it is it the panacea that it was sold as? No. Not not at all. And so in a certain sense, I wanna say there's nothing that's different from AI, that there are use cases that have been abject failures for reasons that are, you know, banal. And and there's there are use cases that don't work because the technology can't do what it's being asked to do.
Alex John London, PhD:So for me, it's less enthusiasm versus pessimism. I'm always I have anxiety, but mostly because what I want what bothers me is reduplicated effort. Like, when you look at the pandemic, how many models were trained to try to detect and prognosticate COVID? There's been retrospective analyses that the thousands of models that were created, most of them weren't very good. It was a bunch of effort thinly spread out that didn't sort of come together synergistically, build anything.
Alex John London, PhD:And so it enabled a bunch of people to show something proof of concept, but it didn't add much value clinically. And I think the to me, the real challenge here is create pathways that allow the AI side to move beyond proof of concept. It's one thing to write a paper with a research group, get it up on archive or get it into a journal or get it at NeurIPS or some other big conference. And it's quite another thing to mature that technology to the point where it's sufficiently reliable and that it could be embedded in a healthcare system in a way that's gonna actually have a net positive for people. And that's I don't know how you wanna divide the field.
Alex John London, PhD:Proof of concept is 10%, 50%. The rest of it is the other 50, the other 90. But those are the practices that we don't have. So a bunch of things aren't perfect in drug development, like phase one, phase two, phase three, phase four. We have a paper saying this.
Alex John London, PhD:We shouldn't think about it that way. We should divide it in a different sort of way. Right? Okay. But even still, it's helpful that there was a time when you'd read a headline and you'd think, oh my god.
Alex John London, PhD:We just cured Parkinson's disease. And then you'd read the article and it'd say, in mice, you know, we can do blah blah blah. And so we we need these structures that allow us to when you read some of these AI papers and then to have it say more clearly the analog to in mice, right, that in silico or proof of concept so that you get a sense of, well, are we at the beginning here, or is this the sort of, like, you've done the pivotal trial that shows that this is ready to go into the clinic?
Dan McLean:In terms of downside risk, there's some material in these reports. And the phrase that comes out a lot is bias or algorithmic bias. Is that the largest potential harm that could be encoded into LLM? And what would the ramifications of that be? Some of the other things are and you mentioned the digital records.
Dan McLean:There's concerns about hacking and but that those were all the same concerns about digital records. Is it the bias in the data? Is that the potential downside?
Alex John London, PhD:I actually think that there are bigger downsides. One is just the sort of presumption that something is gonna work. And the need for better quality evidence that something works is likely to have external validity, that it's likely to work in the setting you are gonna actually use it. There's a big difference between the way that we do most research on, like, in a laboratory, in clinical trials, where you start with a hypothesis, and then you test that, and the data gets generated from that test. And the reason why you structure the study a particular way is to get data that will be sufficiently clean in a sense, like that where you have control for confounding so the data can answer the question that you want.
Alex John London, PhD:Mhmm. But the AI pipeline starts with the data that you have. Mhmm. And most of the data that we have is confounded and incomplete and drawn from the populations that get access to health systems or that use the particular technology, and that's often not representative. And so bias is one of many problems that can infect real world evidence, like real world data.
Alex John London, PhD:Should say real world data, not evidence.
Catherine Batsford:Are there things we can do with that creating data that is clean as we move forward, systems we can put in place that would help train the AI?
Alex John London, PhD:Yeah. I I think, look, awareness is is definitely the first step. And so and there's a so as a philosopher, I wouldn't say, yeah, there's like, this is a novel and interesting question. Like, we've there's a lot of literature about where bias comes from in the machine learning pipeline And that they are the important questions. I'm not trying to I'm saying that discovery has been made.
Alex John London, PhD:We understand it. And now the hard work is actually taking it to heart. But there are other there are other problems that are more philosophically interesting because they're less well understood. So for instance, most AI systems are about predicting, And that makes it sound like they're magical. They're gonna predict the future.
Alex John London, PhD:And they can predict the future as long as the event that you're trying to predict regularly happened in the past the way it's gonna happen in the future. But if, in medicine is it's rarely the case that we're circling back to things that we've done before. Very often, we want new things to try to do things differently than we've done before. And that is often then not a kind of prediction. It's a kind of intervention.
Alex John London, PhD:And intervention, these are really basically sort of technical terms at a certain level. They're plain English terms, but what we mean by prediction and what we mean by intervention at a kind of formal level are actually very different. And so AI systems are often not well suited for assessing what's gonna happen after intervention, unless that's an intervention that we have carried out in the past or the world is sufficiently similar to what we've done in the past. And so I think this is less I mean, it's known, but less well appreciated than the potential for bias. The reason the potential for bias is important, but it already presupposes some type of efficacy because bias is usually a concern if the idea is the system is gonna work well for some groups, but it's not gonna work well for others.
Alex John London, PhD:Mhmm. And my concern is a little bit more fundamental. Maybe it's not gonna work well for anybody. Mhmm. Mhmm.
Alex John London, PhD:Right? Because you're taking a system meant to predict, and you're trying to use it to intervene on the world.
Catherine Batsford:Could actually do harm in some cases. Absolutely.
Alex John London, PhD:Yeah. Well, think about this isn't that far in the past. If you think about the Watson Enterprise, right? So IBM had Watson on Jeopardy, winning Jeopardy. And then it was like, where are you gonna go after Jeopardy?
Alex John London, PhD:And it's they're like, oncology. It was like health care. Like, we're gonna it was the first big AI and health care push from a blue chip company.
Catherine Batsford:Mhmm.
Alex John London, PhD:Billions of dollars put into that that enterprise. Other health care systems spent in the tens of millions of dollars to be part of the Watson Health program. And at the time, people there were lots of publications talking about how wonderful what we were learning from the Watson Health Initiative was. But I think now my sense from reading the things that I've read is that there isn't anybody who thinks that we actually got anything of value out of the Watson Enterprise. And mostly that it was because that technology really wasn't suited to the types of questions that they wanted it to use it.
Alex John London, PhD:That there were meaningful ways that technology could have helped in healthcare, But but that wasn't the those weren't the tasks that were really chosen for it. And so I I worry about that as not that long ago and the real cautionary tale for what can happen when we put a lot of eggs in an AI basket without that the science being mature and without there necessarily being a good fit between what the technology can do and and what people and patients need.
Dan McLean:So to connect this all with your your book and a focus on social value and the common good, how do we best move forward as AI is developed in this space to try to get advances from it that will benefit the common good.
Alex John London, PhD:Yeah, I mean, well, a certain sense, some people are like, do you work on AI and you work on clinical trials and drugs and research? Aren't those things miles apart? And for me, they're just two phases of the same thing. There's uncertainty where we need to generate evidence to answer our questions. And the kinds of AI that interests me, AI in drug development or AI in clinical medicine, we're trying to use AI to overcome problems.
Alex John London, PhD:And clinical trials are tools that we use to try to reduce uncertainty, to overcome problems. So we don't know what this virus is, for instance, or and how it spreads, and how to stop it, and how to treat it. And and so in that sense, I think step one is seeing AI, to some degree, as less unique than people think that it is, that very often, many of the same issues that come up in medical testing. A lot of people got a quick refresher on the ethics of medical testing during the pandemic because we all were struggling with these little self testing kits. And then we were asking questions like, well, if it's positive, what does that mean?
Alex John London, PhD:And if it's negative, what does that mean? And you know, everyone had their tutorials about what's sensitivity and what's specificity. Right? And if you just take it when you feel fine and it's positive, you shouldn't believe it. And these were things that in your stats or epidemiology class, right, would have learned.
Alex John London, PhD:And a lot of those same questions arise for AI systems. So I think seeing them as slightly less novel and asking the right methodological questions is step one. Step two is just the maturity of understanding what it takes to develop safe and effective systems in this space so that there's a proof of concept and then a pipeline to mature those things, generate the kinds of evidence that we need in order to be able to say, yeah, this actually works. There's this paper called the illusion of preparedness. Part of what it has is it takes some of the major large language models and the multimodal models in particular.
Alex John London, PhD:There's tons of stuff about these things can pass the boards. So they're they're at human level performance. They do that. And they show them an image, like, on a board test. So here's a picture of a fingernail that's discolored, and then there's the question, like, what's the diagnosis here?
Alex John London, PhD:And there's, like, five four or five option. I think it's five options. And then it gives you the right answers as d was the right answer. Then they say then they give the same question, right, to the models, but they don't show it the picture. And they get it says d.
Alex John London, PhD:Mhmm. Right? It says the right answer even though it's not seeing the picture. And that's because the way these things work, if there are complex statistical relationships between the right answer and the the wrong answers that you usually put in something like this for a question like this. Right?
Alex John London, PhD:Then it will get it right more than chance, and that's what it does. Mhmm. And then when you ask it, well, why did you say that? What it generates isn't necessarily relevant. It's just that it also doesn't say anything about the the image.
Alex John London, PhD:And so to me, it's it's incredibly interesting not because these things are not capable. They're incredibly capable. It's that it's a very old scientific question about confounding and whether what we're seeing gets explained by it's looking at the image and giving you a diagnosis versus there's just a pattern in this thing you're showing it allows it to give you an output that for you looks like a right answer. Mhmm. But in the real world, if you show it a picture of this person's thumbnail, it's unlikely to give you the right diagnosis.
Alex John London, PhD:And so I really enjoy seeing those methodological papers, a, because it gives you insight into what the models are actually doing. And then because once we figure that out, now we are far closer to being able to solve whatever the underlying problem is.
Catherine Batsford:Mhmm.
Alex John London, PhD:And it goes to, you know, a problem that goes all the way back. Herb Simon was at Carnegie Mellon. He was a a Nobel laureate and one of the founders of artificial intelligence. And I I had the pleasure of meeting him when I was a very young assistant professor. And he talks about how most computer programs, you understand because you can go through it line by line and analyze them.
Alex John London, PhD:But if you get a sufficient level of complexity, then it's gonna be like an alien or or another being. Like, you won't understand how it works, and you'll have to learn that from the outside. Well, that's exactly where we are. Trying to know how these things work at a certain level, but we don't know what they're doing in very specific cases necessarily. And in that sense, then we have to evaluate them kinda like a drug.
Alex John London, PhD:We can know what the effects are, but not know why. And in order to know whether the effect is so the analogy here is, like, I get a headache and I take a pill and the headache goes away. What you wanna know is, would it have gone away if I didn't take the pill? That's why we randomize. We have a control.
Alex John London, PhD:And the study I just mentioned to you, what it's doing is it's saying, well, it got the right answer, but would it have got the right answer if it didn't have the picture? Would it have got the right answer if you said if you took the right answer out and said none of the above, doing that makes makes accuracy drop precipitously in that study. And then that leads you to say, it's not there's a confounding here. What I'm seeing looks good only because of certain things that are unlikely to be there in the actual treatment context. So those are the kind of issues, methodological uncertainty issues that interest me as a philosopher, as an ethicist, as a research ethicist.
Alex John London, PhD:And I think the more progress we make on those things, the better we'll be to actually leverage these technologies for good.
Catherine Batsford:I love it.
Dan McLean:Really great. Professor London, thanks for joining us today and
Catherine Batsford:So much.
Dan McLean:And talking with us about your book and your thoughts. For listeners who want the book, it's available where books are sold in the Oxford University Press. We'll make those links available online. But once again, thanks.
Alex John London, PhD:Thanks. It was really my pleasure. I enjoyed it. And you can buy the book, but it's also an open access title. So you can download it for free if you're interested.
Catherine Batsford:Fantastic.