Intro. [Recording date: August 22, 2023.]
Russ Roberts: Today is August 22nd, 2023, and my guest is physician and economist, Anupam Bapu Jena. He is the Joseph P. Newhouse professor of healthcare policy at Harvard Medical School and a physician in the Department of Medicine at Massachusetts General Hospital. He has an M.D. and Ph.D. in economics from the University of Chicago, and he hosts the Freakonomics, M.D. podcast.
He is also the author, along with Christopher Worsham, of the book Random Acts of Medicine: The Hidden Forces that Sway Doctors, Impact Patients, and Shape Our Health, which is our topic for today. Bapu, welcome to EconTalk.
Anupam Bapu Jena: Thank you for having me.
Russ Roberts: This is a fantastic book of very interesting empirical work on very interesting questions. I found the questions as interesting as the answers, and I’m sure that’s often the case, but they’re both fantastic. But, you also raise a lot of issues along the way, besides the questions that you examine from your research and that of others. And so, I hope we’re going to get into both those things, results from the book, and the implications for healthcare policy and medical care generally.
Let’s start with what’s at the heart of the book, which is something we haven’t talked about in a while on this program, which are natural experiments. Talk about what those are and how they come up in medical care and healthcare policy.
Anupam Bapu Jena: Sure. So, maybe the best way to describe them is to start with something that might be familiar to many of us, which is something called a randomized trial. We see that a lot in medicine. We see that in other places as well. But, whenever you take a drug, ideally, we know whether or not that drug works in a population of people because it’s been tested in a trial, where a bunch of people are randomized to receive the drug and another group are randomized to receive a different drug or maybe a placebo.
And the idea is that everything about those two groups of people is otherwise similar. And so, any differences in outcomes, health benefits, or health effects that we see in the treated group versus the control group can be attributed to the drug itself. So, it says something about the causal effect of the drug on the outcome.
Now, those sorts of questions come up all over the place in public policy, in medicine, in the natural sciences as well. In economics, a lot of economists have, for decades now, been using natural experiments.
We don’t see them being used as often in medicine. And, the idea behind a natural experiment is that a group of people are exposed by chance to something–not by the hands of an investigator, but by some natural phenomenon. They’re living their lives in a group of people who are exposed, for example, to a hurricane or not, or to a public policy based on an income threshold or their age, and another group of otherwise similar people are not exposed to that thing. And so that allows you to say something about the causal effect of a particular intervention on some outcome that you care about.
Russ Roberts: So, let’s start with an example from the book which I found extremely interesting. You talk about the Boston Marathon, where tragically there was a terrorist attack there, where–in the past–where three people were killed and hundreds of others were injured. And I think it was your wife who had the deep insight that that was not the only effect of a marathon. So, talk about why marathons can be bad for your health even when there aren’t terrorist attacks, and how you might go about trying to figure out if that matters or not.
Anupam Bapu Jena: Sure. So, a few years ago, my wife ran this race. It was called the Race to Remember, and it started in one part of Boston and went to Cambridge. And, it passed by the hospital where I work, and she wanted me to watch her on the race route. So, I said, ‘Okay. All right, I’m going to do that, but I’m going to park at the hospital because I’ve got a parking spot there.’
So, I’m driving along the main thoroughfare to go to the hospital, but the exit is blocked. And so, I turn around and I go back home. And, the exit was blocked because of the race.
And so, I see my wife a few hours later, and I tell her what happened, and she says to me in this offhand way, ‘Well, what happened to everybody that needed to get to the hospital that day?’ And, that was just a totally offhand comment that she made.
And then, fast forward seven, eight months later, we published a paper in the New England Journal of Medicine just a week before the Boston Marathon in that year. And, we put together data from about 10 or 11 cities over 10, 11 years. We identified where marathons were running the routes, and we figured out who was living close to the marathon route versus just outside of the marathon route in that same city or town. And then, all we did is we look at the mortality rate of people who have a cardiac arrest, which is when your heart stops or a heart attack. So, these are very serious conditions. You don’t choose to have them any particular time. And, we looked at what happens to those conditions: the mortality rate on the day of a marathon compared to the surrounding days in the affected areas–these are areas that are along the marathon route–and then the unaffected areas, which are just outside of the marathon route, on those same days, the marathon day and the non-marathon days. It’s sort of a second control group.
And, the main findings are twofold. One is that mortality goes up quite a bit on the day of a marathon compared to the surrounding days in the areas that are right on the marathon route. And the second thing is that ambulance transport times go up by about 30% on the mornings and the early afternoons of the marathons. But then in the evening, when the roads reopen, there is no more delay.
And, you might ask, Russ, ‘Well, is this because of people running the marathon and keeling over in the street?’ No, it’s not that.
The people that we’re looking at are late 70s and above. If you look at people who have dementia or who are on dialysis, a smaller group, those are not running. Those people are not running marathons. And, yet you see a very similar pattern in that group as well. So, it’s something about the delay in care that older Americans who have cardiac conditions experience because the roads are closed on the dates of the marathon. So, that’s the main finding.
Russ Roberts: And, in all of your studies, one of the things I really appreciated is that when these natural experiments occur–in this case, the natural experiment is there’s a marathon. And, of course, it’s unlikely but possible that people have cardiac events on the day of a marathon because for reasons you can’t observe, not because they’re excited about the race. It just so happens that on certain days where the marathon happened to be, it happened to be picked on a day when it was extra hot or extra cold, or extra something. And, these are, as you identify in the book, these are confounding factors, and you often want to check and make sure that the group that you’re calling your control–because it’s not a controlled randomized trial–that they are similar.
So, almost every study you did in this book, at least I think every study, you look and see whether at least on the things you can measure, whether the traits that might affect outcomes–income, for example, education–and various other related–age–whether they’re similar in the two groups.
I’m curious: Often you just say, ‘We checked it and turned out they were,’ but it’s a little more complicated than that. So, why don’t you talk a little bit about how you try to assess that and what do you do if it turns out not to be the case?
Anupam Bapu Jena: Good question. So, in any randomized trial, and let’s say in a medical study, the first exhibit that is presented in a scientific paper is typically this comparison of the treatment group and the control group to establish that they look very similar, and you would expect that because they’ve been randomized to an intervention versus not. So, it’s not a surprise, but you always have to show that.
And so, in any study that we do where we are arguing that people are by chance exposed to something–in this case, a marathon versus not–the first thing that we show is that on all observable characteristics that we have, that the groups are similar.
And, we go through painstaking detail to show this. We show on demographics that they’re similar. We show that in terms of the chronic conditions that they have, that they are similar. For example, rates of high cholesterol, high blood pressure, diabetes, atrial fibrillation, heart failure. I mean, I could go on with a long list of medical conditions.
If I noticed that some of those conditions were more prevalent among people who had a heart attack on marathon day versus other days, I’d have some pause because I’d say, ‘Wait, okay, maybe these groups are not as good as randomized. Maybe there is something different about the people who are having a heart attack on marathon day versus not.’
We also look at things like the types of procedures that might be performed on marathon versus non-marathon days. So, I sort of gave you a story of: Look, there are delays in care that are happening. But, what if, instead, what happens is that there are just different doctors and different nurses in the hospital, or that they are short-staffed on marathon day because healthcare workers are at the scene, or they decide they can’t come to the hospital that day because the roads are closed?
We can look at that, as well, by looking at staffing levels. We can look at the types of procedures that are performed in the hospital. We can look at the types of conditions for which people are hospitalized.
So, the main thing when you’re trying to say that there’s a natural experiment that’s happening–an experiment in nature–is to try and suss out all of these different ways in which the groups might be different and just to make sure that they are the same.
The other thing, though, that it’s really important is that even when you have a natural experiment, you can say that the effect that we observe is because of the marathon. But, an interesting thing, and I don’t think we think about this enough even in randomized trials, is it doesn’t tell you the mechanism by which this is happening.
So, marathon day is random. Let’s say we can establish that. But, is the effect because of road closures? Is the effect because of–we talked about staffing problems in the hospital? Is it because people are upset that–you joked about people being excited, but maybe people are upset because the roads are closed and that stress and anxiety provokes a heart problem.
Maybe people know that there’s a marathon happening, and then they delay calling 911. So, it’s not actually that there’s an ambulance delay because they’re delayed because the roads are closed, but they just call 911 later than they otherwise would have. That, by the way, we can look at. We actually have the timestamps of when 911 is activated, and we don’t see that there’s any delays in when they actually call the ambulance.
So, there’s all of these sorts of things that we want to be thoughtful of when we do this study.
Russ Roberts: The other thing I think that’s important that often gets lost in this kind of work in academic life–less so in a book for the general public–but it’s a very important distinction between the way economists use the word ‘significant’ and the way normal human beings use the word.
So, economists use the word ‘significant’ in this kind of a study to mean we have a pretty good likelihood that this did not occur by chance but was due to the marathon. In other words, the numbers are always going to be different because of randomness. No two days are alike. But to attribute it to the marathon, it has to be sufficiently large, given the size of the sample. This is a standard thing that leads to a published paper.
But it doesn’t mean it’s significant in the everyday sense of the word. And, in this particular case, and I don’t know if you have it off the top of your head, but the magnitudes matter a lot. Is this one extra person? Is it ten? Is it 10% of the heart attacks turn out worse? And, you’re very careful in the book, usually, to share that measure of everyday use of the word ‘significant.’ So, in this case–the case of the marathon–how bad is it for people who go into cardiac arrest on marathon day?
Anupam Bapu Jena: Yeah, great question. So, a couple thoughts. So, one is you’re absolutely right. We always want to distinguish between a pure statistical significance versus economic significance; and, very closely related to economic significance, when you’re talking about medical issues, I’d call that clinical significance. So, is this a clinically significant finding?
And, I think it is for a few reasons. So, one is that–just percentage terms, the mortality rate goes up by about 20% on marathon days compared to non-marathon days.
Now, so that’s a meaningful increase. That’s the first data point for you. It’s not reassuring, but it’s meaningful.
The second data point is that–and you started off by describing the Boston Marathon bombings, injured a lot of people, killed several people; that event is salient in most people’s minds who live in Boston and probably outside of Boston–but, more people would be projected to die from marathon-related road closures in any given city in any given year than died in the Boston Marathon bombings.
So that gives a sense of, ‘Well, it’s meaningful,’–you know, what do we mean by meaningful? I think that’s meaningful in terms of interpretation, as well.
And then, the other thing, which I always think about as well, is: Sometimes a finding can be meaningful or economically or clinically significant, but you always want to have–there’s got to be, like, a sniff test. Because sometimes it’s too large. And, if it’s too large, you’ve got to say, ‘Okay, what’s going on here? Either there’s something that I’m probably missing,’ if it’s too large. Either there’s a literal data error–some calculation problem in the analysis–or there may be something that you actually are not observing, a true confounder. One of those two things is going on.
And so, whenever we have a study like this where there is sort of a clinical dovetail, you know, the story is about delay. So, there is an estimate that could come from our study about how much do minutes of delays matter in terms of mortality? And, if that is consistent with what we might see in other non-randomized settings, ballpark, that’s more of a reassuring thing to see.
Russ Roberts: It’s a great example.
Russ Roberts: Now, when you’re presenting these results, you mention, ‘Well, we’re not going to not hold marathons.’ But, you could argue–you can make a legitimate argument for it. It doesn’t mean it’s decisive. But, if it turns out that your view holds up–this empirical finding–I would give it some serious thought.
But, more importantly, I would give some thought to how we route marathons and what roads we close. And, you would start to think about whether there are other facilities we ought to make available on marathon days. I mean, the Boston Marathon has to be a somewhat iconic route. It’s been run for a long time. It would be emotionally, I think, challenging for the people of Boston to have a different route. But, certainly, you could route part of it maybe away from the hospital. You could, again, put facilities in other places for that day, as an emergency. I don’t know–have people talked about that? Have you talked about it?
Anupam Bapu Jena: They have not as much as I would have thought. So, I remember when this paper first came out, we actually called a bunch of emergency medical service agencies in different cities, and some of them were aware that this might be an issue, and others were not really aware. Because, what they typically focus on–and this is not just true of marathons: marathons are an extreme example because there’s so many miles of road that are shut down–but it could be something like a Taylor Swift concert, which leads to a couple of hours of delay in a major metropolitan area. Any sort of large public gathering–concert, July 4th celebration, whatever–may have that same effect.
And I think we focus on the safety and health of the participants of those events, not so much the bystanders.
And I think, so, some agencies are not as familiar with that idea as they probably should be. But I think where they could come in and say, ‘All right, look, well: Here’s how we would be able to address it.’
So, a simple example–maybe not simple–but one thing would be: if you’re not willing to bisect the marathon route, then maybe what you do is you have sort of pods–areas where on one side of the marathon route there’s an open field. So, if you have a cardiac arrest–a heart attack–you can be driven there, and there are multiple helicopters there waiting to transport you right above the route to the place where you need to go.
Now, I don’t know how feasible that would be. Certainly, there’s a cost consideration, but you could do that cost-effectiveness analysis and figure out whether or not it makes sense to do that.
But, the key thing is that people should be thinking about that issue in general.
And, the other thing I’d say, Russ, is, so that’s sort of–like, what do you take away from this paper?–is: Okay, that’s one thing. How do you structure these events?
But, there’s another thing that I think is also important, which is it firmly establishes how time matters when it comes to acute medical problems. It’s something that, in medicine, we all think is true, but it’s hard to show that it’s true.
And, one of the most important questions that any of us face when it comes to medical care–whether you’re on the receiving end or the deliverer of that care–is, ‘Do I need to do something now?’ You know, you’ve got a kid who is three years old at home in the middle of the night, and they’ve got a fever and headache. Do you need to call the pediatrician now? Can you wait a few hours? Do you take them to the emergency room?
You’re never going to conduct a randomized trial that says, ‘All right, a thousand people who have chest pain, go to the ER [Emergency Room] immediately, and the rest of you tune into this podcast.’ You can’t do that. But here, nature gives us that information.
Russ Roberts: Yeah, it’s incredibly powerful, actually, in thinking about exactly the question of, ‘How much effort you should put into being timely in various situations?’ Like you say, we all know intuitively sooner is better than later. But, in many, many things, sooner is not particularly better, and later is a lot cheaper–
Anupam Bapu Jena: Yeah, exactly–
Russ Roberts: because you don’t really want to go to the hospital in the middle of the night.
Anupam Bapu Jena: Exactly. Sooner is almost always more costly. Yeah.
Russ Roberts: Yeah. And, by the way, there’s also weather, would be another example. I always wonder about–in snowier climates, I assume it takes longer to get to the hospital, and there are often snow events that are not well dealt with in certain climates. As an economist, I’m willing to entertain the idea that people with heart issues are more likely to avoid snowy areas so they can get to the hospital more quickly. But, I don’t think it’s a big effect. But I don’t think it’s a big effect.
So, I think the delays due to weather would be another obvious example of where you, on those days, you might want to have some alternative procedures. And, another way to deal with it, as you mention, is to deliver care on the spot rather than getting them to the hospital, which is another way to cope with this, the delay challenge.
Anupam Bapu Jena: In some places, they actually try to do that. And, I think it depends on the particular medical care that is being administered. In some European countries, they have almost these hospitals in an ambulance. And, for some things, that might work. But, I could imagine for something like an infection–let’s say pneumonia or sepsis–here’s an area where you could certainly imagine, if you thought this person had sepsis, giving them antibiotics early. And there, would a matter of 30 minutes or an hour matter?
I think they might. It’s hard to show empirically, but it is a sort of question where the implications of getting that answer are quite large. If you could show credibly that the administration of antibiotics 30 minutes to an hour earlier at the scene of where you first see a patient in their home or a nursing home improves outcomes, then that’s something we should be doing, and we certainly could be doing quite well.
But, you know, weather is another interesting area because it is random oftentimes. So, it’s good for that, and it also affects a lot of different things. I was having a discussion with someone the other day, and they were talking about the effect of rain; and it could be the case that, for example, storms reduce mortality. So, one of these is they increase mortality because of delays and the direct effects of the storms. But, if people drive on the roads less often because of storms, which is totally reasonable, you might see actually a reduction in traffic fatalities. Or people drive more carefully.
So, if you have as your denominator the number of people on the road, car injuries might go down, not only because there’s fewer cars in the road, which is one effect. But, also, people are vigilant. They know what to expect, and so they drive more safely accordingly.
Russ Roberts: Yeah. A related question would be whether, how many ambulances crash in their zeal to reduce time to the hospital? I assume it happens–I’ve never seen one–but I assume it happens every day in America: an ambulance crashes somewhere.
Anupam Bapu Jena: Yeah, it must happen. It is also an interesting question about whether or not they should go with lights and sirens, because those things can be distracting, and what happens to other cars who are trying to evade the ambulance? I mean, these are interesting questions.
Russ Roberts: One of the things that kept echoing for me in your book–and I’ll just mention, I didn’t say it before: a lot of the work you’re reporting is work you’ve done with various co-authors using these kind of techniques. There are many, many things in life that can’t be measured, and there are many things that we desperately want to measure that we can’t measure.
I thought about this both in terms of doctors, which we’ll talk about, but also patients, where we know some things about a patient–we might know their age, we might have a lot of their prior medical history–but we might not know how healthy they really are. In fact, we don’t know. And, we’re using these various measures that we have data on as proxies. And, I kept thinking about this. So, let’s talk about a couple examples that I thought were really fascinating.
You ask a profound question in one chapter: ‘What makes a good doctor?’ And, you look at older versus younger doctors, and you look at doctors from prestigious medical schools versus not-prestigious. In an earlier chapter, you looked at doctors who were doing research and are in theory on the cutting edge who didn’t go to conferences, and the doctors who were left behind to take care of their patients, who were not as–in theory–not as experienced, not as skilled, not as sophisticated. And, you find very interesting things in those cases.
So, try to go through each of those and talk about what you’ve learned in the studies you’ve done.
Anupam Bapu Jena: So, let me start with the second one because there’s things about what is it that makes a good doctor; and then there’s the things that doctors do more or less of. And, I think that can separate why we’re finding these different things here.
So, in the second example, there’s a chapter, ‘What happens when all the cardiologists leave town?’ And, what we showed was during the dates of these major cardiology conferences–like the American Heart Association or American College of Cardiology–patients who have cardiac problems, acute cardiac problems, they actually do better on the dates of those meetings, if they by chance are hospitalized when the cardiologists or some cardiologists are out of town.
Russ Roberts: So, their regular doctor is out of town. They’re stuck with the kid–the young upstart who couldn’t get a paper accepted at the conference–and they have lower mortality rates during the conference. It’s shocking, but possibly true.
Anupam Bapu Jena: Yeah, exactly. And, it’s a type of thing where I actually had the opposite instinct. When I thought about the idea, I thought, ‘Okay, staffing is going to be lower, patients are going to do worse because the staffing is lower, and maybe the staffing of the better cardiologists might be lower.’ And then we find this finding, which is the opposite of what we thought. And then, this goes back to our earlier discussion: ‘Well, what do you do then?’
Well, the first thing you got to do is establish: ‘Is this a natural experiment?’ Show that these characteristics of these patients are identical on cardiology meeting days and non-meeting days.
All right, check. We do that.
Now we’re into the: ‘What is the mechanism that explains this finding?’–land. And, there is an additional data point that we figured out, which I think tells a story, which is that: Rates of certain procedures fall. They fall by about 30% on the day of these meetings versus the non-meeting days.
So, how do you put that together?
The way that I put it together–and I’ll give you maybe two stories–is as follows.
So, the first is a story about a young man and an older woman. The man is 40 years old. He’s a construction worker. He has chest pain while he’s working at a construction site. No other medical problems. He gets rushed to the emergency department. They do an electrocardiogram–an EKG–of his heart. He has a heart attack. They rush him to the cardiac stenting lab. They put in a stent. He’s discharged from the hospital in two days, lives a long, happy life. Great.
Then, there’s another woman. There’s another person who is a woman. She’s 90 years old. She lives in a nursing home. She’s got 10 different medical problems because she’s lived a long life. She’s 90 years old. And, she has the exact same chest pain. She is brought to the emergency department. She gets an EKG–and she has the exact same EKG. And, she also has a heart attack.
So, they take her to the cardiac stenting lab. She also gets a stent, but she dies within two weeks of that procedure because of the complications of the procedure.
That sort of story I think will resonate with anybody, whether you’re in or not in medicine. And, the idea there was that for the first guy, it was black-and-white whether he should get a stent or not. In the second person, we might act like it’s black-and-white in part because either we don’t have the data to tell us otherwise or because we have an inclination to want to do something to help people. And, maybe there’s financial things at play as well. Though, I think that’s less important than the desire to do something to help somebody who has a physiologic problem that we think we understand and that we can solve.
But, those types of people, we’re actually probably in either the gray area of medicine or maybe the other portion of black-and-white where you don’t do anything.
And, the second story is the following. If you were to take any given cardiologist and say to them, ‘All right, I want you to think about all of the procedures that you did in the last year’–let’s say a cardiologist did 400 procedures–and you said to them, ‘Pick the hundred procedures where you’re most confident that you delivered a procedure for which the benefit exceeded the risk.’ I’m pretty sure that they could give you close to 100 people where that was actually true. Right? Maybe not perfectly, but for the most part.
But, the challenge is: in the real world, they’re not constrained to doing it in 100 people. They can go all the way down the distribution of risk/benefit; and they may end up in a place where the benefit is outweighed by the risk. And, they don’t know that. They don’t know that that’s true. Otherwise, they wouldn’t do it, I presume.
But, the challenge in what I think these sorts of studies can illuminate is: All right, something might work on average. We know that cardiovascular care is on average good. But, it is also true that on the margins, there may be people for whom the benefit is outweighed by the risk. How do you show that? How do you quantify the magnitude?
And then, for me, the most important question is: ‘Is it even possible that cardiologists could figure out who those people are?’
And I think what these types of studies say, is, ‘Yeah, they probably can. If push comes to shove, they can prioritize.’
So, on the dates of these meetings, there are people who don’t get procedures. But that’s not random. The people who don’t get procedures are not randomly chosen to not be given a procedure. They are selected intentionally by the cardiologists in the hospital to not receive that procedure. And that, to me, is very powerful.
Russ Roberts: And then, the question is, ‘Why is it that when the conference isn’t going and the high-level research cardiologist is in charge–not the stand-in colleague replacement–why in those settings the cardiologist is more likely to order those procedures?’ Because that’s the claim, right?
Anupam Bapu Jena: Yeah, that is the claim.
The short answer there is, I can’t tell you. I don’t know. I think there’s a few possibilities.
So, one is that there is an incentive to do it. Now, the question is, ‘What is the incentive? Is it a financial incentive?’ A lot of times when I give a talk about something like this, people say, ‘Well, in America, we’re paid to do more. The cardiologists have an incentive to do more.’
And, the short answer there is, I don’t think that that’s driving it. For a few reasons. One is, in most of these settings, we’re looking at academic teaching hospitals where their doctors are not often paid the same way that they might be paid outside of that setting. So, it is true that there is some fee-for-service type payment, but it’s not nearly as big a deal as in other places.
And so, I don’t think that that’s what’s happening.
And also, there’s other evidence suggests that the responsiveness of procedures or the type of care that you provide relative to what you’re paid, it’s there, but it’s small. It doesn’t explain this.
So, I think what’s going on is that there’s just a desire to do something, right? Think about what’s happening here.
There’s a woman who has a blockage of her heart. We know that. If you unblock something that is blocking an artery, wouldn’t that make it better? Yeah, it makes sense that it would make it better.
And, unless you had convincing evidence to the contrary, you would do what you think physiologically makes sense. And, I think that’s a big driver. We operate in medicine, in an area where we have some physiologic information or data or evidence, but it’s not perfect.
And, that is where the art of medicine comes into play. It’s to know: Who is this person in front of me? Do I think the risk and the benefit are aligned, or are they wildly different in one direction or another?
Russ Roberts: Yeah. We had a recent episode–it hasn’t aired yet, but by the time our conversation is aired, it will have been released–with Vinay Prasad–on this question of screening and the natural–not just the financial–incentive, but the natural emotional need that we all have to find out and to rule out or to feel comforted. And, I think that’s part of it.
But I’m going to give a different answer, that I like to think about. It’s not an answer: it’s a speculation. And I’d love your reaction to it.
So, you come out of medical school, and you think everything works because you’ve drunk the Kool-Aid and you’re in love with your profession, and it’s extraordinary. And, at some point, along the line, I think you probably start to wonder whether some of the things that you’re told are indicated in this setting or maybe not the right thing.
But, if you’re really successful and you’re doing the 400 or maybe more procedures a year, I would suspect that when you’re at the top of your career, in the peak of your skill, you kind of get a little overconfident. And maybe rightfully so. You’re really good at these things. You’ve done them a lot. And, although a young person might not be confident doing a stent on a 90-year-old woman, the veteran’s done it so many times: ‘I think I can handle. I think she’s going to be okay.’ And, they convince themselves.
And then, there’s probably some point–there’s a U-shape or inverted-U here–where you start to think, maybe later on, you start wondering, ‘All those times I did that procedure, maybe they did not work out the way I thought they did.’ And, you don’t get a lot of direct feedback. Your patients die. No one sends you a card three years later and say, ‘Well, they made it three years. You thought it was five, but just want to let you know.’ So, there’s not a lot of direct evidence.
And, ego plays a role. And you also want to convince yourself–you’ve been doing this procedure for so long to so many people–the idea that it might not be always indicated, it’s kind of unnerving. So, I don’t know. What do you think of that?
Anupam Bapu Jena: So, I think it’s perfect. I completely agree. I think that is definitely going on.
And, the way I would think about it–another way to say it–is a following: is, if you’ve got a sick person in front of you, if you do a procedure and they benefit–or you’ve got a sick person in front of you, if you do a procedure and they, quote-unquote, “do well,”–you pat yourself on the back. You say, ‘I made a good decision. I intervened at the right time. This person benefited.’
If you do that to the same sick person in front of you and you do a procedure, they do poorly, you don’t think to yourself: ‘Wow, the procedure caused that.’ Well, you know what? ‘This was a really high-risk person. The likelihood of this thing working out was not that high because of all these 10 other medical problems that this person had, and she’s 90 years old. These things happen.’
So, there’s ways that people can rationalize what they want to be true.
There’s another way to kind of get at this question. I do think you’re right: that, as you get more experience, confidence does rationally build. There might be irrational components as well, but I think first order probably is a rational response, which is that you get more experience, you get better at things, and you can do things in places where you perhaps couldn’t have done them before.
But, that might come with some trade-offs that you don’t recognize.
One idea that I’ve thought a lot about is whether or not doctors are able to actually suss out the difference between what they did and what something else did.
So, here’s a good example. Imagine you have a cancer patient who is unlikely to respond to a treatment. And, you give them a medication. And they do really, really–a cancer medication–and they do really, really well.
That could be totally random, right?
But, my guess is what you would see is that doctors who see patients who are subsequently very similar to that patient would be more likely to give that medication because they ascribe the benefit to the medication–as opposed to something else. Like this person just–even though they looked like they weren’t doing well, physiologically, they were quite strong, they’re vigorous, they’re not that frail compared to other people. So, they might have been able to do better anyway.
Maybe it’s through the medication, but maybe not.
And so, there’s all of these instances where doctors are exposed–first of all, they’re often not exposed to any feedback. But sometimes they get feedback, but they might interpret it in different ways.
Russ Roberts: Sure, yeah. I like the–one of my favorite doctors, a friend of mine, was–he’s very aware that most things get better on their own, and he’s very careful not to attribute his own intervention or the drug or whatever it is. And, it creates a natural skepticism that I–as listeners will not be surprised to hear–I find I’m very biased toward believing that. But I don’t know how widespread that is in the profession. But, that’s a person who’s older and wiser.
Russ Roberts: But, I want to try something different on you. And this is kind of a crazy idea. Because I got thinking about when you say, ‘What makes a good doctor?’
Our system–the system in America–is actually designed to produce good students, not good doctors. We like to think they’re related, but they’re increasingly–that relationship is increasingly unimportant in a world with more AI and a world of the Internet, right?
So, the fact that a really smart person with a really good memory–that used to be a really helpful thing in diagnosing, in diagnostics, and possibly in treatment.
But, in a world of AI [Artificial Intelligence] and a world of Google, those skills are not so important. There are other things.
And I’m going to pick one for fun. You didn’t talk about this, but you raise a wonderful question in the book. You see somebody come through the door to be your doctor. How do you feel when you find out it’s male/female, black/white, fancy medical school/not fancy? Etc., etc?
But, I would suspect that the best doctors aren’t easily measured by any of those things. And, age–all the factors that you write about in the book–that there’s something intangible we might call–I’m going to call it intuition, a term that I’ve struggled with. I used to sneer at it, but after hearing Patrick House talk about it–the neuroscientist in the episode we did with him–where he sees intuition, he says, ‘It’s not your gut. Your gut is good for dissolving food.’
Intuition is when–it’s the part of your brain that processes a lot of data and evidence and things that have happened, but without your conscious knowledge. And, that’s what you’re drawing on when you’re really a great diagnostician, I would think.
So, we don’t spend, I assume, any effort in applying and looking at candidates for medical school to find out what might correlate with great intuition. Or other characteristics–bedside manner being an obvious example that is important but not generally cared about in admissions.
Have you thought about how we might improve that? Instead of just picking the doctors who got the best grades, we actually picked the ones who would be the best doctors?
Russ Roberts: It’s two different things–
Anupam Bapu Jena: I’ve thought a lot of–I mean, there’s six different ways we could go with that. But, remind me at the end, I do have a thought experiment. I’d be curious to get your answer on it because it sort of meshes with this idea of intuition, or what I might call the art of medicine, which I think are related.
But, yes: I think that the way that we pick doctors is–the way that we pick doctors and the way that we train doctors is arcane.
And, one way you could show that empirically is I bet that if you looked at the gradient between, sort of, doctors who went to certain medical schools–quote/unquote “the best ones,” versus “the worst ones”–50 years ago versus now, the gradient in outcomes, however we might measure it, is probably much narrower now because there’s a lot of other factors that go into a good clinical outcome besides whether or not the doctor had an encyclopedic knowledge. Which might’ve mattered before there was things like computers where you could look up information.
We teach a lot of things in medical school that are completely irrelevant to the practice of medicine. A lot of things, actually.
And then, we do not teach things which are extraordinarily irrelevant to the practice of medicine.
So, I think one of the–and I think Vinay talks a lot about it actually–is that one of the key things that doctors have to be able to do now is interpret evidence. Interpret data.
And, a new clinical study comes out: Is it a high-quality study? Is it a low-quality study? Do you allow it to influence your decision-making in some sense?
That is a key skill that is required, I think, of physicians and other providers today that we don’t teach at all in medical school. Which is why I think in many instances we see very low-quality studies being used, produced, and relied upon.
So, that’s one thing.
Then, there’s another thing which relates to, sort of the art of diagnosis. I don’t think we spend enough time, in medical school, training people to be good diagnosticians. Because ultimately, that’s the core–one of the core purposes of a doctor is to be able to make that right diagnosis.
And only in the last 10 years or so have we sort of understood that one of the key quality problems in this country is misdiagnosis. We focus a lot on safety errors–like, amputate the wrong leg, or you operate on the wrong part of the body. Those sorts of things. You leave a tool in the body during surgery. Those are sort of obvious problems.
The less obvious ones are: Someone came in with chest pain. And you thought about a clot in the lungs, you thought about a heart attack, you thought about a ruptured lung; but you didn’t think about a ruptured aorta.
Why didn’t you think about that? What is it that prevented you from making that correct diagnosis?
So, that is an area where I think the training could be a lot better. Now, in terms of selection–you know, how would you choose medical students differently? I mean, I still think you want to try to get as smart as people as possible to be in medicine. And, is the testing going to be any different? Like, we have standardized tests. Would you change them in any way to attract a different type of person? I don’t know.
I think probably what we want is really smart minds. So, the selection I think may not be that different, but the training might need to be very different. So, I think there’s a lot there.
But, let me get to your response to that, then I’ll give you a thought experiment. [More to come, 43:34]
