In breast cancer pathology, a 2 percent chance of malignancy is the accepted threshold at which a radiologist refers the patient for further study.
In reality, that threshold varies among doctors; some are more conservative, others less so. The result is either more false positives, in which a healthy patient worries unnecessarily they have cancer, or more-worrisome false negatives, in which a patient is told they are fine when they are not.
One researcher working to reduce that gap is Stanford’s Ross Shachter. He is a professor of management science and engineering and an expert in using probability to improve decision making. Though Shachter is an engineer, he applies his approaches not to operational efficiency or business management, but to the high-stakes field of mammography, where decisions often have life or death consequences.
He says that probability and decision making theory could be integrated into artificial intelligence applications that could help doctors better evaluate patient options, outcomes and preferences to improve care.
Full Transcript
Russ Altman: Today, on The Future of Everything: The future of medical decision making.
Well healthcare is filled with decisions. Patients make decisions about how to live their life. What should I eat? Should I exercise? How much should I exercise? What kind of exercise? How hard should I work and how much leisure time should I have? How often do I want to visit the doctor? Should I take these medications that the doctor prescribed for me? I laugh because it’s well known that patients don’t always take the medications that they get prescribed. And many other decisions.
Physicians of course also make decisions about their patients and with their patients. What is the diagnosis? What treatment should I offer? How should I monitor the treatment? How often do I wasn’t to see the patient? How should I look at the patient as a whole considering their personal priorities and goals in the context of their health challenges that they face now and in the future? Decisions, decisions, decisions. What is a decision?
Well, we’ll be talking about this, but one definition that I have always liked is an irrevocable, you can’t take it back allocation of resources. Money, time, communication, something. Some would argue that you can’t take it back or it’s not a decision, it’s just playing around.
Now, there is a discipline called decision theory it’s the discipline of studying decisions, how they’re made and how they should be analyzed in an attempt to help people make good ones. Importantly decision theory tries to help make good decisions especially when there are uncertainties and unknowns. Decisions when everything is known are generally much easier, so really we’re focusing on where we have uncertainty.
Professor Ross Shachter is a Professor of Management Science and Engineering at Stanford University and an expert in decision theory, decision-making and he has a special interest in medical decision making. Ross, what makes medical decisions so interesting and do they represent a particularly difficult class of decisions that require some sort of special considerations?
Ross Shachter: They fit the paradigm of applied decision theory, what we call decision analysis really well. There is the uncertainty that you were just talking about. There are in most cases several different choices for how a patient can proceed, so there’s alternatives to be considered. And their preferences, what one patient might prefer as a treatment might be different for another patient based on their…
Russ Altman: Kind of life perspective?
Ross Shachter: How they feel about potential side effects? How they feel about the treatment itself? So there’s a lot of things that make this a really nice field in which to bring decision analysis, it’s also really important. A lot of the decisions whether it’s for me of if it’s for someone else in my family or someone else I care about these are pivotal decisions in their life, so it fits very well. You were talking about the definition of a decision and the notion.
Russ Altman: Yes and I did that with some hesitation knowing that I had a world expert sitting next to me.
Ross Shachter: Actually your definition is really good. It’s the idea that decision comes from the same root as scissors. You’re cutting the cord and you’re committing to one choice as opposed to the others that are available to you. And so some people say when I’ve announced to my friends this is what I’m going to do. That’s not really a decision, but when you put down a deposit that you can’t get back. On the day the deposit is no longer refundable —
Russ Altman: That’s a decision.
Ross Shachter: — you’ve made a decision.
Russ Altman: ’Cause there’s no going back.
Ross Shachter: Sometimes a leader who announces something there is a cost to changing their mind and so you might say that they made a decision by announcing it, but the real decision is when you sign. When it’s going to cost you to change. And you were mentioning the cost can be time. In the Emergency Room not acting right away for something, performing tests and waiting to find out the results is a decision because the opportunity, the alternative can disappear. There’s a million offers that are available to us all the time, that expire. Things that are on sale at the places where you shop.
Russ Altman: For a limited time only.
Ross Shachter: And if you don’t do anything, you’re making a decision.
Russ Altman: Right, so in your description about the features of medicine, which was great. There are two things that I wanted to highlight.
One is you talked about uncertainty and I want to ask you about that. The question is how good are patients and for that matter physicians at really thinking about probability theory and using it effectively in their thing? And that’s the first question and I think we can spend some time on that.
The second question is, you said it’s very important to understand patient preferences and that strikes me as a particularly hard thing to do because when I’m sitting in my living room I might have one set of priorities and preferences, but when I’m sitting in an Emergency Room, staring at a monitor and a doctor my ability to rationally tell you what my preferences are might be severely handicapped. And I do want to get to your recent work where all of this comes to a head, but we will get to that, but as a pre-activity tell me a little bit about patients abilities to understand probabilities and to really understand what their preferences are or our ability to help them with that?
Ross Shachter: I guess that probabilities show up more in our society. It is now standard, the weather forecast puts a probability on the weather and the weather forecasters actually get graded on those probabilistic forecasts and we use something like that as a device in some of the assignments in our decision analysis class, but where we ask the students on the multiple choices rather than to just pick one of the choices they put probabilities on the different choices and that reflects how well they understand the problem. If they really know the right answer, they can put much higher probability on that answer than the others, but if they’re not so sure it can still be the largest probability, but it’s not all of their probability.
Russ Altman: So you’re saying that there are multiple choice tests in your department where instead of saying, I choose C. I can say, I’m gonna choose 50% B. 50% C. ’cause I can’t make up my mind.
Ross Shachter: That’s correct.
Russ Altman: And that shows you the state of my understanding, but I can still get some partial credit if it’s either B. or C., if it’s D., I’m in trouble.
Ross Shachter: If you’re making decisions or you’re supplying information to somebody in the rest of your life and your not sure whether it’s B. or C., you’re not doing that person a favor by putting all of the answer on B. or C. It’s much better to give them the best knowledge you have, which is probabilities.
Russ Altman: In your experience do patients understand these concepts?
Ross Shachter: It’s hard. I think a lot of people have trouble understanding it. If I can go to a political example, Nate Silver put a prediction of 70% on Hilary Clinton being elected President, 30% on Donald Trump. It’s might have been 28% and 72%. And a lot of people said well he was wrong, but he is expressing it as a probability. The only thing where you can provably say it’s wrong is if somebody puts probability zero on something happening and then it happens. So, in our society people try to short circuit that and people want certainty from their doctor. They want certainty from the mechanic, from whoever it is —
Russ Altman: That’s what I’m paying you for.
Ross Shachter: That’s right and a lot of times we don’t have that certainty. We have is an educated guess and the probability is the best way to describe that prediction of what might happen.
Russ Altman: And then you would have to either use some tools or verbal counseling to make sure that the patient understands. I mean I think most people probably understand 50/50, but beyond 50/50 I think it becomes difficult to think about 80/20, 90/10, 70/30.
Ross Shachter: Absolutely, I mean for example, when a jury says that somebody is guilty beyond a reasonable doubt. They’re not saying a 100%, but for some people that’s 95%, for some people it’s 99%, maybe for one of the jurors it’s 99.9%.
Russ Altman: And that can explain why a jury gets hung or not?
Ross Shachter: Absolutely, plus the fact that they have different opinions. Even before they can hear the evidence they have different opinions about the situation, so we can all make different decisions there, but it is tricky to think about uncertainty and in the culture of medicine, you know more about this than I do, there’s a desire for people who are looking at cells or looking at images to state exactly what’s going on. But the information is only so accurate.
Russ Altman: So, I really want to get to your recent work, but I just want to ask you about how good are we at figuring out what patients want and don’t want, as people helping them make decisions patient preferences that you talked about. Are we pretty good at getting from patients what they’re preferences are or is that a big problem? Or both?
Ross Shachter: There’s a lot of classic research in decision analysis that says that people don’t always predict how they will feel in a given situation and so sometimes patients have biases in advance of experiencing the outcome that’s different from how they really experience it.
Russ Altman: Exactly.
Ross Shachter: Another issue in medicine is the idea of informed consent and informed consent, at least anytime I have been asked to fill out the form, is more about covering the liability for the…
Russ Altman: CYA might be the technical term.
Ross Shachter: Yes, exactly.
Russ Altman: Which I won’t define. People can Google it.
Ross Shachter: Yes, but it’s not explaining, it’s not providing the information someone needs to be able to be truly informed and give that kind of consent. Now, I’ve had some great doctors who can explain to me for each of the alternatives what the experience would be like.
Russ Altman: And that helps you figure out what do I think of that?
Ross Shachter: Yes.
Russ Altman: Okay, this is The Future of Everything. I’m Russ Altman.
I’m speaking with Dr. Ross Shachter and now I want to go to the recent work you’ve done using all of these principles and we’ve laid the foundation beautifully in the setting of mammography for breast cancer. So can you tell me about the recent work? What motivated and what did you find?
Ross Shachter: Okay, so one of the issues especially for example with a screening mammogram where the idea is to make giving the mammogram as convenient as possible, so it can be set-up at a shopping center or other places and there don’t have to be any of the physicians there. There just somebody who can operate the x-ray machine. In looking at those images somebody has to decide does this patient need to come in for further study for a higher magnification, higher powered mammographic image just on the basis of this one image. And if there’s nothing of concern on the image then it’s fairly easy to be able to say, “Okay, you’ve tested negative.”
Russ Altman: We’ll see you in a couple years.
Ross Shachter: And if there was something that’s frightful on the image they can say, “You really need to come in for treatment.” But there’s lots of things you might treat.
Russ Altman: Not necessarily treatment maybe another test.
Ross Shachter: A further study, but thank you. But there’s lots of things in between that might be a cause of concern and might be worth studying and a lot of those turn out because somebody is invited to come in. It’s a source of great anxiety for the patient, but then it turns out that there isn’t anything, on further study there is no need to go any further and we call those false-negatives. I’m sorry — false-positives. Contrarily false-positive because you’re saying the test turned out positive for disease, but it turns out it really isn’t positive.
Russ Altman: Right, gotcha.
Ross Shachter: On the other hand, there’s the notion of false-negative, there’s something on the image that later turns out to be a malignancy and telling the patient you’re fine that delays any further study and treatment. And we can all agree false-negative is bad and a false-positive is bad and there’s a question, how do you trade those off? How do you think about what to call needing further study and which things can you give the patient a clean bill of health and say, “Come back again in a year or two years”, whatever the protocol is for that patient and their age.
Russ Altman: So if you have a low threshold for sending them for the more expensive follow-up test you’re spending a lot of society’s resources and maybe only picking up a couple of extra cancers in thousands of cases those couple of people certainly benefit, but society has a huge financial and other burden. On the other hand, if you have a very high threshold, you’re gonna be sending people home too often with cancer and so there’s bad on both sides. You called it a tradeoff.
Ross Shachter: And our measurement devices are not perfect, so when we’re looking at an image or even looking at the cells under a microscope there’s a limit to how accurately one can predict the course of that disease.
Russ Altman: This is The Future of Everything. I’m Russ Altman. I’m speaking with Dr. Ross Shachter and we’re now getting into his recent work.
So what question did you ask in this paper and what’s the answer?
Ross Shachter: So what we were looking at was whether some of our machine learning and artificial intelligence of tools and in this case belief networks, which we used for probabilistic modeling. How can they help us improve the performance of physicians making this call between the false-positive and false-negative?
Well first of all, the real question is, what is the threshold? You were saying at what probability should a physician say, this needs further study and below that it doesn’t. So this is called the sensitivity and the textbook for breast cancer is 2%. That’s what the standard is it’s through something called BIRADS, which is the radiological standard for breast imaging.
Russ Altman: And what’s the significance of the 2% again?
Ross Shachter: If there’s a 2% chance this is malignant then a patient should get further study and that further study might involve higher magnification, x-rays and then it might require taking a piece of tissue, called a biopsy in order to do further study.
Russ Altman: So it’s up to a radiologist to make this call about whether we’re at 2% or not. How good are they?
Ross Shachter: Well, they’re making a holistic judgment and in our study most of the radiologists where operating at around 1%, so they were being extra cautious and causing a lot more false-positives in order to avoid the false-negatives. We did see some radiologists in our study who were up around 3%, so there can be a lot of variations among radiologists and within and across radiological practices.
Russ Altman: We’ll have to explore that more. This is The Future of Everything. I’m Russ Altman. More with Dr. Ross Shachter about his recent work with others on mammographic screening and the variation in radiology performance. Next on Sirius XM insight 121.
Welcome back to The Future of Everything. I’m Russ Altman. I’m speaking with Dr. Ross Shachter about decision making in the context of mammography screening. At the end of the last segment you gave this great story from your work. That the radiology world recommends a 2% chance of cancer and then you send them for more testing, but that when you empirically looked at it in your study, some radiologists were much more conservative and they essentially used a 1% cut off and others were a little bit less conservative and they used a 3%. What did your model do and what’s the use of the model in the context of this variability? How should we think about it?
Ross Shachter: The radiologists see themselves as making a holistic judgment based on all the data they have available, including the clinical history for the patient and they’re trying to decide should this person be sent and they don’t want to reduce it down to a probability, but in the reporting system they use. And I’m sorry, I gave the initials for it before, it’s the Breast Imaging Reporting and Data Systems.
Russ Altman: This is what BIRAD means?
Ross Shachter: This is the standard in mammography for the descriptions of the different features they’re seeing in the image and for the conclusions they make about whether somebody needs further study or not.
Russ Altman: And BIRAD is the group or the standard that says 2%.
Ross Shachter: And it’s the standard that says 2%, but there’s a resistance from patients to accept uncertainty and it’s not how the physicians want to think about it either, so it’s not just the patients who struggle with that.
Russ Altman: If I understand what you’re saying, when you say 2% that’s going to turn mathematically into a certain number of false-positives and false-negatives. False positives are expensive, but there’s no bad news except for the loss of resources. False negatives are terrible because you’ve told somebody they’re okay and they’re not, so when you say 2% it’s gonna be a ratio of those two. But what I’m hearing you saying is the radiologists don’t like to think of it that way.
Ross Shachter: And as one of my colleagues pointed out to me once the radiological image lasts forever. There’s always sort of people worried about a lawyer looking over their shoulder and looking at the image. That’s correct.
Russ Altman: Did you take these issues on in the paper or are they a side conversation?
Ross Shachter: I think we’ve been trying to deal with some of these issues through years of research on this. I don’t think we’ve closed the book on this, but what we’re trying to do is come up with a fairly robust system for figuring out what the percentage a particular physician is operating at and if we know that percentage and we’ve got this belief network system to help look at their features and try and predict the probability of disease then we can tell them when we would disagree based on their standard of practice. Or if they choose to operate at the nominal standard of 2% we can tell them when they’re invariance with that 2% standard.
Russ Altman: So, it’s kind of like just shining light on what the situation is.
Ross Shachter: It’s like when you are backing up a car and a warning system tells you you’re about to hit something. It just allows you to be assisted by a device that says you might want to think twice about this. It’s your decision, but know that based on this other system we get a different answer.
Russ Altman: In the paper that most recently came out. Did you or your colleagues feed this information back to the 1% people or the 3% people and did you see what their reactions were to seeing your a little bit more conservative than the guidelines say or you’re a little bit more liberal than the guidelines say.
Ross Shachter: No, we did not.
Russ Altman: That would count as fun in my opinion.
Ross Shachter: And one of the issues is that a certain amount of time had to pass from their judgements in order for us to be able to verify what actually happened to the patients.
Russ Altman: So in some sense this is old news and they could say, “Well that was me 10 years ago. “I’m much better now.”
Ross Shachter: They could say that.
Russ Altman: And it might be true.
Ross Shachter: And it might be true, but these issues that you’re talking about. About dealing with the uncertainty and wanting to give an answer that doesn’t seem to be uncertain. There’s a tremendous desire to do that.
Russ Altman: This is The Future of Everything. I’m Russ Altman. I’m speaking with Dr. Ross Shachter about mammography screening and really dilemma that some radiologists face in calling these x-rays.
So, this raises an issue you know you mentioned a couple of times that your using these Bayesian networks and I know a little bit about them, and I know that they would fall generally under the category of AI applications.
Ross Shachter: I’m actually the chair of a group called The Association for Uncertainty in Artificial Intelligence and our sole purpose it to have a conference every year and it’s been going on since 1984. It’s one of these artificial intelligence conferences.
Russ Altman: So that means that are conversation right now about mammography is impinging on this issue of what the role of these AI systems will be in medicine. So it sounds like you built one and it was kinda in the background watching these radiologists, it was making calls about the 2%, the 1%, the 3% and you could imagine somebody saying, “Why are we using these radiologists, Ross and his colleagues have built this beautiful, very objective, doesn’t have any emotion system. That’s how we should be making these calls about which women or men go from screening to more diagnostic studies.” So how should I think about AI systems being injected into the medical decision-making milieu?
Ross Shachter: I have colleagues who are radiologists on this research team and they tell me that the state of the art is that the artificial intelligence systems can outperform the radiologists on match races, but when it comes to…
Russ Altman: Do you mean head to head?
Ross Shachter: If you have a set of curated images, but when you actually are using what occurs in practice, weird things show up and the AI Systems don’t do as well as the physicians, so there is still a role for radiologists and other doctors making these decisions, but they can have the assistance. They can work in partnership with a system that points out to them whether it’s what to look at in the image from detection systems or from our analytical system, whether they want to think twice about this particular judgment. It’s their call, but it’s better for them to have extra information and working in partnership.
Russ Altman: With the AI system.
Ross Shachter: With the AI system. The AI system is going to be vigilant. It’s not going to be emotional. It’s not going to be biased. It doesn’t get tired.
Russ Altman: Right, which can be a real issue when you’re reading films. But I just want to make sure I understand. They did head to head tests where they just show them the x-ray?
Ross Shachter: Well actually it’s a little more complicated than that because were using the features that are identified by the radiologists and themselves, so we’re taking their features to make their prediction and the radiologists who give us the features and from whom we built the model don’t do as well in the diagnostic test for the most part as the system that’s built based on their knowledge. However, they sometimes leave off obvious things ’cause they don’t need it to reach their conclusion, so they don’t always mention all the features and there’s still information in their holistic judgment.
Russ Altman: So at least for now, until or unless the AI folks come up with much more robust systems that can pick up these external things that would make it a different answer, we’re looking at a future where the AI system augments the performance of the radiologist, but we’re not talking about radiologists large scale going to unemployment lines?
Ross Shachter: That’s correct.
Russ Altman: So it’s good news at least for the radiologists?
Ross Shachter: Well it’s good news for all of us because if we can make the radiologists can perform better working in partnership with these systems.
Source: engineeringstanford
