Posts Tagged ‘medical decisionmaking’

Science Policy Forum: Should we trust healthcare explanations from AI predictive systems?

Some in industry voice their concerns

Curator: Stephen J. Williams, PhD

Post on AI healthcare and explainable AI

   In a Policy Forum article in ScienceBeware explanations from AI in health care”, Boris Babic, Sara Gerke, Theodoros Evgeniou, and Glenn Cohen discuss the caveats on relying on explainable versus interpretable artificial intelligence (AI) and Machine Learning (ML) algorithms to make complex health decisions.  The FDA has already approved some AI/ML algorithms for analysis of medical images for diagnostic purposes.  These have been discussed in prior posts on this site, as well as issues arising from multi-center trials.  The authors of this perspective article argue that choice of type of algorithm (explainable versus interpretable) algorithms may have far reaching consequences in health care.


Artificial intelligence and machine learning (AI/ML) algorithms are increasingly developed in health care for diagnosis and treatment of a variety of medical conditions (1). However, despite the technical prowess of such systems, their adoption has been challenging, and whether and how much they will actually improve health care remains to be seen. A central reason for this is that the effectiveness of AI/ML-based medical devices depends largely on the behavioral characteristics of its users, who, for example, are often vulnerable to well-documented biases or algorithmic aversion (2). Many stakeholders increasingly identify the so-called black-box nature of predictive algorithms as the core source of users’ skepticism, lack of trust, and slow uptake (3, 4). As a result, lawmakers have been moving in the direction of requiring the availability of explanations for black-box algorithmic decisions (5). Indeed, a near-consensus is emerging in favor of explainable AI/ML among academics, governments, and civil society groups. Many are drawn to this approach to harness the accuracy benefits of noninterpretable AI/ML such as deep learning or neural nets while also supporting transparency, trust, and adoption. We argue that this consensus, at least as applied to health care, both overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable.

Source: https://science.sciencemag.org/content/373/6552/284?_ga=2.166262518.995809660.1627762475-1953442883.1627762475

Types of AI/ML Algorithms: Explainable and Interpretable algorithms

  1.  Interpretable AI: A typical AI/ML task requires constructing algorithms from vector inputs and generating an output related to an outcome (like diagnosing a cardiac event from an image).  Generally the algorithm has to be trained on past data with known parameters.  When an algorithm is called interpretable, this means that the algorithm uses a transparent or “white box” function which is easily understandable. Such example might be a linear function to determine relationships where parameters are simple and not complex.  Although they may not be as accurate as the more complex explainable AI/ML algorithms, they are open, transparent, and easily understood by the operators.
  2. Explainable AI/ML:  This type of algorithm depends upon multiple complex parameters and takes a first round of predictions from a “black box” model then uses a second algorithm from an interpretable function to better approximate outputs of the first model.  The first algorithm is trained not with original data but based on predictions resembling multiple iterations of computing.  Therefore this method is more accurate or deemed more reliable in prediction however is very complex and is not easily understandable.  Many medical devices that use an AI/ML algorithm use this type.  An example is deep learning and neural networks.

The purpose of both these methodologies is to deal with problems of opacity, or that AI predictions based from a black box undermines trust in the AI.

For a deeper understanding of these two types of algorithms see here:


or https://www.bmc.com/blogs/machine-learning-interpretability-vs-explainability/

(a longer read but great explanation)

From the above blog post of Jonathan Johnson

  • How interpretability is different from explainability
  • Why a model might need to be interpretable and/or explainable
  • Who is working to solve the black box problem—and how

What is interpretability?

Does Chipotle make your stomach hurt? Does loud noise accelerate hearing loss? Are women less aggressive than men? If a machine learning model can create a definition around these relationships, it is interpretable.

All models must start with a hypothesis. Human curiosity propels a being to intuit that one thing relates to another. “Hmm…multiple black people shot by policemen…seemingly out of proportion to other races…something might be systemic?” Explore.

People create internal models to interpret their surroundings. In the field of machine learning, these models can be tested and verified as either accurate or inaccurate representations of the world.

Interpretability means that the cause and effect can be determined.

What is explainability?

ML models are often called black-box models because they allow a pre-set number of empty parameters, or nodes, to be assigned values by the machine learning algorithm. Specifically, the back-propagation step is responsible for updating the weights based on its error function.

To predict when a person might die—the fun gamble one might play when calculating a life insurance premium, and the strange bet a person makes against their own life when purchasing a life insurance package—a model will take in its inputs, and output a percent chance the given person has at living to age 80.

Below is an image of a neural network. The inputs are the yellow; the outputs are the orange. Like a rubric to an overall grade, explainability shows how significant each of the parameters, all the blue nodes, contribute to the final decision.

In this neural network, the hidden layers (the two columns of blue dots) would be the black box.

For example, we have these data inputs:

  • Age
  • BMI score
  • Number of years spent smoking
  • Career category

If this model had high explainability, we’d be able to say, for instance:

  • The career category is about 40% important
  • The number of years spent smoking weighs in at 35% important
  • The age is 15% important
  • The BMI score is 10% important

Explainability: important, not always necessary

Explainability becomes significant in the field of machine learning because, often, it is not apparent. Explainability is often unnecessary. A machine learning engineer can build a model without ever having considered the model’s explainability. It is an extra step in the building process—like wearing a seat belt while driving a car. It is unnecessary for the car to perform, but offers insurance when things crash.

The benefit a deep neural net offers to engineers is it creates a black box of parameters, like fake additional data points, that allow a model to base its decisions against. These fake data points go unknown to the engineer. The black box, or hidden layers, allow a model to make associations among the given data points to predict better results. For example, if we are deciding how long someone might have to live, and we use career data as an input, it is possible the model sorts the careers into high- and low-risk career options all on its own.

Perhaps we inspect a node and see it relates oil rig workers, underwater welders, and boat cooks to each other. It is possible the neural net makes connections between the lifespan of these individuals and puts a placeholder in the deep net to associate these. If we were to examine the individual nodes in the black box, we could note this clustering interprets water careers to be a high-risk job.

In the previous chart, each one of the lines connecting from the yellow dot to the blue dot can represent a signal, weighing the importance of that node in determining the overall score of the output.

  • If that signal is high, that node is significant to the model’s overall performance.
  • If that signal is low, the node is insignificant.

With this understanding, we can define explainability as:

Knowledge of what one node represents and how important it is to the model’s performance.

So how does choice of these two different algorithms make a difference with respect to health care and medical decision making?

The authors argue: 

“Regulators like the FDA should focus on those aspects of the AI/ML system that directly bear on its safety and effectiveness – in particular, how does it perform in the hands of its intended users?”

A suggestion for

  • Enhanced more involved clinical trials
  • Provide individuals added flexibility when interacting with a model, for example inputting their own test data
  • More interaction between user and model generators
  • Determining in which situations call for interpretable AI versus explainable (for instance predicting which patients will require dialysis after kidney damage)

Other articles on AI/ML in medicine and healthcare on this Open Access Journal include

Applying AI to Improve Interpretation of Medical Imaging

Real Time Coverage @BIOConvention #BIO2019: Machine Learning and Artificial Intelligence #AI: Realizing Precision Medicine One Patient at a Time

LIVE Day Three – World Medical Innovation Forum ARTIFICIAL INTELLIGENCE, Boston, MA USA, Monday, April 10, 2019

Cardiac MRI Imaging Breakthrough: The First AI-assisted Cardiac MRI Scan Solution, HeartVista Receives FDA 510(k) Clearance for One Click™ Cardiac MRI Package


Read Full Post »

wearables and patient centric data

Larry H. Bernstein, MD, FCAP, Curator





The Science of the Art of Medicine Author Interviewed

; John E. Brush, MD

Interview with Robert A. Harrington, MD






Robert A. Harrington, MD: Hi. This is Bob Harrington from Stanford University on Medscape Cardiology and the heart.org. Thanks for joining us today. Over the last couple of months, I have interviewed a couple of physician authors who have commented on some of the issues of the time, whether that’s related to the electronic health records, as in our discussions with Bob Wachter, or the revolution that’s taking place with wearables and patient centric data, as with my conversation with Eric Topol. These shows have been well received, and those conversations were useful to the community to have a frame of reference not just in regard to the daily practice of medicine but also in terms of how the bigger issues in society around digital health, wearables, electronic records, etc., might influence the practice of medicine.

Today we’re also going to interview a physician author but with a different approach to some of the issues within medicine. He is trying to get at how physicians and other clinical providers take what we learn from the literature, from scientific studies that have been done, and use some combination of statistics coupled with the art of caring for other human beings in a way that ends up with first-rate medical care. And this author not only wondered how we do that but also how we begin to think about teaching that to our trainees, to our students, our residents, our fellows, etc. I’m really pleased to have with me today the author of the book, The Science of the Art of Medicine, which is just coming out in its second edition. The author is Dr John Brush. John is a professor of medicine at Eastern Virginia Medical School and a practicing cardiologist in Norfolk, Virginia. John and I are friends and colleagues through his service as a recent board of trustee member for the American College of Cardiology (ACC). John, thanks for joining us here today on Medscape Cardiology.

John E. Brush, MD: Thanks very much, Bob. It’s a pleasure being here, and I look forward to our conversation.

Cognitive Psychology

Dr Harrington: You heard my opening remarks reflecting on you as a physician author. What made you write this book? What were some of the issues that compelled you to put pen to paper, so to speak, and try to formulate some view of the world in medicine as you saw it?

Dr Brush: One of the origins of this book was actually my work with the ACC related to quality of care and quality and care initiatives. I was chairman of the Quality Strategic Direction Committee for 3 years and really got thinking about quality and why people make errors. What leads to good judgment? What leads to bad judgment? And that led me to stumble upon the cognitive psychology literature. When I was in college, there was no such thing as cognitive psychology; it was mostly behavioral psychology. The field of cognitive psychology has arisen in the past 30 years. Cognitive psychologists are interested in understanding how people make decisions. Doctors make decisions every single day, and yet we don’t teach anything about cognitive psychology, and we don’t learn anything about it in medical school. It seemed like it was a relevant area to think about, and so I jumped in. I taught myself cognitive psychology. I bought book after book after book and just decided I was going read about it.
What cognitive psychologists have figured out is that we’re not purely rational; we think we are, but we make mistakes. We fall into typical traps. We have biases because we use heuristics or mental short cuts to try to make rapid decisions under conditions of uncertainty. If you think about it, that’s what doctors do every single day: make rapid decisions under conditions of uncertainty. I wanted to learn more about it myself. And then the more I read about it, I jokingly said to my wife, “You know, I could write a book on this stuff.” And then I stumbled on this ibook authoring tool, and I figured out that it’s actually easy to write an ibook—easier than I thought. And so I said, “I am going to write a book about it.” I thought it was going to take me 5 years; it took me about 3 months because I was so consumed by this.

Dr Harrington: Before you dive further, let me ask you a couple of things about cognitive psychology because it is interesting that you took this observation of people making errors and then did the deeper dive, which I applaud you for. I’m thinking of some other authors in this area that some of our listeners might know about, such as Dan Ariely from Duke and the book,Predictably Irrational. Would you put that in the context of cognitive psychology?

Dr Brush: Yes, he’s a well-known cognitive psychologist. There’s also Daniel Kahneman who wrote, Thinking Fast, Thinking Slow. Daniel Kahneman is from Princeton and a very prominent writer in cognitive psychology. A guy named Gerd Gigerenzer from Germany has written books on risk and how to evaluate risk—basically how to think about probability. His books were very influential for me. There’s Herbert Simon who was a computer scientist in Pittsburgh, and he is generally thought of as the father of cognitive psychology; he was at Carnegie Mellon. Cognitive psychology is also referred to as behavioral economics, by the way, and both Herbert Simon and Daniel Kahneman have won the Nobel Prize in Economics.

 Dr Harrington: That’s what I wanted you to get at for the audience—that it’s not just interesting, but it’s useful to read outside of medicine and then bring those ideas into how we think about medicine. You’ve hit upon something in that we don’t teach our students and resident how people actually make decisions, but, as you’ve described, that’s what a clinician does every day. You said that the ibook was an easy thing to do. Why did you pick a book and not write a commentary for a journal or give a lecture on the topic? Why did you go right into a book? That’s a pretty big leap.

Natural Bayesians Placing Bets

Dr Brush: The topic is too broad to cover in a single lecture or single commentary. I did write a series of blog posts that were posted on cardioexchange.org. Harlan Krumholz invited me to do that, and it was very helpful because it enabled me to collect my thoughts and to test out ways of explaining this. One of the challenges is explaining a subject matter that is outside medicine. You’re bringing a new area to people’s attention, and so you have to figure out how to go about explaining it. I’d done this series of blog posts; I did a series of lectures for our residents here in Norfolk. And I have to tell you that the first lectures I did on this utterly failed because they were dull, they were boring. I had to figure out how to make it interesting because if you discuss it in the setting of a patient and you study it on rounds, it’s absolutely fascinating. But if you talk about it in isolation, it’s like a statistics course in isolation where the subject matter is too dry. The way you make it interesting is by relating it to patients, so I started collecting examples, and there are simple examples in cardiology.

Cardiology is the ideal place to teach medical reasoning because we almost get immediate feedback. You can put your money down and make your bets as to whether somebody is having a myocardial infarction or whether somebody has heart failure or whether somebody has a pulmonary embolism. Generally by the next day you find out, and so you get this immediate feedback where you can train your intuition. We have so many numerical things from troponin levels, D-dimer, stress tests, and what have you. You can begin to develop estimates of probability so that you can start to calibrate your intuition. And there’s no better place in medicine than cardiology to that as an exercise.

Dr Harrington: One of the things I love about the book is that you spend time, particularly at the beginning of the book, going through the basics of statistics that help us in the practice of medicine. And one area you spend time talking about is Bayes’ theorem and the notion that while we use both frequentist and Bayesian statistics in medicine, in many ways clinicians are natural Bayesians, aren’t we?

Dr Brush: We are, but we don’t think hard enough about what that means. We wander into being Bayesians, and we pick that up in our training. It’s a good idea to step back, dig a little bit deeper, and ask exactly what that means. The cognitive psychologists tell us that we make mistakes by deviating from a normative mode of thinking in one of two ways. We either do things that are illogical, or we make bad probability estimates. I wanted to think about the logic of what we do as well as the probability.

 I talk a little bit in the book about deductive reasoning and inductive reasoning, but really what we do is something called abductive reasoning, where we reason to the most plausible hypothesis. Most of the time, we don’t really know. If somebody comes in with heart failure, we think it may be untreated hypertension or alcoholism or whatever, but we don’t really know. We reason towards the most plausible hypothesis. When we’re using Bayesian reasoning, we’re using probability estimates, and so knowing something about probability is a good idea. We think that we know about probability because we play cards, we throw dice, we play games of chance all the time, but probability can be simple and complex and complementary and cumulative probability and conditional probability. Bayesian reasoning is all about conditional probability.

We think like Bayesians, but we never do the calculations. Instead we use a heuristic called anchoring and adjusting. Anchoring and adjusting is a natural way that you use Bayesian reasoning where you come up with an anchor, which is your pretest probability, and you adjust that anchor (that pretest probability) based on new information to give you a posttest probability. We intuitively know how to use this heuristic called anchoring and adjusting, and that’s actually what we do in practice, rather than pulling out our calculator and calculating the Bayesian probabilities.

Dr Harrington: What you’re describing in part is what makes the great clinician, and the great clinician as you described has to have a grounding in statistical methods, has to understand what probability is and how one thinks about it. At the same time, the great clinician has to have a body of experience upon which to anchor that probability; and then finally, the great clinician has to be able to be observant and trust his or her intuition. Is that a reasonable way to put it?

Teaching Good Decision-Making

Dr Brush: When you look back on your colleagues or mentors who just seemed to get it, one characteristic that describes the person who just seemed to get it is that they made good bets. They had a good sense of what’s most likely, what’s least likely. They were savvy about how they made those calculations. They had good intuition about how they made those judgments.

 How do you teach that? Well, you can teach people about the processes and give them an idea of where they want to be at the end of the day. If you can break it down a little bit, it helps. You can use simple numbers. If you simplify the numbers, you can use the numbers to calibrate your intuition. There’s something in medicine that we are leaving out. We talk all the time about sensitivity and specificity, and yet when you ask physicians, “give me a quick definition of sensitivity and specificity,” they’re not able to do it. You lose track of what’s in the numerator and what’s in the denominator.

People get all balled up with those definitions that sensitivity relates to true positive and specificity to true negative. That’s a little bit more descriptive of what you’re actually talking about. But there’s something even better, and that’s likelihood ratios because a positive likelihood ratio is the probability of something divided by the probability that something is not there. It’s the evidence for something divided by the evidence against it, and so it’s a dimensionless number. You don’t even have to keep track of what’s in the numerator or the denominator because it’s a dimensionless number.

 Likelihood ratios are incredibly powerful. Say you’re trying to teach a trainee how important a positive troponin is. How important is a positive nuclear stress test? How important is a D dimer? If a D dimer test is negative, that is a very important negative piece of information; but if it’s positive, it’s not quite so strong. That’s because the positive likelihood ratio for a D dimer is about 2, and the negative likelihood ratio is more like 0.1. You can use likelihood ratios to really hone your intuition, which is very useful when you’re trying to import years of wisdom to new trainees. You can’t wait 30 years for them to finally get it; you want then to get it on Day 1.
 We’ve been talking about diagnosis so far, but for therapeutics you need to figure out how to calibrate your intuition on treatment decisions. The inverse of the absolute risk reduction is the number needed to treat, and the smaller the better. If you’ve got some sense of what a good number needed to treat is, then suddenly you can calibrate your intuition about what’s really important and what’s not so important. Beta blockers and aspirin and angiotensin-converting enzyme inhibitors are really important; maybe glycoprotein IIb/IIIa antagonists are less important. The number needed to treat gives you a way to gauge the importance of therapeutic interventions.

Dr Harrington: I love the direction you’re going, which is to take what we do and recognize that there’s a science behind it. For a lot of the therapeutics you just mentioned, there are clinical trials that inform us about what those numbers needed to treat are and what the numbers needed to harm are. But as you rightly said, the art of medicine is understanding who is likely to benefit from this particular therapy when you make that bet. We’re not yet in this era of precision health of being able to say, “Based on your characteristics (which may include genomic characteristics), you are more likely to benefit from this and less likely to benefit from that.” But you’re helping move us to that, and I love that you’re helping us think about a framework for teaching it. You clearly wrote this as a teacher to students, and whether that’s a teacher to a colleague like me or a teacher to residents, students, and fellows in some ways doesn’t matter. What’s been the reaction from the student community, “student” being broadly defined?

Dr Brush: I use it to teach internal medicine residents. I have a new internal medicine resident with me each month; we have 12 locally and 12 months of the year, so I have a new one each month. I have them read the book at night, and we talk about the book during the day, but we talk about the book in the context of taking care of patients. In a way, it’s an inverted classroom where they’re getting the lecture from me by reading the book (it would be odd for me to sit and lecture them on these things), but we talk about it in the setting of a real patient. We’ll go in and see a patient in the emergency department who’s short of breath, and maybe our pretest probability that its heart failure is 50/50, and so the pretest odds is 1. You can then multiply that by the likelihood ratios of paroxysmal nocturnal dyspnea and orthopnea and rales and S3 gallop and congestion or a chest x-ray. And you end up with a posttest probability that you can calculate in your head; when we do that during the day, it suddenly makes what they’ve read at night very real. I’ve gotten really good feedback on that. It’s one of those things where you hit your forehead and go, “Oh, now I get it.”

 You can see the light bulb turning on, and they really appreciate it. It also helps them get a handle on uncertainty. Uncertainly scares the heck out of us when we’re in training. Some of us run away from it and create this illusion of certainty, but—let’s face it—the uncertainty and the ambiguity of medicine scares the heck out of all of us. But it really scares people during training, and you don’t know whether you don’t know it or whether it’s unknown to everybody. You don’t know the level of uncertainty of things, and by addressing that straight on, you can try to get a handle on how you can reason through that uncertainty and how you can move from a place where you don’t know to a place where you have a better idea but also understand that there are some things that you’ll never know for sure but you’ve got to get as close as you can. For the residents that I work with, that takes a little bit of pressure off and helps them. It helps them grasp what the challenge is, and the challenge is learning as much as they possibly can, getting as much experience as they can so that they can put it all into action in a way that’s systematic and organized and tends to make sense.

Dr Harrington: John, my last question for you: What have you learned from this? You entered this as a student trying to understand decision-making, and it led you to an exploration of cognitive psychology. You felt that it would help make you a better teacher, and you wrote a book. What did you learn from the whole exercise of the last few years?

Dr Brush: I’ve learned so much about cognitive psychology, about probability, about logic, and that has helped me be a better decision-maker in general. Even beyond medicine, but certainly it’s made me a better clinician and a better teacher. I’ve been able to articulate what I’m thinking in a way that makes better sense to students. What I’ve learned, though, is that we’re not teaching this, and we’re also not testing on this. It’s much easier to test on content; you can turn some medical fact into a multiple choice question a whole lot easier than you can figure out whether somebody actually gets it and is able to put the knowledge into action. We don’t want to just impart knowledge; we want to impart wisdom. We want to be able to use knowledge wisely. We’ve now got this rush of information that’s coming at as from every direction over the Internet and in so many different ways. We need to turn that information into knowledge that helps us get closer to the truth, and to do that we’ve got to learn how to use the knowledge wisely. As a community of physicians, of professionals, and also a community of academicians, we need to think about how to teach this stuff. How do we test it? How do we develop competencies in this area? For me, it’s created a whole new way of looking at medical education and really what matters and what we really ought to be concentrating on.

 Dr Harrington: This has been a fascinating conversation, John, and I applaud you for taking the initiative not just to learn the background material but to write the book and share it with the rest of us. And I certainly recommend to our listener audience John Brush’s book, The Science of the Art of Medicine. You can get it through iBooks or you can get a hard copy of it. So my guest today has been Dr John Brush, a professor of medicine at Eastern Virginia Medical School and a practicing cardiologist in Norfolk, Virginia. John, thanks for joining us on Medscape Cardiology.


Read Full Post »

%d bloggers like this: