Stanford University Neuroradiology Talks: Personalizing and Customizing AI Explanations for Clinicians

Editor's Notes

• #2: Hello, my name is Francisco Calisto and I am presenting our work in progress, titled as the “Personalizing and Customizing AI Explanations for Clinicians” applied to breast cancer.
• #3: … I am a researcher from Portugal and currently a visiting scholar of the Human-Computer Interaction Institute at CMU.
• #4: While being involved with a great team from Human-Computer Interaction researchers to Machine Learning and Radiologists…
• #5: … we are trying to surpass some of the breast cancer challenges.
• #6: Every year about 10 million women die from breast cancer. They can be our grandmothers, wifes or even our daughters.
• #7: These numbers are just 10% of suspected women from having breast cancer, but…
• #8: … radiologists are yielding about 40% of false-positives, which can lead women to unnecessary biopsies.
• #9: The cancer burden can be reduced through early detection of cancer, as well as management of patients who develop cancer.
• #10: For an early detection, breast cancer is usually diagnosed with several medical imaging modalities.
• #11: Across this multimodality workflow, clinicians perform a kind of loop for the inspection of lesions.
• #12: From here, clinicians are using the BI-RADS for classification of the lesion severity. However, the severity classification is not trivial and consensus is not always achieved.
• #13: AI can help on this…
• #14: … but the “black box” nature of AI introduces a large element of opacity into decision-making. Such problem can be solved by the introduction of eXplainable-AI (XAI) techniques.
• #15: Prior work in behavioral sciences for medical relation extraction substantiate the disagreement relations between inter-variability and intra-variability.
• #16: Disagreement relations are addressed as a function of three phenomena: (1) differences among clinical professionals, such as the medical background and bias; (2) heterogeneous characteristics of the dataset, such as noisy and heterogeneous modalities; and (3) the nature of the diagnostic guidelines, with subjective and ambiguous classifications of the BI-RADS. In fact, clinical experts often rely on complex viewing technology to inspect medical data.
• #17: In medical imaging, Doctors need to trust that AI is being used safely, and for their benefit during decision-making. Intrinsically, humans feel the need to understand how decisions are made.
• #18: Specifically, decision-making on the radiology reading room is done by professionals with different medical experiences and clinical profile characteristics.
• #19: We expect that strategies, such as adapting the agent communication by personalizing and customizing the AI explanations to each clinician, will promote the unbiased behaviour per each category of professional experience, improving the rates of false-positives and false-negatives during diagnostic.
• #20: … which will influence the burning rates of clinicians.
• #21: Since AI models are developed and measured using a pipeline of several characteristics, we studied the deliberation process of clinicians during diagnosis without and with the introduction of an intelligent agent. So that we can understand the expectation levels of tolerance on satisfaction and acceptance, as well as tolerance of the model accuracy.
• #22: Currently, our user studies are involving 52 clinicians from nine public and private medical institutions in Portugal.
• #23: We are training our AI models with four hundred ninety one patients from a multimodality dataset of medical images, such as MammoGrams, UltraSound and MRI.
• #24: Our demo hour…
• #25: While using our agent, 98% of the 52 clinicians answered that they do understand what the AI system is thinking...
• #26: … and 93% trust on the system capability.
• #27: We also divided our participant results into groups of inter-variability and intra-variability. Not only, the inter-variability was reduced between groups of patients, but also the intra-variability was reduced for the groups of interns, juniors, middles and seniors…
• #28: … as well as, we could improve the final clinician’s performance due to the Clinician-AI diagnostic.
• #29: With this studies, we could understand the behaviour characteristics of each medical group, while diagnosing the different groups of patients. Now, we have information from whom and when should the intelligent agent provide a suggestion and how an explanation will influence interpretability and the decision-making.
• #30: Indeed, as humans, radiologists are exposed to fatigue levels, where performance is related to environmental factors, such as the number of working hours. As a future direction, we will study and implement a system that will give the hardest patient cases in the beginning of the clinician’s shift, while the most trivial cases are diagnosed in the end of the shift. For that, we will need to study and implement an ambiguous-aware AI system that will classify all cases in our dataset as more or less ambiguous to diagnose. This idea will transform the way radiologists are screening and diagnosing breast cancer.
• #31: To conclude, we identified how different levels of suggestions will impact the radiologists' decision-making process. We developed a system that enables clinicians to accept or reject the AI breast analysis by adapting the communication through personalization and customization to each medical professional experience. We propose a series of recommendations and future directions for the development of intelligent agents in breast cancer diagnosis.
• #32: During this presentation, about 5 women died for breast cancer, while other 10 went to unnecessary biopsies in Portugal, making our NHS lose 50 thousand euros in just 10 minutes…
• #33: I humbly ask the help of the most noble professions, healthcare professionals, to help us cross this path together!
• #34: Thank you! Any question?
• #35: Please be free to contact me.