Explainable AI - making ML and DL models more interpretable
1 like404 views
The document discusses explainable AI (XAI) and making machine learning and deep learning models more interpretable. It covers the necessity and principles of XAI, popular model-agnostic XAI methods for ML and DL models, frameworks like LIME, SHAP, ELI5 and SKATER, and research questions around evolving XAI to be understandable by non-experts. The key topics covered are model-agnostic XAI, surrogate models, influence methods, visualizations and evaluating descriptive accuracy of explanations.
![2 3
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
[Kim et. al., 2018]
Zebra
(0.97)
How important is the notion of “stripes” for this prediction?
Testing with Concept Activation Vectors (TCAV) is an interpretability method to understand what signals
your neural networks models uses for prediction.
https://github.com/tensorflow/tcav
Pattern representation plays
a key role in decision making
from both images and text.](https://image.slidesharecdn.com/explainableaiadityabhattacharya-210716143326/85/Explainable-AI-making-ML-and-DL-models-more-interpretable-23-320.jpg)
![2 4
E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
[Tan et. al., 2019]
Model
Predictions
Label 1
Label 1
.
Label 2
.
v1, v2.
v11,
v12
.
Data
Explainer
Interpretable Mimic Learning – Compressing information from Deep Networks to Shallow Network](https://image.slidesharecdn.com/explainableaiadityabhattacharya-210716143326/85/Explainable-AI-making-ML-and-DL-models-more-interpretable-24-320.jpg)
![2 5
Explainable AI: Making ML and DL models more interpretable
What features need to be changed and by how much to flip a model’s prediction?
[Goyal et. al., 2019]](https://image.slidesharecdn.com/explainableaiadityabhattacharya-210716143326/85/Explainable-AI-making-ML-and-DL-models-more-interpretable-25-320.jpg)
More Related Content
Similar to Explainable AI - making ML and DL models more interpretable (20)
Explainable AI - making ML and DL models more interpretable
- 1. Explainable AI – Making ML and DL models more interpretable Explainable AI – Making ML and DL models more interpretable
- 2. About Me E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 2 Aditya Bhattacharya I am currently working as the Lead AI/ML Engineer at West Pharmaceutical Services with the responsibility of leading and managing a global AI team and creating AI products and platforms at West. I am well seasoned in Data Science, Machine Learning, IoT and Software Development. and has established the AI Centre of Excellence and worked towards democratizing AI practice for West Pharmaceuticals and Microsoft. In the Data Science domain, Computer Vision, Time-Series Analysis, Natural Language Processing and Speech analysis are my forte. Apart from my day job, I am an AI Researcher at an NGO called MUST Research, and I am one of the faculty members for the MUST Research Academy : https://must.co.in/acad Website : https://aditya-bhattacharya.net/ LinkedIn: https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/
- 3. Key Topics 1. Necessity and Principles of Explainable AI 2. Model Agnostic XAI for ML models 3. Model Agnostic XAI for DL models. 4. Popular frameworks for XAI 5. Research Questions to consider E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3
- 4. Necessity and Principles of Explainable AI
- 5. 5 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 6. 6 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 7. 7 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 8. 8 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e XAI Trace model prediction from logic of math to nature of data Understand the reasoning behind each model predictions Understand the model using which AI decision making is based Traceable AI Reasonable AI Understand able AI
- 9. 9 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 10. 1 0 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 11. Model Agnostic Results Visualizations Influence Methods Example Based Methods Knowledge Extractions 1 1 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Using Surrogate models like linear models or decision trees to explain complex models Estimates the importance or relevant features. Extracting statistical information from input and the output Select instances of the datasets that explains the behaviour of the model
- 12. 1 2 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Problem, Data, Audience Post Hoc Analysis Model Predictive Accuracy Descriptive Accuracy Iterative Explainability
- 13. Model Agnostic XAI for ML models
- 14. 1 4 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 15. 1 5 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 16. 1 6 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Explainer Surrogate Models Predictions Blackbox ML Model
- 17. 1 7 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e
- 18. E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 1 8 Prediction: Deny Loan Loan Application Suggestion: Increase your salary by 50K & pay your credit card bills on time for next 3 months Predictive Model Loan Applicant Counterfactual Generation Algorithm
- 19. Model Agnostic XAI for DL models 1 9
- 20. 2 0 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Understanding flow of information through gradient flow between the layers of Deep Neural Network model using the following approaches: 1. Saliency Maps 2. Guide Backpropagation 3. Gradient Class Activation Methods • Layer GRAD CAM • Layer Conductance using GRAD CAM • Layer Activation using GRAD CAM Saliency Maps Guided Backprop GRAD CAM Layer Conductance Layer Activation
- 21. Can such explainability methods be applied for complex models? 2 1 Explain ab l e AI: Making ML and DL models more interpr et a b l e
- 22. 2 2 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Image Captioning using Attention based Encoder-Decoder Architecture
- 23. 2 3 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e [Kim et. al., 2018] Zebra (0.97) How important is the notion of “stripes” for this prediction? Testing with Concept Activation Vectors (TCAV) is an interpretability method to understand what signals your neural networks models uses for prediction. https://github.com/tensorflow/tcav Pattern representation plays a key role in decision making from both images and text.
- 24. 2 4 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e [Tan et. al., 2019] Model Predictions Label 1 Label 1 . Label 2 . v1, v2. v11, v12 . Data Explainer Interpretable Mimic Learning – Compressing information from Deep Networks to Shallow Network
- 25. 2 5 Explainable AI: Making ML and DL models more interpretable What features need to be changed and by how much to flip a model’s prediction? [Goyal et. al., 2019]
- 26. Popular frameworks for XAI Explain ab l e AI: Making ML and DL models more interpr et a b l e
- 27. Popular frameworks for XAI 2 7 LIME Local Interpretable Model-agnostic Explanations is interpretability framework that works on structured data, text and image classifiers. SHAP SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. ELI5 Explain like I am 5 is another popular framework that helps to debug machine learning classifiers and explain their predictions. SKATER Skater is a unified framework for XAI for all forms of models both globally(inference on the basis of a complete data set) and locally(inference about an individual prediction). TCAV Testing with Concept Activation Vectors (TCAV) is a new interpretability method to understand what signals your neural networks models uses for prediction. Explainable AI: Making ML and DL models more interpretable
- 28. 2 8 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e • Behind the workings of LIME lies the assumption that every complex model is linear on a local scale. LIME tries to fit a simple model around a single observation that will mimic how the global model behaves at that locality. • Create the perturbed data and predict the output on the perturbed data • Create discretized features and find the Euclidean distance of perturbed data to the original observation • Convert distance to similarity score and select the top n features for the model • Create a linear model and explain the prediction
- 29. 2 9 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e The lime package is on PyPI. `pip install lime`
- 30. 3 0 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e There is a high-speed exact algorithm for tree ensemble methods (Tree SHAP arXiv paper). Fast C++ implementations are supported for XGBoost, LightGBM, CatBoost, and scikit-learn tree models! • SHAP assigns each feature an importance value for a particular prediction. • Its novel components include: the identification of a new class of additive feature importance measures, and theoretical results showing there is a unique solution in this class with a set of desirable properties. • Typically, SHAP values try to explain the output of a model (function) as a sum of the effects of each feature being introduced into a conditional expectation. Importantly, for non-linear functions the order in which features are introduced matters. SHAP can be installed from PyPI
- 31. 3 1 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e The following figure from the KDD 18 paper, Consistent Individualized Feature Attribution for Tree Ensembles summarizes this in a nice way! SHAP Summary Plot SHAP Dependence Plots
- 32. 3 2 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Available from pypi. pip install eli5 Check docs for more.
- 33. 3 3 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e SKATER provides an unified framework for both Global and Local Interpretation. Feature Importance Partial Dependency Plots LIME integration for explanability Project Link:
- 34. 3 4 E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e Testing with Concept Activation Vectors (TCAV) is a new interpretability method to understand what signals your neural networks models uses for prediction. What's special about TCAV compared to other methods? TCAV instead shows importance of high level concepts (e.g., color, gender, race) for a prediction class - this is how humans communicate! TCAV gives an explanation that is generally true for a class of interest, beyond one image (global explanation). For example, for a given class, we can show how much race or gender was important for classifications in InceptionV3. Even though neither race nor gender labels were part of the training input! pip install tcav https://github.com/tensorflow/tcav
- 35. E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3 5 The Concept Activation Vectors (CAVs) provide an interpretation of a neural net’s internal state in terms of human-friendly concepts. TCAV uses directional derivatives to quantify the degree to which a user-defined idea is vital to a classification result–for example, how sensitive a prediction of “zebra” is to the presence of stripes. TCAV essentially learns ‘concepts’ from examples. For instance, TCAV needs a couple of examples of ‘female’, and something ‘not female’ to learn a “gender” concept. The goal of TCAV is to determine how much a concept (e.g., gender, race) was necessary for a prediction in a trained model even if the concept was not part of the training.
- 36. Research question to consider …
- 37. All these frameworks are great and can bring interpretability to a great extent, but can non- expert consumers of AI models interpret these interpretability methods? 3 7 Explain ab l e AI: Making ML and DL models more interpr et a b l e
- 38. Summary • Why is Explainable AI (XAI) important? • Commonly used Model Agnostic XAI for ML models • Commonly used Model Agnostic XAI for DL models. • Popular frameworks for XAI • Can we evolve XAI and extend explainability to non-expert users? 3 8 Explain ab l e AI: Making ML and DL models more interpr et a b l e
- 39. Thank you Aditya Bhattacharya https://aditya-bhattacharya.net/ [email protected] https://www.linkedin.com/in/aditya-bhattacharya-b59155b6/ E x p l a i n a b l e A I : M a k i n g M L a n d D L m o d e l s m o r e i n t e r p r e t a b l e 3 9