Train, explain, acclaim. Build a good model in three steps
2 likes311 views
The document discusses a workshop focused on training machine learning models using automated machine learning (AutoML) techniques, specifically highlighting methodologies like sequential model-based optimization and ensemble learning. It emphasizes the importance of explainable AI, advocating for model transparency through approaches like local model approximations and Shapley value explanations. Additionally, it touches on the iterative nature of model development, including validation, interpretation of predictions, and the need to monitor model performance over time.
Train, explain, acclaim. Build a good model in three steps
- 1. Train, explain, acclaim Build a good model in three steps AI & NLP workshop day Przemysław Biecek Samsung SRPOL Warsaw University of Technology
- 4. Doing your n+1 dataset …, should you take into account what you have learnt on previous datasets?
- 6. Defaults (package defaults (Def.P) and optimal defaults (Def.O)), tunability of the hyperparameters with the package defaults (Tun.P) and our optimal defaults (Tun.O) as reference and tuning space quantiles (q0.05 and q0.95) for different parameters of the algorithms
- 8. Sequential Model-based Algorithm Configuration The open-source solution of AAD Freiburg uses a heterogeneous ensemble of learning machines (AutoSklearn (Feurer et al., 2015a,c)) combining the machine learning library scikit-learn (Pedregosa et al., 2011) with the state-of-the-art SMBO method SMAC to find suitable machine learning pipelines for a data set at hand. This is essentially a reimplementation of Auto-WEKA. To speed up the optimization process they employed a metalearning technique (Feurer et al., 2015b) which starts SMAC from promising configurations of scikit-learn. Furthermore, they used the outputs of all models and combined these into an ensemble using ensemble selection.
- 9. This work has been supported in part by the Defense Advanced Research Projects Agency (DARPA) Data-Driven Discovery of Models (D3M) Program.
- 14. This work has been supported in part by the Defense Advanced Research Projects Agency (DARPA) Data-Driven Discovery of Models (D3M) Program.
- 15. https://github.com/hibayesian/awesome-automl-papers
- 17. Auto-Keras auto-sklearn automl-gs Auto-Weka FeatureTools h2o automl Ludwig mljar-supervised Neural Network Intelligence (NNI) tpot TransmografAI Auto_ml
- 19. Explain with Interpretable Machine Learning
- 22. • “You don’t see a lot of skepticism,” she says. “The algorithms are like shiny new toys that we can’t resist using. We trust them so much that we project meaning on to them.” • Ultimately algorithms, according to O’Neil, reinforce discrimination and widen inequality, “using people’s fear and trust of mathematics to prevent them from asking questions”. https://www.theguardian.com/books/2016/oct/27/cathy-oneil-weapons-of-math- destruction-algorithms-big-data !22 Cathy O'Neil: The era of blind faith in big data must end black boxes Why do we need explanations for complex models?
- 23. Right to explanation !23 Why do we need explanations for complex models?
- 29. !29 Local Model approximations "Why Should I Trust You?" Explaining the Predictions of Any Classifier. Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin (2016). https://arxiv.org/pdf/1602.04938.pdf Port to R: Thomas Lin Pedersen (2017) https://github.com/thomasp85/lime Other implementations: live (Staniak, Biecek 2018) and iml (Molnar 2018) A different approach to model explanation is to locally approximate the complex black-box model with an easier to interpret white-box model constructed on interpretable features.
- 30. !30 Local Model approximations 1.Generate a fake dataset around x. 2.Use black-box estimator to get target values y. 3.Train a new white-box estimator for (y,x). 4.Check prediction quality of a white-box classifier. 5.Use white-box estimator as an explanation of black-box model. Properties: model-agnostic interpretable representation local fidelity
- 32. LIME / live vs Break Down Model debugging
- 33. Biecek P (2018). “DALEX: Explainers for Complex Predictive Models in R.” Journal of Machine Learning Research, 19(84), 1-5. URL:http://jmlr.org/papers/v19/18-416.html>
- 34. What would you explain? https://kmichael08.github.io
- 35. What would you explain? https://kmichael08.github.io What If? Why?
- 37. https://www.encyclopedia-titanica.org/ What are the odds of surviving?
- 38. What If?
- 39. Ceteris Paribus Individual Conditional Expectations
- 40. Ceteris Paribus Individual Conditional Expectations
- 45. Why?
- 46. iBreakDown: Uncertainty of Model Explanations for Non-additive Predictive Models Alicja Gosiewska, Przemyslaw Biecek (2019) https://arxiv.org/abs/1903.11420v1
- 48. SHAP (SHapley Additive exPlanations) Lundberg (2017) IME complexity is O(2 p ). Shapley values are known for some time and we have methods to approximate them efficiently.
- 53. What If? Interactive with D3
- 54. What If? Comparison of models between languages
- 55. Acclaim With Human Centered AI
- 61. On Explainable Machine Learning Misconceptions and A More Human-Centered Machine Learning; Patrick Hall
- 62. Predictive Models: Visual Exploration, Explanation and Debugging Production Development Concept ValidateForge Model debugging Model development is an iterative process. Each iteration brings new insights. Early phases: Crisp modeling, general understanding of the problem. Medium phases: Selective modeling, here we select the best type of model. Late phases: Fine tuning of model parameters or variable engineering . In each iteration model development starts with some concepts, ideas, then the model is trained and finally model needs to be validated. Predictions need to be explained. Here the instance level explanation helps. With time the model performance may deteriorate, thus it requires constant monitoring, e.g. with the drifter package. Drop-out loss Variable importance GBM baseline ditrict surface floor construction.year no. rooms full model 250 500 750 1000 1250 3000 3200 3400 surface prediction 3600 3800 Surface 6019 100 148 Random Forest feature influence Variables attributions GBM intercept district: Srodmiescie surface: 22 no.rooms: 2 construction.year: 2005 floor: 1 prediction Random Forest intercept district: Srodmiescie surface: 22 no.rooms: 2 construction.year: 2005 floor: 1 prediction LM intercept district: Srodmiescie surface: 22 no.rooms: 2 construction.year: 2005 floor: 1 prediction 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 2046 2614.9 +358 +160 +78 -39.5 +12.4 2800 2425 -338 -112 +74 -53 +26 2378 2324.9 -239 +160 +68 +39.5 -12.4 feature influence Variables attributions GBM intercept district: Srodmiescie surface: 22 no.rooms: 2 construction.year: 2005 floor: 1 prediction 2000 2100 2200 2300 2400 2500 2600 2700 2046 2614.9 +358 +160 +78 -39.5 +12.4 3000 3200 3400 surface prediction 3600 3800 Surface 6019 100 148 Random Forest 3000 3200 3400 2 4 6 prediction 3600 3800 3000 3200 3400 3600 3800 Surface 1920 1940 1960 1980 2010 no.roomsfloor surfaceconstruction.year 4020 80 120 144 2.50.8 1.15.0 7.5 11.2 Variable selection Feature engineering Random Forest Factor Merger Srodmiescie Ochota Mokotow Zoliborz Ursus Bielany Bemowo Wola Ursynow Praga 0 2000 4000 group frequencyname price mean 5109.19 3954.83 3946.96 3918.55 3058.52 3045.79 3028.58 3011.69 3009.72 2991.48 nr 1 2 2 3 4 4 5 6 6 7 Prediction explanations What-If analysis Concept drift detection explain
- 64. Like it? Let us know! Find a bug? Fire an issue!