Explainable AI in Industry (WWW 2020 Tutorial)

Explainable AI in Industry
WWW 2020 Tutorial
Krishna Gade, Sahin Cem Geyik,
Krishnaram Kenthapadi, Varun Mithal,
Ankur Taly
https://sites.google.com/view/www20-explainable-ai-tutorial 1

Agenda
● Part I: Introduction and Motivation
○ Motivation and Need for Explainable AI
○ Challenges for Explainable AI @ Scale
● Part II: Explainable Machine Learning
○ Overview of Explainable AI Techniques
● Part III: Case Studies from Industry
○ Applications, Key Challenges, and Lessons Learned
● Part IV: Open Problems, Research Challenges, and Conclusion
2

Explanation - From a Business Perspective
4

COMPAS recidivism black bias
… but not only Critical Systems (1)

community.fico.com/s/explainable-machine-learning-
challenge
https://www.ft.com/content/e07cee0c-3949-11e7-821a-6027b8a20f23
▌Finance:
• Credit scoring, loan approval
• Insurance quotes

Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, Noemie Elhadad: Intelligible Models
for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. KDD 2015: 1721-1730
Patricia Hannon ,https://med.stanford.edu/news/all-news/2018/03/researchers-say-use-of-ai-in-medicine-
raises-ethical-questions.html
▌Healthcare
• Applying ML methods in medical care
is problematic.
• AI as 3rd-party actor in physician-
patient relationship
• Responsibility, confidentiality?
• Learning must be done with available
data.
Cannot randomize cares given to
patients!
• Must validate models before use.

Black-box AI creates business risk for Industry

Internal Audit, Regulators
IT & Operations
Data Scientists
Business Owner
Can I trust our AI
decisions?
Are these AI system
decisions fair?
Customer Support
How do I answer this
customer complaint?
How do I monitor and
debug this model?
Is this the best model
that can be built?
Black-box
AI
Why I am getting this
decision?
How can I get a better
decision?
Poor Decision
Black-box AI creates confusion and doubt

Explanation - From a Model Perspective
12

Why Explainability: Debug (Mis-)Predictions
13
Top label: “clog”
Why did the network label
this image as “clog”?

14
Why Explainability: Improve ML Model
Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18

Why Explainability: Verify the ML Model / System
15

16
Why Explainability: Learn New Insights

17
Why Explainability: Learn Insights in the Sciences

Explanation - From a Regulatory Perspective
18

Immigration Reform and Control Act
Citizenship
Rehabilitation Act of 1973;
Americans with Disabilities Act
of 1990
Disability status
Civil Rights Act of 1964
Race
Age Discrimination in Employment Act
of 1967
Age
Equal Pay Act of 1963;
Civil Rights Act of 1964
Sex
And more...
Why Explainability: Laws against Discrimination
19

Fairness Privacy
Transparency Explainability
20

GDPR Concerns Around Lack of Explainability in AI
“
Companies should commit to ensuring
systems that could fall under GDPR, including
AI, will be compliant. The threat of sizeable
fines of €20 million or 4% of global
turnover provides a sharp incentive.
Article 22 of GDPR empowers individuals with
the right to demand an explanation of how
an AI system made a decision that affects
them.
”
- European Commision
VP, European Commision

Fairness Privacy
22

Fairness Privacy
23

Why Explainability: Growing Global AI Regulation
● GDPR: Article 22 empowers individuals with the right to demand an explanation of how an
automated system made a decision that affects them.
● Algorithmic Accountability Act 2019: Requires companies to provide an assessment of the risks posed by
the automated decision system to the privacy or security and the risks that contribute to inaccurate,
unfair, biased, or discriminatory decisions impacting consumers
● California Consumer Privacy Act: Requires companies to rethink their approach to capturing,
storing, and sharing personal data to align with the new requirements by January 1, 2020.
● Washington Bill 1655: Establishes guidelines for the use of automated decision systems to
protect consumers, improve transparency, and create more market predictability.
● Massachusetts Bill H.2701: Establishes a commission on automated decision-making,
transparency, fairness, and individual rights.
● Illinois House Bill 3415: States predictive data analytics determining creditworthiness or hiring
decisions may not include information that correlates with the applicant race or zip code.

25
SR 11-7 and OCC regulations for Financial Institutions

Model Diagnostics
Root Cause Analytics
Performance monitoring
Fairness monitoring
Model Comparison
Cohort Analysis
Explainable Decisions
API Support
Model Launch Signoff
Model Release Mgmt
Model Evaluation
Compliance Testing
Model Debugging
Model Visualization
Explainable
AI
Train
QA
Predict
Deploy
A/B Test
Monitor
Debug
Feedback Loop
“Explainability by Design” for AI products

AI @ Scale - Challenges for Explainable AI
27

LinkedIn operates the largest professional
network on the Internet
645M+ members
30M+
companies
are
represented
on LinkedIn
90K+
schools listed
(high school &
college)
35K+
skills listed
20M+
open jobs
on
LinkedIn
Jobs
280B
Feed updates

29© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |
The AWS ML Stack
Broadest and most complete set of Machine Learning capabilities
VISION SPEECH TEXT SEARCH NEW CHATBOTS PERSONALIZATION FORECASTING FRAUD NEW DEVELOPMENT NEW CONTACT CENTERS
NEW
Amazon SageMaker Ground
Truth
Augmented
AI
SageMaker
Neo
Built-in
algorithms
SageMaker
Notebooks NEW
SageMaker
Experiments NEW
Model
tuning
SageMaker
Debugger NEW
SageMaker
Autopilot NEW
Model
hosting
SageMaker
Model Monitor NEW
Deep Learning
AMIs & Containers
GPUs &
CPUs
Elastic
Inference
Inferentia FPGA
Amazon
Rekognition
Amazon
Polly
Amazon
Transcribe
+Medical
Amazon
Comprehend
+Medical
Amazon
Translate
Amazon
Lex
Amazon
Personalize
Amazon
Forecast
Amazon
Fraud Detector
Amazon
CodeGuru
AI SERVICES
ML SERVICES
ML FRAMEWORKS & INFRASTRUCTURE
Amazon
Textract
Amazon
Kendra
Contact Lens
For Amazon
Connect
SageMaker Studio IDE NEW
NEW
NEW
NEW
NEW

Explanation - In a Nutshell
30

What is Explainable AI?
Data
Black-Box
AI
AI
product
Confusion with Today’s AI Black
Box
● Why did you do that?
● Why did you not do that?
● When do you succeed or fail?
● How do I correct an error?
Black Box AI
Decision,
Recommendation
Clear & Transparent Predictions
● I understand why
● I understand why not
● I know why you succeed or fail
● I understand, so I trust you
Explainable AI
Data
Explainable
AI
Explainable
AI Product
Decision
Explanation
Feedback

- Humans may have follow-up questions
- Explanations cannot answer all users’ concerns
Weld, D., and Gagan Bansal. "The challenge of
crafting intelligible intelligence." Communications of
ACM (2018).
Example of an End-to-End XAI System

Neural Net
CNNGAN
RNN
Ensemble
Method
Random
Forest
XGB
Statistical
Model
AOG
SVM
Graphical Model
Bayesian
Belief Net
SLR
CRF HBN
MLN
Markov
Model
Decision
Tree
Linear
Model
Non-Linear
functions
Polynomial
functions
Quasi-Linear
functions
Accuracy
Explainability
InterpretabilityLearning
• Challenges:
• Supervised
• Unsupervised learning
• Approach:
• Representation
Learning
• Stochastic selection
• Output:
• Correlation
• No causation
How to Explain? Accuracy vs. Explainability

Oxford Dictionary of
English
XAI Definitions - Explanation vs. Interpretation

TextTabular
Images
On the Role of Data in XAI

KDD 2019 Tutorial on Explainable AI in Industry - https://sites.google.com/view/kdd19-explainable-ai-tutorial
Evaluation (1) - Perturbation-based Approaches

Evaluation criteria for Explanations [Miller, 2017]
○ Truth & probability
○ Usefulness, relevance
○ Coherence with prior belief
○ Generalization
Cognitive chunks = basic explanation units (for different explanation needs)
○ Which basic units for explanations?
○ How many?
○ How to compose them?
○ Uncertainty & end users?
[Doshi-Velez and Kim 2017, Poursabzi-Sangdeh
18]
Evaluation (2) - Human (Role)-based Evaluation is
Essential… but too often based on size!

Comprehensibilit
y
How much effort
for correct human
interpretation?
Succinctness
How concise and
compact is the
explanation?
Actionability
What can one
action, do with
the explanation?
Reusability
Could the
explanation be
personalized?
Accuracy
How accurate and
precise is the
explanation?
Completeness
Is the explanation
complete, partial,
restricted?
Source: Accenture Point of View. Understanding Machines: Explainable AI. Freddy Lecue, Dadong Wan
Evaluation (3) - XAI: One Objective, Many Metrics

Explainable AI
(from a Machine Learning Perspective)
39

Achieving Explainable AI
Approach 1: Post-hoc explain a given AI model
● Individual prediction explanations in terms of input features, influential examples,
concepts, local decision rules
● Global prediction explanations in terms of entire model in terms of partial
dependence plots, global feature importance, global decision rules
Approach 2: Build an interpretable model
● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive
Models (GAMs)
40

Slide credit: https://twitter.com/chandan_singh96/status/1138811752769101825
Integrated
Gradients

Models (GAMs)
42

Individual Prediction Explanations
43

Top label: “clog”
this image as “clog”?
44

Top label: “fireboat”
this image as “fireboat”?
45

Credit Line Increase
Fair lending laws [ECOA, FCRA] require credit decisions to be explainable
Bank Credit Lending Model
Why? Why not?
How?
? Request Denied
Query AI System
Credit Lending Score = 0.3
Credit Lending in a black-box ML world

Attribute a model’s prediction on an input to features of the input
Examples:
● Attribute an object recognition network’s prediction to its pixels
● Attribute a text sentiment network’s prediction to individual words
● Attribute a lending model’s prediction to its features
A reductive formulation of “why this prediction” but surprisingly useful
The Attribution Problem

Application of Attributions
● Debugging model predictions
E.g., Attribution an image misclassification to the pixels responsible for it
● Generating an explanation for the end-user
E.g., Expose attributions for a lending prediction to the end-user
● Analyzing model robustness
E.g., Craft adversarial examples using weaknesses surfaced by attributions
● Extract rules from the model
E.g., Combine attribution to craft rules (pharmacophores) capturing prediction
logic of a drug screening network
48

Next few slides
We will cover the following attribution methods**
● Ablations
● Gradient based methods (specific to differentiable models)
● Score Backpropagation based methods (specific to NNs)
We will also discuss game theory (Shapley value) in attributions
**Not a complete list!
See Ancona et al. [ICML 2019], Guidotti et al. [arxiv 2018] for a comprehensive survey 49

Ablations
Drop each feature and attribute the change in prediction to that feature
Pros:
● Simple and intuitive to interpret
Cons:
● Unrealistic inputs
● Improper accounting of interactive features
● Can be computationally expensive
50

Feature*Gradient
Attribution to a feature is feature value times gradient, i.e., xi* 𝜕y/𝜕
xi
● Gradient captures sensitivity of output w.r.t. feature
● Equivalent to Feature*Coefficient for linear models
○ First-order Taylor approximation of non-linear models
● Popularized by SaliencyMaps [NIPS 2013], Baehrens et al. [JMLR 2010]
51
Gradients in the
vicinity of the input
seem like noise?

Local linear approximations can be too local
52
score
“fireboat-ness” of image
Interesting gradients
uninteresting gradients
(saturation)
1.0
0.0

Score Back-Propagation based Methods
Re-distribute the prediction score through the neurons in the network
● LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014]
Easy case: Output of a neuron is a linear function
of previous neurons (i.e., ni = ⅀ wij * nj)
e.g., the logit neuron
● Re-distribute the contribution in proportion to
the coefficients wij
53
Image credit heatmapping.org

Tricky case: Output of a neuron is a non-linear
function, e.g., ReLU, Sigmoid, etc.
● Guided BackProp: Only consider ReLUs that
are on (linear regime), and which contribute
positively
● LRP: Use first-order Taylor decomposition to
linearize activation function
● DeepLift: Distribute activation difference
relative a reference point in proportion to
edge weights 54

Pros:
● Conceptually simple
● Methods have been empirically validated to
yield sensible result
Cons:
● Hard to implement, requires instrumenting
the model
● Often breaks implementation invariance
Think: F(x, y, z) = x * y *z and
G(x, y, z) = x * (y * z)

Baselines and additivity
● When we decompose the score via backpropagation, we imply a normative
alternative called a baseline
○ “Why Pr(fireboat) = 0.91 [instead of 0.00]”
● Common choice is an informationless input for the model
○ E.g., Black image for image models
○ E.g., Empty text or zero embedding vector for text models
● Additive attributions explain F(input) - F(baseline) in terms of input features

score
intensity
Interesting gradients
uninteresting gradients
(saturation)
1.0
0.0
Baseline … scaled inputs ...
… gradients of scaled inputs ….
Input
Another approach: gradients at many points

IG(input, base) ::= (input - base) * ∫0 -1▽F(𝛂*input + (1-𝛂)*base) d𝛂
Original image Integrated Gradients
Integrated Gradients [ICML 2017]
Integrate the gradients along a straight-line path from baseline to input

Integrated Gradients in action
59

Original image “Clog”
Why is this image labeled as “clog”?

Original image Integrated Gradients
(for label “clog”)
“Clog”
Why is this image labeled as “clog”?

Detecting an architecture bug
● Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site
● Finding: Some atoms had identical attributions despite different connectivity

● Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site
● Finding: Some atoms had identical attributions despite different connectivity
Detecting an architecture bug
● Bug: The architecture had a bug due to which the convolved bond features
did not affect the prediction!

● Deep network predicts various diseases from chest x-rays
Original image
Integrated gradients
(for top label)
Detecting a data issue

● Deep network predicts various diseases from chest x-rays
● Finding: Attributions fell on radiologist’s markings (rather than the pathology)
Original image
Integrated gradients
(for top label)
Detecting a data issue

Cooperative game theory in attributions
66

Classic result in game theory on distributing gain in a coalition game
● Coalition Games
○ Players collaborating to generate some gain (think: revenue)
○ Set function v(S) determining the gain for any subset S of players
Shapley Value [Annals of Mathematical studies,1953]

Classic result in game theory on distributing gain in a coalition game
● Coalition Games
○ Players collaborating to generate some gain (think: revenue)
○ Set function v(S) determining the gain for any subset S of players
● Shapley Values are a fair way to attribute the total gain to the players based on
their contributions
○ Concept: Marginal contribution of a player to a subset of other players (v(S U {i}) - v(S))
○ Shapley value for a player is a specific weighted aggregation of its marginal over all
possible subsets of other players
Shapley Value for player i = ⅀S⊆N w(S) * (v(S U {i}) - v(S))
(where w(S) = N! / |S|! (N - |S| -1)!)
Shapley Value [Annals of Mathematical studies, 1953]

Shapley values are unique under four simple axioms
● Dummy: If a player never contributes to the game then it must receive zero attribution
● Efficiency: Attributions must add to the total gain
● Symmetry: Symmetric players must receive equal attribution
● Linearity: Attribution for the (weighted) sum of two games must be the same as the
(weighted) sum of the attributions for each of the games
Shapley Value Justification

SHAP [NeurIPS 2018], QII [S&P 2016], Strumbelj & Konenko [JMLR 2009]
● Define a coalition game for each model input X
○ Players are the features in the input
○ Gain is the model prediction (output), i.e., gain = F(X)
● Feature attributions are the Shapley values of this game
Shapley Values for Explaining ML models

SHAP [NeurIPS 2018], QII [S&P 2016], Strumbelj & Konenko [JMLR 2009]
● Define a coalition game for each model input X
○ Players are the features in the input
○ Gain is the model prediction (output), i.e., gain = F(X)
● Feature attributions are the Shapley values of this game
Challenge: Shapley values require the gain to be defined for all subsets of players
● What is the prediction when some players (features) are absent?
i.e., what is F(x_1, <absent>, x_3, …, <absent>)?
Shapley Values for Explaining ML models

Key Idea: Take the expected prediction when the (absent) feature is sampled
from a certain distribution.
Different approaches choose different distributions
● [SHAP, NIPS 2018] Use conditional distribution w.r.t. the present features
● [QII, S&P 2016] Use marginal distribution
● [Strumbelj et al., JMLR 2009] Use uniform distribution
Modeling Feature Absence
Preprint: The Explanation Game: Explaining Machine Learning Models with Cooperative Game
Theory

Exact Shapley value computation is exponential in the number of features
● Shapley values can be expressed as an expectation of marginals
𝜙(i) = ES ~ D [marginal(S, i)]
● Sampling-based methods can be used to approximate the expectation
● See: “Computational Aspects of Cooperative Game Theory”, Chalkiadakis et al. 2011
● The method is still computationally infeasible for models with hundreds of
features, e.g., image models
Computing Shapley Values

● Values of Non-Atomic Games (1974): Aumann and Shapley extend their
method → players can contribute fractionally
● Aumann-Shapley values calculated by integrating along a straight-line path…
same as Integrated Gradients!
● IG through a game theory lens: continuous game, feature absence is modeled
by replacement with a baseline value
● Axiomatically justified as a result:
○ Integrated Gradients is the unique path-integral method satisfying: Sensitivity, Insensitivity,
Linearity preservation, Implementation invariance, Completeness, and Symmetry
Non-atomic Games: Aumann-Shapley Values and IG

Baselines (or Norms) are essential to explanations [Kahneman-Miller 86]
● E.g., A man suffers from indigestion. Doctor blames it to a stomach ulcer. Wife blames
it on eating turnips. Both are correct relative to their baselines.
● The baseline may also be an important analysis knob.
Attributions are contrastive, whether we think about it or not.
Lesson learned: baselines are important

Some limitations and caveats for attributions

Some things that are missing:
● Feature interactions (ignored or averaged out)
● What training examples influenced the prediction (training agnostic)
● Global properties of the model (prediction-specific)
An instance where attributions are useless:
● A model that predicts TRUE when there are even number of black pixels and
FALSE otherwise
Attributions don’t explain everything

Attributions are for human consumption
Naive scaling of attributions
from 0 to 255
Attributions have a large
range and long tail
across pixels
After clipping attributions
at 99% to reduce range
● Humans interpret attributions and generate insights
○ Doctor maps attributions for x-rays to pathologies
● Visualization matters as much as the attribution technique

Other individual prediction explanation methods

Local Interpretable Model-agnostic Explanations
(Ribeiro et al. KDD 2016)
80
Figure credit: Anchors: High-Precision Model-Agnostic
Explanations. Ribeiro et al. AAAI 2018
Figure credit: Ribeiro et al. KDD 2016

Anchors
81
Figure credit: Anchors: High-Precision Model-Agnostic Explanations. Ribeiro et al. AAAI 2018

Influence functions
● Trace a model’s prediction through the learning algorithm and
back to its training data
● Training points “responsible” for a given prediction
82
Figure credit: Understanding Black-box Predictions via Influence Functions. Koh and Liang. ICML 2017

Example based Explanations
83
● Prototypes: Representative of all the training data.
● Criticisms: Data instance that is not well represented by the set of prototypes.
Figure credit: Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Kim, Khanna and Koyejo. NIPS 2016
Learned prototypes and criticisms from Imagenet dataset (two types of dog breeds)

Global Explanations Methods
● Partial Dependence Plot:
Shows the marginal effect one or two
features have on the predicted
outcome of a machine learning model
85

Global Explanations Methods
● Permutations: The importance of a feature is the increase in the prediction error of the model
after we permuted the feature’s values, which breaks the relationship between the feature and the
true outcome.
86

Models (GAMs)
87

Decision Trees
88
Is the person fit?
Age < 30 ?
Eats a lot of pizzas?
Exercises in the morning?
Unfit UnfitFit Fit
Yes No
Yes
Yes No
No

Decision Set
89
Figure credit: Interpretable Decision Sets: A Joint Framework for Description and Prediction, Lakkaraju, Bach, Leskovec

Decision List
91
Figure credit: Interpretable Decision Sets: A Joint Framework for Description and Prediction, Lakkaraju, Bach, Leskovec

Falling Rule List
A falling rule list is an ordered list of if-then rules (falling rule lists are a type of
decision list), such that the estimated probability of success decreases
monotonically down the list. Thus, a falling rule list directly contains the decision-
making process, whereby the most at-risk observations are classified first, then
the second set, and so on.
92

Box Drawings for Rare Classes
93
Figure credit: Box Drawings for Learning with Imbalanced. Data Siong Thye Goh and Cynthia Rudin

Supersparse Linear Integer Models for Optimized
Medical Scoring Systems
Figure credit: Supersparse Linear Integer Models for Optimized Medical Scoring Systems. Berk Ustun and Cynthia Rudin
94

K- Nearest Neighbors
95
Explanation in terms of nearest training
data points responsible for the decision

GLMs and GAMs
96
Intelligible Models for Classification and Regression. Lou, Caruana and Gehrke KDD 2012
Accurate Intelligible Models with Pairwise Interactions. Lou, Caruana, Gehrke and Hooker. KDD 2013

XAI Case Studies in Industry:
Applications, Lessons Learned, and Research Challenges
97

Case Study:
Talent Platform
“Diversity Insights and Fairness-Aware Ranking”
Sahin Cem Geyik, Krishnaram Kenthapadi
98

Guiding Principle:
“Diversity by Design”

Insights to Identify
Diverse Talent
Pools
Representative
Talent Search
Results
Diversity
Learning
Curriculum
“Diversity by Design” in LinkedIn’s Talent Solutions

Inclusive Job Descriptions / Recruiter Outreach

Representative Ranking for Talent Search
S. C. Geyik, S. Ambler,
K. Kenthapadi, Fairness-Aware
Ranking in Search &
Recommendation Systems with
Application to LinkedIn Talent
Search, KDD’19.
[Microsoft’s AI/ML
conference
(MLADS’18). Distinguished
Contribution Award]
Building Representative
Talent Search at LinkedIn
(LinkedIn engineering blog)

Intuition for Measuring and Achieving Representativeness
● Ideal: Top ranked results should follow a desired distribution on
gender/age/…
○ E.g., same distribution as the underlying talent pool
● Inspired by “Equal Opportunity” definition [Hardt et al, NIPS’16]
● Defined measures (skew, divergence) based on this intuition

Desired Proportions within the Attribute of Interest
Compute the proportions of the values of the attribute (e.g., gender, gender-
age combination) amongst the set of qualified candidates
● “Qualified candidates” = Set of candidates that match the search query
criteria
● Retrieved by LinkedIn’s Galene search engine
Desired proportions could also be obtained based on legal mandate / voluntary
commitment

Fairness-aware Reranking Algorithm (Simplified)
Partition the set of potential candidates into different buckets for each attribute
value
Rank the candidates in each bucket according to the scores assigned by the
machine-learned model
Merge the ranked lists, balancing the representation requirements and the
selection of highest scored candidates
Representation requirement: Desired distribution on gender/age/…
Algorithmic variants based on how we achieve this balance

Validating Our Approach
Gender Representativeness
● Over 95% of all searches are representative compared to the qualified
population of the search
Business Metrics
● A/B test over LinkedIn Recruiter users for two weeks
● No significant change in business metrics (e.g., # InMails sent or
accepted)
Ramped to 100% of LinkedIn Recruiter users worldwide

Lessons
learned
● Post-processing approach desirable
○ Model agnostic
■ Scalable across different model choices
for our application
○ Acts as a “fail-safe”
■ Robust to application-specific business
logic
○ Easier to incorporate as part of existing
systems
■ Build a stand-alone service or component
for post-processing
■ No significant modifications to the existing
components
○ Complementary to efforts to reduce bias from
training data & during model training
● Collaboration/consensus across key stakeholders

Acknowledgements
LinkedIn Talent Solutions Diversity team, Hire & Careers AI team, Anti-abuse AI team, Data Science
Applied Research team
Special thanks to Deepak Agarwal, Parvez Ahammad, Stuart Ambler, Kinjal Basu, Jenelle Bray, Erik
Buchanan, Bee-Chung Chen, Patrick Cheung, Gil Cottle, Cyrus DiCiccio, Patrick Driscoll, Carlos Faham,
Nadia Fawaz, Priyanka Gariba, Meg Garlinghouse, Gurwinder Gulati, Rob Hallman, Sara Harrington,
Joshua Hartman, Daniel Hewlett, Nicolas Kim, Rachel Kumar, Nicole Li, Heloise Logan, Stephen Lynch,
Divyakumar Menghani, Varun Mithal, Arashpreet Singh Mor, Tanvi Motwani, Preetam Nandy, Lei Ni,
Nitin Panjwani, Igor Perisic, Hema Raghavan, Romer Rosales, Guillaume Saint-Jacques, Badrul Sarwar,
Amir Sepehri, Arun Swami, Ram Swaminathan, Grace Tang, Ketan Thakkar, Sriram Vasudevan,
Janardhanan Vembunarayanan, James Verbus, Xin Wang, Hinkmond Wong, Ya Xu, Lin Yang, Yang Yang,
Chenhui Zhai, Liang Zhang, Yani Zhang

Engineering for Fairness in AI Lifecycle
Problem
Formation
Dataset
Construction
Algorithm
Selection
Training
Process
Testing
Process
Deployment
Feedback
Is an algorithm an
ethical solution to our
problem?
Does our data include enough
minority samples?
Are there missing/biased
features?
Do we need to apply debiasing
algorithms to preprocess our
data?
Do we need to include fairness
constraints in the function?
Have we evaluated the model
using relevant fairness
metrics?
Are we deploying our model
on a population that we did
not train/test on?
Are there unequal effects
across users?
Does the model encourage
feedback loops that can
produce increasingly unfair
outcomes?
Credit: K. Browne & J. Draper

Engineering for Fairness in AI Lifecycle
S.Vasudevan, K. Kenthapadi,
FairScale: A Scalable Framework for Measuring Fairness in AI Applications, 2019

FairScale System Architecture [Vasudevan & Kenthapadi, 2019]
• Flexibility of Use
(Platform agnostic)
• Ad-hoc exploratory
analyses
• Deployment in offline
workflows
• Integration with ML
Frameworks
• Scalability
• Diverse fairness
metrics
• Conventional fairness
metrics
• Benefit metrics
• Statistical tests

Fairness-aware experimentation
[Saint-Jacques and Sepehri, KDD’19 Social Impact Workshop]
Imagine LinkedIn has 10 members.
Each of them has 1 session a day.
A new product increases sessions by +1 session per member on average.
Both of these are +1 session / member on average!
One is much more unequal than the other. We want to catch that.

Case Study:
Talent Search
Varun Mithal, Girish Kathalagiri, Sahin Cem Geyik
116

LinkedIn Recruiter
● Recruiter Searches for Candidates
○ Standardized and free-text search criteria
● Retrieval and Ranking
○ Filter candidates using the criteria
○ Rank candidates in multiple levels using ML
models
117

Modeling Approaches
● Pairwise XGBoost
● GLMix
● DNNs via TensorFlow
● Optimization Criteria: inMail Accepts
○ Positive: inMail sent by recruiter, and positively responded by candidate
○ Mutual interest between the recruiter and the candidate
118

Feature Importance in XGBoost
119

How We Utilize Feature Importances for GBDT
● Understanding feature digressions
○ Which a feature that was impactful no longer is?
○ Should we debug feature generation?
● Introducing new features in bulk and identifying effective ones
○ An activity feature for last 3 hours, 6 hours, 12 hours, 24 hours introduced (costly to compute)
○ Should we keep all such features?
● Separating the factors for that caused an improvement
○ Did an improvement come from a new feature, or a new labeling strategy, data source?
○ Did the ordering between features change?
● Shortcoming: A global view, not case by case
120

GLMix Models
● Generalized Linear Mixed Models
○ Global: Linear Model
○ Per-contract: Linear Model
○ Per-recruiter: Linear Model
● Lots of parameters overall
○ For a specific recruiter or contract the weights can be summed up
● Inherently explainable
○ Contribution of a feature is “weight x feature value”
○ Can be examined in a case-by-case manner as well
121

TensorFlow Models in Recruiter and Explaining Them
● We utilize the Integrated Gradients [ICML 2017] method
● How do we determine the baseline example?
○ Every query creates its own feature values for the same candidate
○ Query match features, time-based features
○ Recruiter affinity, and candidate affinity features
○ A candidate would be scored differently by each query
○ Cannot recommend a “Software Engineer” to a search for a “Forensic Chemist”
○ There is no globally neutral example for comparison!
122

Query-Specific Baseline Selection
● For each query:
○ Score examples by the TF model
○ Rank examples
○ Choose one example as the baseline
○ Compare others to the baseline example
● How to choose the baseline example
○ Last candidate
○ Kth percentile in ranking
○ A random candidate
○ Request by user (answering a question like: “Why was I presented candidate x above
candidate y?”)
123

Example - Detailed
125
Feature Description Difference (1 vs 2) Contribution
Feature………. Description………. -2.0476928 -2.144455602
Feature………. Description………. -2.3223877 1.903594618
Feature………. Description………. 0.11666667 0.2114946752
Feature………. Description………. -2.1442587 0.2060414469
Feature………. Description………. -14 0.1215354111
Feature………. Description………. 1 0.1000282466
Feature………. Description………. -92 -0.085286277
Feature………. Description………. 0.9333333 0.0568533262

Pros & Cons
● Explains potentially very complex models
● Case-by-case analysis
○ Why do you think candidate x is a better match for my position?
○ Why do you think I am a better fit for this job?
○ Why am I being shown this ad?
○ Great for debugging real-time problems in production
● Global view is missing
○ Aggregate Contributions can be computed
○ Could be costly to compute
126

Lessons Learned and Next Steps
● Global explanations vs. Case-by-case Explanations
○ Global gives an overview, better for making modeling decisions
○ Case-by-case could be more useful for the non-technical user, better for debugging
● Integrated gradients worked well for us
○ Complex models make it harder for developers to map improvement to effort
○ Use-case gave intuitive results, on top of completely describing score differences
● Next steps
○ Global explanations for Deep Models
127

Case Study:
Model Interpretation for Predictive Models in B2B
Sales Predictions
Jilei Yang, Wei Di, Songtao Guo
128

Problem Setting
● Predictive models in B2B sales prediction
○ E.g.: random forest, gradient boosting, deep neural network, …
○ High accuracy, low interpretability
● Global feature importance → Individual feature reasoning
129

Revisiting LIME
● Given a target sample 𝑥 𝑘, approximate its prediction 𝑝𝑟𝑒𝑑(𝑥 𝑘) by building a
sample-specific linear model:
𝑝𝑟𝑒𝑑(𝑋) ≈ 𝛽 𝑘1 𝑋1 + 𝛽 𝑘2 𝑋2 + …, 𝑋 ∈ 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟(𝑥 𝑘)
● E.g., for company CompanyX:
0.76 ≈ 1.82 ∗ 0.17 + 1.61 ∗ 0.11+…
131

xLIME
Piecewise Linear
Regression
Localized Stratified
Sampling
132

Piecewise Linear Regression
Motivation: Separate top positive feature influencers and top negative feature influencers
133

Impact of Piecewise Approach
● Target sample 𝑥 𝑘=(𝑥 𝑘1, 𝑥 𝑘2, ⋯)
● Top feature contributor
○ LIME: large magnitude of 𝛽 𝑘𝑗 ⋅ 𝑥 𝑘𝑗
○ xLIME: large magnitude of 𝛽 𝑘𝑗
− ⋅ 𝑥 𝑘𝑗
● Top positive feature influencer
○ LIME: large magnitude of 𝛽 𝑘𝑗
○ xLIME: large magnitude of negative 𝛽 𝑘𝑗
− or positive 𝛽 𝑘𝑗
+
● Top negative feature influencer
○ LIME: large magnitude of 𝛽 𝑘𝑗
○ xLIME: large magnitude of positive 𝛽 𝑘𝑗
− or negative 𝛽 𝑘𝑗
+
134

Localized Stratified Sampling: Idea
Method: Sampling based on empirical distribution around target value at each feature level
135

Localized Stratified Sampling: Method
● Sampling based on empirical distribution around target value for each feature
● For target sample 𝑥 𝑘 = (𝑥 𝑘1 , 𝑥 𝑘2 , ⋯), sampling values of feature 𝑗 according to
𝑝𝑗 (𝑋𝑗) ⋅ 𝑁(𝑥 𝑘𝑗 , (𝛼 ⋅ 𝑠𝑗 )2)
○ 𝑝𝑗 (𝑋𝑗) : empirical distribution.
○ 𝑥 𝑘𝑗 : feature value in target sample.
○ 𝑠𝑗 : standard deviation.
○ 𝛼 : Interpretable range: tradeoff between interpretable coverage and local accuracy.
● In LIME, sampling according to 𝑁(𝑥𝑗 , 𝑠𝑗
2).
136
_

LTS LCP (LinkedIn Career Page) Upsell
● A subset of churn data
○ Total Companies: ~ 19K
○ Company features: 117
● Problem: Estimate whether there will be upsell given a set of features about
the company’s utility from the product
138

Key Takeaways
● Looking at the explanation as contributor vs. influencer features is useful
○ Contributor: Which features end-up in the current outcome case-by-case
○ Influencer: What needs to be done to improve likelihood, case-by-case
● xLIME aims to improve on LIME via:
○ Piecewise linear regression: More accurately describes local point, helps with finding correct
influencers
○ Localized stratified sampling: More realistic set of local points
● Better captures the important features
142

Case Study:
Relevance Debugging and Explaining @
Daniel Qiu, Yucheng Qian
143

Debugging Relevance Models
144

Teams
● Search
● Feed
● Comments
● People you may know
● Jobs you may be interested in
● Notification
158

Case Study:
Integrated Gradients for Explaining Diabetic
Retinopathy Predictions
Ankur Taly** (Fiddler labs)
(Joint work with Mukund Sundararajan, Kedar Dhamdhere, Pramod Mudrakarta)
159
**This research was carried out at Google Research

Prediction: “proliferative” DR1
● Proliferative implies vision-threatening
Can we provide an explanation to
the doctor with supporting evidence
for “proliferative” DR?
Retinal Fundus Image
1Diabetic Retinopathy (DR) is a diabetes complication that affects the eye. Deep networks can
predict DR grade from retinal fundus images with high accuracy (AUC ~0.97) [JAMA, 2016].

Integrated Gradients for label: “proliferative”
Visualization: Overlay heatmap on green channel

Lesions
Neovascularization
● Hard to see on original image
● Known to be vision-threatening

Lesions
Neovascularization
● Hard to see on original image
● Known to be vision-threatening
Does attributions help improve doctors better diagnose diabetic retinopathy?

Assisted-read study
9 doctors grade 2000 images under three different conditions
A. Image only
B. Image + Model’s prediction scores
C. Image + Model’s prediction scores + Explanation (Integrated Gradients)

9 doctors grade 2000 images under three different conditions
A. Image only
B. Image + Model’s prediction scores
C. Image + Model’s prediction scores + Explanation (Integrated Gradients)
Findings:
● Model’s predictions (B) significantly improve accuracy vs. image only (A) (p < 0.001)
● Both forms of assistance (B and C) improved sensitivity without hurting specificity
● Explanations (C) improved accuracy of cases with DR (p < 0.001) but hurt accuracy of
cases without DR (p = 0.006)
● Both B and C increase doctor ↔ model agreement
Paper: Using a deep learning algorithm and integrated gradients explanation to assist
grading for diabetic retinopathy --- Journal of Ophthalmology [2018]
Assisted-read study

Case Study:
Building an Explainable AI Engine @
166

All your data
Any data warehouse
Custom Models
Fiddler Modeling Layer
Explainable AI for everyone
APIs, Dashboards, Reports, Trusted Insights
Fiddler’s Explainable AI Engine
Mission: Unlock Trust, Visibility and Insights by making AI Explainable in every enterprise

Credit Line Increase
Fair lending laws [ECOA, FCRA] require credit decisions to be explainable
Bank Credit Lending Model
Why? Why not? How?
? Request Denied
Query AI System
Credit Lending Score =
0.3
Example: Credit Lending in a black-box ML world

How Can This Help…
Customer Support
Why was a customer loan
rejected?
Bias & Fairness
How is my model doing
across demographics?
Lending LOB
What variables should they
validate with customers on
“borderline” decisions?
Explain individual predictions (using Shapley Values)

Customer Support
rejected?
Bias & Fairness
How is my model doing
across demographics?
Lending LOB
What variables should they
validate with customers on
“borderline” decisions?
Explain individual predictions (using Shapley Values)
Probe the
model on
counterfactuals

Customer Support
rejected?
Why was the credit card limit
low?
Why was this transaction
marked as fraud?
Integrating explanations

Global Explanations
What are the primary feature
drivers of the dataset on my
model?
Region Explanations
How does my model
perform on a certain slice?
Where does the model not
perform well? Is my model
uniformly fair across slices?
Slice & Explain

Model Monitoring: Feature Drift
Investigate Data Drift Impacting Model Performance
Time slice
Feature distribution for
time slice relative to
training distribution

Operations
Why are there outliers in
model predictions? What
caused model performance
to go awry?
Data Science
How can I improve my ML
model? Where does it not do
well?
Model Monitoring: Outliers with Explanations
Outlier
Individual
Explanations

Some lessons learned at Fiddler
● Attributions are contrastive to their baselines
● Explaining explanations is important (e.g. good UI)
● In practice, we face engineering challenges as much as
theoretical challenges
176

Recap
● Part I: Introduction and Motivation
○ Motivation and Need for Explainable AI
○ Challenges for Explainable AI @ Scale
● Part II: Explainable Machine Learning
○ Overview of Explainable AI Techniques
● Part III: Case Studies from Industry
○ Applications, Key Challenges, and Lessons Learned
● Part IV: Open Problems, Research Challenges, and Conclusion
177

Challenges & Tradeoffs
178
User PrivacyTransparency
Fairness Performance
?
● Lack of standard interface for ML models
makes pluggable explanations hard
● Explanation needs vary depending on the type
of the user who needs it and also the problem
at hand.
● The algorithm you employ for explanations
might depend on the use-case, model type,
data format, etc.
● There are trade-offs w.r.t. Explainability,
Performance, Fairness, and Privacy.

Explainability in ML: Broad Challenges
Actionable explanations
Balance between explanations & model secrecy
Robustness of explanations to failure modes (Interaction between ML
components)
Application-specific challenges
Conversational AI systems: contextual explanations
Gradation of explanations
Tools for explanations across AI lifecycle
Pre & post-deployment for ML models
Model developer vs. End user focused

Thanks! Questions?
● Feedback most welcome :-)
○ krishna@fiddler.ai, sgeyik@linkedin.com, kenthk@amazon.com,
vamithal@linkedin.com, ankur@fiddler.ai
● Tutorial website: https://sites.google.com/view/www20-explainable-ai-tutorial
● To try Fiddler, please send an email to info@fiddler.ai
180https://sites.google.com/view/www20-explainable-ai-tutorial 180

Explainable AI in Industry (WWW 2020 Tutorial)

More Related Content

What's hot (20)

Similar to Explainable AI in Industry (WWW 2020 Tutorial) (20)

More from Krishnaram Kenthapadi (13)

Recently uploaded (20)

Explainable AI in Industry (WWW 2020 Tutorial)