![Build Model
- Model is income classifier
- Build alibi explainer together with model
# train an RF model
np.random.seed(0)
clf = RandomForestClassifier(n_estimators=50)
#clf.fit(preprocessor.transform(X_train), Y_train)
pipeline = Pipeline([('preprocessor', preprocessor),
('clf', clf)])
pipeline.fit(X_train, Y_train)
print(X_train.shape)
print(pipeline.predict(X_train[0:1]))
print("Creating an explainer")
predict_fn = lambda x: pipeline.predict_proba(x)
predict_fn(X_train[0:1])
predict_fn(np.zeros([1, len(feature_names)]))
explainer = alibi.explainers.AnchorTabular(predict_fn=predict_fn,
feature_names=feature_names,
categorical_names=category_map)
explainer.fit(X_train)
explainer.predict_fn = None # Clear explainer predict_fn as its a lambda and will be reset when loaded
with open("explainer.dill", 'wb') as f:
dill.dump(explainer,f)](https://image.slidesharecdn.com/fromtrainingtoexplainabilityviagitops-191025122549/85/From-training-to-explainability-via-git-ops-10-320.jpg)
![Seldon GitOps Serving apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
name: sklearn
spec:
name: iris
predictors:
- graph:
children: []
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/sklearn/iris
name: classifier
name: default
replicas: 1
Model in storage bucket
Manifest in Git
KFServing too](https://image.slidesharecdn.com/fromtrainingtoexplainabilityviagitops-191025122549/85/From-training-to-explainability-via-git-ops-11-320.jpg)
![GitOps from Pipeline
@dsl.pipeline(
name="Serving gitops",
description="Example of pushing to git from pipeline"
)
#Example to show how serving yaml can be pushed to git
def
serve_gitops(user='ryandawsonuk',email='rd@seldon.io',git_token='xxxxxxxxxx',file='https://raw.githubusercontent.com/ryandawsonuk/seldon_gitop
s_repo_old1/master/default/SeldonDeployment-income-classifier2.json',filename='SeldonDeployment-income-classifier2.json'):
#push file to serving repo
push = dsl.ContainerOp(
name="push",
image="alpine/git:latest",
command=["sh", "-c"],
arguments=["git config --global url.'https://"+str(git_token)+":@github.com/'.insteadOf 'https://github.com/'; wget "+str(file)+"; git
config --global user.name '"+str(user)+"'; git config --global user.email "+str(email)+"; git clone https://github.com/"+str(user)+"/seldon-
gitops; ls; cp "+str(filename)+" ./seldon-gitops/default/"+str(filename)+"; cd ./seldon-gitops/; git add .; git commit -m 'add
"+str(filename)+"'; git push -u origin master;"]
)](https://image.slidesharecdn.com/fromtrainingtoexplainabilityviagitops-191025122549/85/From-training-to-explainability-via-git-ops-13-320.jpg)
![Explainer Deployment
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
name: income
spec:
name: income
predictors:
- graph:
children: []
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/sklearn/income/model
name: classifier
explainer:
type: anchor_tabular
modelUri: gs://seldon-models/sklearn/income/explainer
name: default
replicas: 1
Declarative yaml
Wizards for time to value & sandboxing](https://image.slidesharecdn.com/fromtrainingtoexplainabilityviagitops-191025122549/85/From-training-to-explainability-via-git-ops-22-320.jpg)
From training to explainability via git ops
- 1. From Training to Explainability via GitOps Kubeflow Contributor Summit October 2019
- 2. Outline - Background: What Customers want from Kubeflow - Time to value - Governance - How best to get to live predictions? - GitOps - why and how - Pipeline to serving walkthrough with - Oversight - Observability - Explainability
- 3. What Customers want from an ML Platform Empowerment/Time to value ● Self-service for data science ● DS & Ops collaboration ● Sandboxing ● Repeatable approaches Governance ● Visibility and oversight of running models ● Detailed monitoring ● Audit trails ● Access control ● Repeatable approaches ● Explainability Kubeflow ticks these boxes!
- 4. Kubeflow for Collaboration - Jupyter - Collaboration - Sandboxing (inc fairing) - Share repeatable approaches - Pipelines - Data science and Ops collaboration - Repeatable approaches - Audit Trails (governance)
- 5. Kubeflow for Governance - Metadata/Artifact Management - Track what produced when and how - Multi User Isolation - Control who can do what
- 6. Path to Live Serving Those features aimed at exploration and training Multiple paths to serving (live predictions) with kubeflow. How best to get from training to serving? How do we get to serving with empowerment and governance?
- 7. GitOps for Live Serving ● Cluster state represented declaratively ● ArgoCD/Flux/Jenkins-X ● Audit trails and reverts ● Git permissions ● Favourite with Ops Ok to push to cluster for sandboxing. GitOps great option for prod… but how best to do it?
- 8. From Experimentation To Explainability Example with GitOps
- 9. The scenario ● Classify income (as high or low) based on US Census features incl. age, gender, race, marital status ● Train a scikit-learn classifier ● Deploy from kubeflow pipeline via GitOps ● Serve requests with Seldon ● Deploy alibi explainer and explain predictions
- 10. Build Model - Model is income classifier - Build alibi explainer together with model # train an RF model np.random.seed(0) clf = RandomForestClassifier(n_estimators=50) #clf.fit(preprocessor.transform(X_train), Y_train) pipeline = Pipeline([('preprocessor', preprocessor), ('clf', clf)]) pipeline.fit(X_train, Y_train) print(X_train.shape) print(pipeline.predict(X_train[0:1])) print("Creating an explainer") predict_fn = lambda x: pipeline.predict_proba(x) predict_fn(X_train[0:1]) predict_fn(np.zeros([1, len(feature_names)])) explainer = alibi.explainers.AnchorTabular(predict_fn=predict_fn, feature_names=feature_names, categorical_names=category_map) explainer.fit(X_train) explainer.predict_fn = None # Clear explainer predict_fn as its a lambda and will be reset when loaded with open("explainer.dill", 'wb') as f: dill.dump(explainer,f)
- 11. Seldon GitOps Serving apiVersion: machinelearning.seldon.io/v1alpha2 kind: SeldonDeployment metadata: name: sklearn spec: name: iris predictors: - graph: children: [] implementation: SKLEARN_SERVER modelUri: gs://seldon-models/sklearn/iris name: classifier name: default replicas: 1 Model in storage bucket Manifest in Git KFServing too
- 13. GitOps from Pipeline @dsl.pipeline( name="Serving gitops", description="Example of pushing to git from pipeline" ) #Example to show how serving yaml can be pushed to git def serve_gitops(user='ryandawsonuk',email='[email protected]',git_token='xxxxxxxxxx',file='https://raw.githubusercontent.com/ryandawsonuk/seldon_gitop s_repo_old1/master/default/SeldonDeployment-income-classifier2.json',filename='SeldonDeployment-income-classifier2.json'): #push file to serving repo push = dsl.ContainerOp( name="push", image="alpine/git:latest", command=["sh", "-c"], arguments=["git config --global url.'https://"+str(git_token)+":@github.com/'.insteadOf 'https://github.com/'; wget "+str(file)+"; git config --global user.name '"+str(user)+"'; git config --global user.email "+str(email)+"; git clone https://github.com/"+str(user)+"/seldon- gitops; ls; cp "+str(filename)+" ./seldon-gitops/default/"+str(filename)+"; cd ./seldon-gitops/; git add .; git commit -m 'add "+str(filename)+"'; git push -u origin master;"] )
- 14. GitOps for Serving ● Great for Data Science and Ops Collaboration
- 16. Observability Now we know what’s running… So what is it doing?
- 19. Sidenote: Access Control Can’t have metrics without requests Access from curl or Seldon UI predict/load-test If you don’t have an existing auth preference we like...
- 20. Explainability So now we know what it’s doing… Why is it doing that?
- 21. Request Logging To ask why, need to first know what happened
- 22. Explainer Deployment apiVersion: machinelearning.seldon.io/v1alpha2 kind: SeldonDeployment metadata: name: income spec: name: income predictors: - graph: children: [] implementation: SKLEARN_SERVER modelUri: gs://seldon-models/sklearn/income/model name: classifier explainer: type: anchor_tabular modelUri: gs://seldon-models/sklearn/income/explainer name: default replicas: 1 Declarative yaml Wizards for time to value & sandboxing
- 23. Alibi Explainers - Includes techniques for black-box models - We’ll use anchors for tabular data - Anchors are sufficient conditions to ensure a certain prediction - As long as the anchor holds, the prediction should remain the same regardless of the values of the other features - Anchors are chosen to maximise the range for which the prediction holds
- 26. Wrap-up ● What Seldon Customers want ○ Time to value ○ Governance ● GitOps helps with both ● Pipeline to serving walkthrough with ○ Oversight ○ Observability ○ Explainability
- 27. The Future Very excited about: ● Metadata integrations ● Permissions ● KFServing and MLGraph