Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus

www.verta.ai Conﬁdential
Robust MLOps with Open-Source: ModelDB,
Docker, Jenkins, and Prometheus
!1
Presented by:
Manasi Vartak
CEO, Verta.ai
Michael Liu
Software Engineer, Verta.ai
Slack (Q&A): http://bit.ly/modeldb-mlops
#webinars

About
!2
• Open-core MLOps Platform for the full model
lifecycle
• Model versioning, deployment & ops, monitoring
• Built for data science; able to run at large scale
Manasi Vartak
CEO, Verta.ai
Michael Liu
Software Engineer,
Verta.ai
MIT CSAIL Ph.D. UCSD, Cognitive Science
Creator of ModelDB —
ﬁrst OSS model
management and
versioning system
Neural-network based
audio analysis,
everything about Python,

Verta client libs

Agenda
• Part I: Intro to MLOps (15 mins)
• Part II: Building an MLOps Pipeline (30 mins)
• Part III: Questions (10 mins)
!3
Slack (Q&A): http://bit.ly/modeldb-mlops
#webinars

Models have become Easy to Build
!4
from fastai.vision import *
from fastai.metrics import accuracy
data.normalize(imagenet_stats)
learner = create_cnn(data, models.resnet18,
metrics=[accuracy], callback_fns=ShowGraph)
learner.fit_one_cycle(8, max_lr=slice(1e-3,
1e-2))
learner.save('stage-1')

Delivery and Operations of ML-Products is Broken
!5
It used to take us 20+ weeks to bring
a new version of the model into
production.
A predictive readmission model that was
trained, optimized and deployed at a hospital
would start sharply degrading within two to
three months.

Why is ML Delivery and Ops so Hard?
!6

www.verta.ai Conﬁdential!7
Challenge 1. Model Development is empirical & ad-hoc
Model 1
Accuracy: 62%

Model 3
Accuracy: 76%
val udf1: (Int => Int) = (delayed..)
df.withColumn(“timesDelayed”, udf1)
RandomForestClassifier

Model 5
Accuracy: 68%
credit-default-clean.csv
val lrGrid = new ParamGridBuilder()
.addGrid(rf.maxDepth, Array(5, 10, 15))
.addGrid(rf.numTrees, Array(50, 100))

Model 50
Accuracy: 82%
val labelIndexer1 = new LabelIndexer()
…
val udf2: (String, Int) = …
.withColumn(“percentPaid”, udf2)
.withColumn(“creditUsed”, udf3)
val scaler = new StandardScaler()
.setInputCol(“features”) …

Challenge 2. DS/ML vs. Software are Different worlds
!11
• Flexibility

• Prototyping

• Bespoke code

• Robustness

• Scale

• Generalization

Challenge 3. Existing Tools are not ML-Aware
!12
Data Drift Resource Utilization
Optimizations Interdependencies

MLOps: DevOps for ML
!13

DevOps: Deliver Software Products Faster, More Reliably

Cross-Validation
LIME
Shapley
MLOps: Deliver ML Products Faster, More Reliably
??

This talk!
MLOps: Deliver ML Products Faster, More Reliably

Building an MLOps Pipeline with open-source:
Docker, Jenkins, Prometheus
!17
https://github.com/VertaAI/modeldb/tree/master/demos/webinar-2020-5-6

Running Example: TweetTrader
!18
Trader
DOW
NASDAQ
AI company using

social media analytics

to make $$$ … …
NLP
NLP
NLP

Let’s help TweetTrader do MLOps
!19
Package
Step 2:
Docker
Container
Release
Step 3:
Jenkins
Monitor
Step 4:
Prometheus
+ Logs
Trained
Model
Step 1:
Tweet
Model

What happens in the wild?
!20

Scenario: All our new traffic is from Germany
!21

Model 50
Accuracy: 82%
…
val udf2: (String, Int) = …
.withColumn(“percentPaid”, udf2)
.withColumn(“creditUsed”, udf3)
val scaler = new StandardScaler()
.setInputCol(“features”) …

Scenario: My colleague has an even better
model
!24

What’s missing?
!26

Cross-Validation
LIME
Shapley
I was kidding; we haven’t solved the ML part

DevOps: Deliver Software Products Faster, More Reliably

In code, every change that we make is tracked
!29

In code, every change that we make is tracked
!30
java.lang.NullPointerException: null
...
...
WebBackend
SHA: ed05334

What about models?
!31
Guten Nacht: Negative
Guten Morgen: Negative
...
...
NLPModel
s3://models/final-bert-March12
??
Code?
Data?
Conﬁg?
Env?

What’s missing is ML-specific model versioning
• Uniquely identifies a model
• Enables user to go back in time and fully recreate a model
• Code
• Data
• Config
• Environment
• Allows branching, merging, diffs etc.
• Versioning that integrates into the ML workflow (e.g., library vs. CLI)
!32

ModelDB: open-source model versioning
!33
ModelDB 2.0: https://github.com/VertaAI/modeldb
• Code
• Data
• Conﬁg
• Env
• Code
• Data
• Conﬁg
• Env

Let’s fix the pipeline
!34

Revised MLOps Pipeline
!35
Package
Step 2:
Docker
Container
Release
Step 3:
Jenkins
Monitor
Step 4:
Prometheus
+ Logs
Trained
Model
Step 1:
Tweet
Model + ModelDB

Step 1: Train a Tweet Classification model +
use ModelDB for versioning
!36

Scenario: All our new traffic is from Germany
!37

Scenario: My colleague has an even better
model
!38

lives!
!39
Trader

Revised MLOps Pipeline
!40
Package
Step 2:
Docker
Container
Release
Step 3:
Jenkins
Monitor
Step 4:
Prometheus
+ Logs
Trained
Model
Step 1:
Tweet
Model + ModelDB

Summary
• Part I: Intro to MLOps
• Part II: Building an MLOps Pipeline
• Basic pipeline: Docker, Jenkins, Prometheus
• Real-world simulations
• Pipeline with versioning: ModelDB, Docker, Jenkins, Prometheus
!41

3 Takeaways
• MLOps is DevOps for ML: it helps you ship ML products faster
• Model Versioning : MLOps :: Git : DevOps
• Robust OSS MLOps: ModelDB + Docker + Jenkins + Prometheus
!42

Thanks!
https://github.com/VertaAI/modeldb | Today’s talk: modeldb/demos/ | Slack: http://bit.ly/modeldb-mlops
!43
Pre-register for our MLOps Salon happening in June!
https://info.verta.ai/ml-ops-event

Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus

More Related Content

What's hot (20)

Similar to Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus (20)

Recently uploaded (20)

Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus