SlideShare a Scribd company logo

Raven: End-to-end Optimization of ML Prediction Queries

Databricks
Databricks

The document discusses Raven, an optimizer for machine learning prediction queries at Microsoft, focusing on its ability to improve the performance of SQL-based ML operations. It details how Raven integrates with Azure data engines, utilizing techniques like model projection pushdown and model-to-SQL translation to enhance query efficiency. Performance evaluations indicate that Raven significantly outperforms existing ML runtimes in various scenarios, achieving speed increases of up to 44 times compared to traditional approaches.

Data & Analytics
1 of 29
Downloaded 17 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Raven: End-to-end Optimization of ML Prediction Queries Konstantinos Karanasos, Kwanghyun Park Gray Systems Lab, Microsoft
App logic offline online Model Inference Featurization Model Model optimization policies orchestratio n Data Catalogs Governance Model Tracking & Provenance Access Control Logs & Telemetry policies Decisions Live Data deployment other data featurizatio n Model Training Model Development / Training offline feat. Model Enterprise-grade ML lifecycle
Data Scientist Analyst/Developer model training model scoring data exploration/ preparation data selection/ transformation model deployment Use Case: Length-of-stay in Hospital Model: “Predict length of stay of a patient in the hospital” Prediction query: “Find pregnant patients that are expected to stay in the hospital more than a week”
Featurization Model Container REST Prediction Queries: Baseline Approach policies HTTP WebServer App logic ODBC DBMS Enterprise Features • Security: data and models outside of the DB • Extra infrastructure • High TCO • Lack tooling/best-practices Performance • Data movement • Latency • Throughput on batch-scoring
Prediction Queries: In-Engine Evaluation policies HTTP WebServer App logic ODBC DBMS Enterprise Features • Security: Data and models within the DBMS • Reuse Existing infrastructure • Language/tools/best practices • Low TCO Performance ? • Up to 13x faster on Spark • Up to 330x faster on SQL Server
Raven: An Optimizer for Prediction Queries in Azure Data + data models Unified IR INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Optimized IR INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Embed high-performance ML inference runtimes within our data engines Express data and ML operations in a common graph
Constructing the IR Raven IR operators Relational algebra Linear algebra Other ML operators and data featurizers UDFs Static analysis of the prediction query Support for SQL+ML Adding support for Python INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 MQ: inference query
ML Inference in Azure Data Engines SQL Server PREDICT statement in SQL Server Embedded ONNX Runtime in the engine Available in Azure SQL Edge and SQL DW (part of Azure Synapse Analytics) Spark Introduced a new PREDICT operator Similar syntax to SQL Server Support for different types of models INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; pa Un Static Analysis MQ: inference query
INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Q: “Find pregnant patients expected to stay in the hospital more than a week” INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Raven: An Optimizer for Prediction Queries + + Runtime Code Gen
Raven optimizations in practice (name, model) AS ”, eline import Pipeline rocessing import StandardScaler import DecisionTreeClassifier n’, FeatureUnion(… scaler’,StandardScaler()), …)) reeClassifier())])”); Data Scientist) ng model (Data Analyst) rbinary(max) = ( OM scoring_models e = ”duration_of_stay“ ); nfo AS pi ts AS be ON pi.id = be.id tests AS pt ON be.id = pt.id ngth_of_stay L=@model, DATA=data AS d) ay Pred float) AS p = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model Runti Code 1. Predicate-based model pruning 2. Model projection pushdown 3. Model splitting 4. Model-to-SQL translation 5. NN translation 6. Standard DB optimizations 7. Compiler optimizations
1. Avoid unnecessary computation Information passing between model and data 2. Pick the right runtime for each operation Translation between data and ML operations 3. Hardware acceleration Translation to tensor computations (Hummingbird) INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen Raven optimizations: Key Ideas
0 1,000 2,000 3,000 4,000 5,000 DT-depth5 DT-depth8 LR-.001 GB-20est DT-depth5 DT-depth8 LR-.001 GB-20est Hospital - 2 billion rows Expedia - 500 million rows Elapsed time (seconds) End-to-end inferene query time SparkML Sklearn ONNX runtime Raven Performance Evaluation: Raven in Spark (HDI) Best of Raven: • Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL • Gradient Boost (GB): Model Projection Pushdown SELECT PREDICT(model, col1, …) FROM Hospital SELECT PREDICT(model, S.col1, …) FROM listings S, hotels R1, searches R2 WHERE S.prop_id = R1.prop_id AND S.srch_id = R2.srch_id Raven outperforms other ML runtimes (SparkML, Sklearn, ONNX runtime) by up to ~44x ~44x
0 500 60Est/Dep5 100Est/Dep4 100Est/Dep8 500Est/Dep8 Elapsed time (seconds) Gradient Boost Models (Hospital 200M rows) ONNX runtime Raven - CPU Raven - GPU 2500 3000 3500 End-to-end inference query time Performance Evaluation: Raven in Spark with GPU SELECT PREDICT(model, col1, …) FROM Hospital Raven + GPU outperforms ONNX runtime by up to ~8x for complex models ~8x
1 10 100 1,000 10,000 100,000 DT-depth5 DT- depth8 LR-.001 GB/RF-20est DT-depth5 DT- depth8 LR-.001 GB-20est hospital - 100M rows expedia - 100M rows End-to-end Time (sec) Log Scale End-to-end inference query time MADlib SQL Server (DOP1) Raven (DOP1) SQL Server (DOP16) Raven (DOP16) Performance Evaluation: Raven Plans in SQL Server Potential gains with Raven in SQL Server are significantly large! ~230x ~100x Best of Raven: • Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL • Gradient Boost (GB): Model Projection Pushdown
Performance Evaluation: Raven in SQL Server with GPU Potential gains with Raven and GPU acceleration are significantly large! ~100x Batch size: • CPU: Minimum query time obtained with optimal choice of batch size (50K/100K rows). • GPU: 600K rows. 0 200 400 600 800 1000 1200 1400 depth3- 20est depth5- 60est depth4- 100est depth8- 100est depth8- 500est End-to-end Time (secs) Min. CPU-SKL GPU-HB ~2.6x hospital – 100M rows, GB models
Demo
Conclusion: in-DBMS model inference • Raven is the first step in a long journey of incorporating ML inference as a foundational extension of relational algebra and an integral part of SQL query optimizers and runtimes • Novel Raven optimizer with cross optimizations and operator transformations Ø Up to 13x performance improvements on Spark Ø Up to 330x performance improvements on SQL Server • Integration of Raven within Spark and SQL Server
Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
Backup
Current state of affairs: In-application model inference Use case: hospital length-of-stay “Find pregnant patients that are expected to stay in the hospital more than a week” Security • Data leaves the DB • Model outside of the DB Performance • Data movement • Use of Python for data operations DBMS
Raven: In-DBMS model inference INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; Static Analysis MQ: inference query Inference query: SQL + PREDICT (SQL Server 2017 syntax) to combine SQL operations with ML inference DBMS DBMS Raven model data SQL + ML
Raven: In-DB model inference DBMS Raven Security • Data and models within the DB • Treat models as data User experience • Leverage maturity of RDBMS • Connectivity, tool integration Can in-DBMS ML inference match (or exceed?) the performance of state-of-the-art ML frameworks? Yes, by up to 230x!
Cross-optimizations in practice l (name, model) AS ay”, ipeline import Pipeline eprocessing import StandardScaler ee import DecisionTreeClassifier = ion’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) nTreeClassifier())])”); (Data Scientist) oking model (Data Analyst) varbinary(max) = ( FROM scoring_models ame = ”duration_of_stay“ ); _info AS pi ests AS be ON pi.id = be.id l_tests AS pt ON be.id = pt.id length_of_stay DEL=@model, DATA=data AS d) stay Pred float) AS p t = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model y Run Cod Cross-IR optimizations and operator transformations: Ø Predicate-based model pruning Ø Model projection pushdown Ø Model splitting Ø Model inlining Ø NN translation Ø Standard DB optimizations Ø Compiler optimizations
Raven overview INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Key ideas: 1. Novel cross-optimizations between SQL and ML operations 2. Combine high-performance ML inference engines with SQL Server
Effect of cross optimizations 1 10 102 103 104 1K 10K 100K 1M Inference Time (ms) Log Scale Dataset Size RF (scikit-learn) RF-NN (CPU) RF-NN (GPU) 24.5x 15x 5.3x
Execution modes In-process Deep integration of ONNX Runtime in SQL Server Out-of-process For queries/models not supported by our static analyzer sp_execute_external_script (Python, R, Java) Containerized For languages not supported by out- of-process execution
In-process execution Native predict: execute the model in the same process as SQL Server Rudimentary support since SQL Server 2017 (five hardcoded models) Take advantage of state-of-the-art ML inference engines Compiler optimizations, Code generation, Hardware acceleration SQL Server + ONNX Runtime Some challenges Align schemata between DB and model Transform data to/from tensors (avoid copying) Cache inference sessions Allow for different ML engines INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; St Ana MQ: inference query
Current status In-process predictions Ø Implementation in SQL Server 2019 Ø Public preview in Azure SQL DB Edge Ø Private preview in Azure SQL DW Out-of-process predictions Ø ONNX Runtime as an external language (ongoing)
Benefits of deep integration 1 10 100 1K 10k 1K 10K 100K 1M 10M 1K 10K 100K 1M 10M Total Inference Time (ms) Log Scale Dataset Size Random Forest MLP ORT Raven Raven ext.

More Related Content

What's hot (20)

PPTX
Fine tuning large LMs
SylvainGugger
 
PDF
KSQL - Stream Processing simplified!
Guido Schmutz
 
PDF
Learn to Use Databricks for the Full ML Lifecycle
Databricks
 
PPTX
Introduction to Azure Databricks
James Serra
 
PDF
RDBMS vs Hadoop vs Spark
Laxmi8
 
PDF
KFServing and Kubeflow Pipelines
Animesh Singh
 
PDF
Streaming all over the world Real life use cases with Kafka Streams
confluent
 
PDF
MLFlow: Platform for Complete Machine Learning Lifecycle
Databricks
 
PPTX
Introduction to snowflake
Sunil Gurav
 
PPTX
Databricks Platform.pptx
Alex Ivy
 
PDF
What is MLOps
Henrik Skogström
 
PPTX
Transformers AI PPT.pptx
RahulKumar854607
 
PDF
Machine Learning Operations & Azure
Erlangen Artificial Intelligence & Machine Learning Meetup
 
PDF
Getting Started with Apache Spark on Kubernetes
Databricks
 
PDF
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
HostedbyConfluent
 
PDF
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Márton Kodok
 
PDF
Vector database
Guy Korland
 
PDF
Kappa vs Lambda Architectures and Technology Comparison
Kai Wähner
 
PDF
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Kai Wähner
 
PDF
Active Retrieval Augmented Generation.pdf
Po-Chuan Chen
 
Fine tuning large LMs
SylvainGugger
 
KSQL - Stream Processing simplified!
Guido Schmutz
 
Learn to Use Databricks for the Full ML Lifecycle
Databricks
 
Introduction to Azure Databricks
James Serra
 
RDBMS vs Hadoop vs Spark
Laxmi8
 
KFServing and Kubeflow Pipelines
Animesh Singh
 
Streaming all over the world Real life use cases with Kafka Streams
confluent
 
MLFlow: Platform for Complete Machine Learning Lifecycle
Databricks
 
Introduction to snowflake
Sunil Gurav
 
Databricks Platform.pptx
Alex Ivy
 
What is MLOps
Henrik Skogström
 
Transformers AI PPT.pptx
RahulKumar854607
 
Machine Learning Operations & Azure
Erlangen Artificial Intelligence & Machine Learning Meetup
 
Getting Started with Apache Spark on Kubernetes
Databricks
 
Standing on the Shoulders of Open-Source Giants: The Serverless Realtime Lake...
HostedbyConfluent
 
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Márton Kodok
 
Vector database
Guy Korland
 
Kappa vs Lambda Architectures and Technology Comparison
Kai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Kai Wähner
 
Active Retrieval Augmented Generation.pdf
Po-Chuan Chen
 

Similar to Raven: End-to-end Optimization of ML Prediction Queries (20)

DOCX
5 g smart diabetes toward personalized diabetes diagnosis with healthcare big...
Venkat Projects
 
PDF
IRJET- Disease Prediction System
IRJET Journal
 
DOCX
A new architecture of internet of things and big data ecosystem for
Venkat Projects
 
PPT
Predicting Multiple Metrics for Queries: Better Decision Enabled by Machine L...
Soheila Dehghanzadeh
 
PDF
Comparative Analysis of Various Algorithms for Fetal Risk Prediction
IRJET Journal
 
PDF
Predicting Hospital Bills at Pre-admission
Eugene Yan Ziyou
 
PDF
IRJET- Heart Disease Prediction System
IRJET Journal
 
PDF
IRJET- Human Heart Disease Prediction using Ensemble Learning and Particle Sw...
IRJET Journal
 
PDF
IRJET- A Data Mining with Big Data Disease Prediction
IRJET Journal
 
PDF
Heart Disease Prediction using Machine Learning
IRJET Journal
 
PDF
Smart Healthcare Prediction System Using Machine Learning
IRJET Journal
 
PPTX
hdpm project.pptxPREAPARE PPT ON ARTIFICIAL INTELLIGENCE IN HEALTH CARE AND I...
evil36586
 
PDF
E-Healthcare monitoring System for diagnosis of Heart Disease using Machine L...
IRJET Journal
 
PDF
IRJET- Disease Prediction using Machine Learning
IRJET Journal
 
PDF
Care expert assistant for Medicare system using Machine learning
IRJET Journal
 
PDF
ICU Mortality Rate Estimation Using Machine Learning and Artificial Neural Ne...
IRJET Journal
 
PPTX
HDPM: An Effective Heart Disease Prediction Model for a Clinical Decision Sup...
rolex8627
 
PPTX
Smart Health Disease Prediction django machinelearning.pptx
saiproject
 
PDF
IRJET - Breast Cancer Risk and Diagnostics using Artificial Neural Network(ANN)
IRJET Journal
 
PDF
Heart Disease Prediction Using Random Forest Algorithm
IRJET Journal
 
5 g smart diabetes toward personalized diabetes diagnosis with healthcare big...
Venkat Projects
 
IRJET- Disease Prediction System
IRJET Journal
 
A new architecture of internet of things and big data ecosystem for
Venkat Projects
 
Predicting Multiple Metrics for Queries: Better Decision Enabled by Machine L...
Soheila Dehghanzadeh
 
Comparative Analysis of Various Algorithms for Fetal Risk Prediction
IRJET Journal
 
Predicting Hospital Bills at Pre-admission
Eugene Yan Ziyou
 
IRJET- Heart Disease Prediction System
IRJET Journal
 
IRJET- Human Heart Disease Prediction using Ensemble Learning and Particle Sw...
IRJET Journal
 
IRJET- A Data Mining with Big Data Disease Prediction
IRJET Journal
 
Heart Disease Prediction using Machine Learning
IRJET Journal
 
Smart Healthcare Prediction System Using Machine Learning
IRJET Journal
 
hdpm project.pptxPREAPARE PPT ON ARTIFICIAL INTELLIGENCE IN HEALTH CARE AND I...
evil36586
 
E-Healthcare monitoring System for diagnosis of Heart Disease using Machine L...
IRJET Journal
 
IRJET- Disease Prediction using Machine Learning
IRJET Journal
 
Care expert assistant for Medicare system using Machine learning
IRJET Journal
 
ICU Mortality Rate Estimation Using Machine Learning and Artificial Neural Ne...
IRJET Journal
 
HDPM: An Effective Heart Disease Prediction Model for a Clinical Decision Sup...
rolex8627
 
Smart Health Disease Prediction django machinelearning.pptx
saiproject
 
IRJET - Breast Cancer Risk and Diagnostics using Artificial Neural Network(ANN)
IRJET Journal
 
Heart Disease Prediction Using Random Forest Algorithm
IRJET Journal
 
Ad

More from Databricks (20)

PPTX
DW Migration Webinar-March 2022.pptx
Databricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
Databricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
PPTX
Data Lakehouse Symposium | Day 2
Databricks
 
PPTX
Data Lakehouse Symposium | Day 4
Databricks
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
Databricks
 
PDF
Learn to Use Databricks for Data Science
Databricks
 
PDF
Why APM Is Not the Same As ML Monitoring
Databricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
PDF
Sawtooth Windows for Feature Aggregations
Databricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
Databricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
Databricks
 
PDF
Machine Learning CI/CD for Email Attack Detection
Databricks
 
DW Migration Webinar-March 2022.pptx
Databricks
 
Data Lakehouse Symposium | Day 1 | Part 1
Databricks
 
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
Data Lakehouse Symposium | Day 2
Databricks
 
Data Lakehouse Symposium | Day 4
Databricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
Democratizing Data Quality Through a Centralized Platform
Databricks
 
Learn to Use Databricks for Data Science
Databricks
 
Why APM Is Not the Same As ML Monitoring
Databricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks
 
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
Sawtooth Windows for Feature Aggregations
Databricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks
 
Re-imagine Data Monitoring with whylogs and Spark
Databricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 
Massive Data Processing in Adobe Using Delta Lake
Databricks
 
Machine Learning CI/CD for Email Attack Detection
Databricks
 
Ad

Recently uploaded (20)

PDF
apidays Helsinki & North 2025 - Monetizing AI APIs: The New API Economy, Alla...
apidays
 
PPTX
Module-5-Measures-of-Central-Tendency-Grouped-Data-1.pptx
lacsonjhoma0407
 
PDF
How to Connect Your On-Premises Site to AWS Using Site-to-Site VPN.pdf
Tamanna
 
PDF
Web Scraping with Google Gemini 2.0 .pdf
Tamanna
 
PDF
Merits and Demerits of DBMS over File System & 3-Tier Architecture in DBMS
MD RIZWAN MOLLA
 
PDF
Avatar for apidays apidays PRO June 07, 2025 0 5 apidays Helsinki & North 2...
apidays
 
PPT
AI Future trends and opportunities_oct7v1.ppt
SHIKHAKMEHTA
 
PPTX
apidays Munich 2025 - Building an AWS Serverless Application with Terraform, ...
apidays
 
PPTX
AI Presentation Tool Pitch Deck Presentation.pptx
ShyamPanthavoor1
 
PPTX
Listify-Intelligent-Voice-to-Catalog-Agent.pptx
nareshkottees
 
PDF
Choosing the Right Database for Indexing.pdf
Tamanna
 
PDF
R Cookbook - Processing and Manipulating Geological spatial data with R.pdf
OtnielSimopiaref2
 
PPTX
apidays Singapore 2025 - From Data to Insights: Building AI-Powered Data APIs...
apidays
 
PDF
apidays Helsinki & North 2025 - How (not) to run a Graphql Stewardship Group,...
apidays
 
PPTX
ER_Model_Relationship_in_DBMS_Presentation.pptx
dharaadhvaryu1992
 
PPTX
b6057ea5-8e8c-4415-90c0-ed8e9666ffcd.pptx
Anees487379
 
PDF
The European Business Wallet: Why It Matters and How It Powers the EUDI Ecosy...
Lal Chandran
 
PDF
apidays Helsinki & North 2025 - REST in Peace? Hunting the Dominant Design fo...
apidays
 
PDF
AUDITABILITY & COMPLIANCE OF AI SYSTEMS IN HEALTHCARE
GAHI Youssef
 
PDF
JavaScript - Good or Bad? Tips for Google Tag Manager
📊 Markus Baersch
 
apidays Helsinki & North 2025 - Monetizing AI APIs: The New API Economy, Alla...
apidays
 
Module-5-Measures-of-Central-Tendency-Grouped-Data-1.pptx
lacsonjhoma0407
 
How to Connect Your On-Premises Site to AWS Using Site-to-Site VPN.pdf
Tamanna
 
Web Scraping with Google Gemini 2.0 .pdf
Tamanna
 
Merits and Demerits of DBMS over File System & 3-Tier Architecture in DBMS
MD RIZWAN MOLLA
 
Avatar for apidays apidays PRO June 07, 2025 0 5 apidays Helsinki & North 2...
apidays
 
AI Future trends and opportunities_oct7v1.ppt
SHIKHAKMEHTA
 
apidays Munich 2025 - Building an AWS Serverless Application with Terraform, ...
apidays
 
AI Presentation Tool Pitch Deck Presentation.pptx
ShyamPanthavoor1
 
Listify-Intelligent-Voice-to-Catalog-Agent.pptx
nareshkottees
 
Choosing the Right Database for Indexing.pdf
Tamanna
 
R Cookbook - Processing and Manipulating Geological spatial data with R.pdf
OtnielSimopiaref2
 
apidays Singapore 2025 - From Data to Insights: Building AI-Powered Data APIs...
apidays
 
apidays Helsinki & North 2025 - How (not) to run a Graphql Stewardship Group,...
apidays
 
ER_Model_Relationship_in_DBMS_Presentation.pptx
dharaadhvaryu1992
 
b6057ea5-8e8c-4415-90c0-ed8e9666ffcd.pptx
Anees487379
 
The European Business Wallet: Why It Matters and How It Powers the EUDI Ecosy...
Lal Chandran
 
apidays Helsinki & North 2025 - REST in Peace? Hunting the Dominant Design fo...
apidays
 
AUDITABILITY & COMPLIANCE OF AI SYSTEMS IN HEALTHCARE
GAHI Youssef
 
JavaScript - Good or Bad? Tips for Google Tag Manager
📊 Markus Baersch
 

Raven: End-to-end Optimization of ML Prediction Queries

  • 1. Raven: End-to-end Optimization of ML Prediction Queries Konstantinos Karanasos, Kwanghyun Park Gray Systems Lab, Microsoft
  • 2. App logic offline online Model Inference Featurization Model Model optimization policies orchestratio n Data Catalogs Governance Model Tracking & Provenance Access Control Logs & Telemetry policies Decisions Live Data deployment other data featurizatio n Model Training Model Development / Training offline feat. Model Enterprise-grade ML lifecycle
  • 3. Data Scientist Analyst/Developer model training model scoring data exploration/ preparation data selection/ transformation model deployment Use Case: Length-of-stay in Hospital Model: “Predict length of stay of a patient in the hospital” Prediction query: “Find pregnant patients that are expected to stay in the hospital more than a week”
  • 4. Featurization Model Container REST Prediction Queries: Baseline Approach policies HTTP WebServer App logic ODBC DBMS Enterprise Features • Security: data and models outside of the DB • Extra infrastructure • High TCO • Lack tooling/best-practices Performance • Data movement • Latency • Throughput on batch-scoring
  • 5. Prediction Queries: In-Engine Evaluation policies HTTP WebServer App logic ODBC DBMS Enterprise Features • Security: Data and models within the DBMS • Reuse Existing infrastructure • Language/tools/best practices • Low TCO Performance ? • Up to 13x faster on Spark • Up to 330x faster on SQL Server
  • 6. Raven: An Optimizer for Prediction Queries in Azure Data + data models Unified IR INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Optimized IR INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Embed high-performance ML inference runtimes within our data engines Express data and ML operations in a common graph
  • 7. Constructing the IR Raven IR operators Relational algebra Linear algebra Other ML operators and data featurizers UDFs Static analysis of the prediction query Support for SQL+ML Adding support for Python INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 MQ: inference query
  • 8. ML Inference in Azure Data Engines SQL Server PREDICT statement in SQL Server Embedded ONNX Runtime in the engine Available in Azure SQL Edge and SQL DW (part of Azure Synapse Analytics) Spark Introduced a new PREDICT operator Similar syntax to SQL Server Support for different types of models INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; pa Un Static Analysis MQ: inference query
  • 9. INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Q: “Find pregnant patients expected to stay in the hospital more than a week” INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Raven: An Optimizer for Prediction Queries + + Runtime Code Gen
  • 10. Raven optimizations in practice (name, model) AS ”, eline import Pipeline rocessing import StandardScaler import DecisionTreeClassifier n’, FeatureUnion(… scaler’,StandardScaler()), …)) reeClassifier())])”); Data Scientist) ng model (Data Analyst) rbinary(max) = ( OM scoring_models e = ”duration_of_stay“ ); nfo AS pi ts AS be ON pi.id = be.id tests AS pt ON be.id = pt.id ngth_of_stay L=@model, DATA=data AS d) ay Pred float) AS p = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model Runti Code 1. Predicate-based model pruning 2. Model projection pushdown 3. Model splitting 4. Model-to-SQL translation 5. NN translation 6. Standard DB optimizations 7. Compiler optimizations
  • 11. 1. Avoid unnecessary computation Information passing between model and data 2. Pick the right runtime for each operation Translation between data and ML operations 3. Hardware acceleration Translation to tensor computations (Hummingbird) INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen Raven optimizations: Key Ideas
  • 12. 0 1,000 2,000 3,000 4,000 5,000 DT-depth5 DT-depth8 LR-.001 GB-20est DT-depth5 DT-depth8 LR-.001 GB-20est Hospital - 2 billion rows Expedia - 500 million rows Elapsed time (seconds) End-to-end inferene query time SparkML Sklearn ONNX runtime Raven Performance Evaluation: Raven in Spark (HDI) Best of Raven: • Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL • Gradient Boost (GB): Model Projection Pushdown SELECT PREDICT(model, col1, …) FROM Hospital SELECT PREDICT(model, S.col1, …) FROM listings S, hotels R1, searches R2 WHERE S.prop_id = R1.prop_id AND S.srch_id = R2.srch_id Raven outperforms other ML runtimes (SparkML, Sklearn, ONNX runtime) by up to ~44x ~44x
  • 13. 0 500 60Est/Dep5 100Est/Dep4 100Est/Dep8 500Est/Dep8 Elapsed time (seconds) Gradient Boost Models (Hospital 200M rows) ONNX runtime Raven - CPU Raven - GPU 2500 3000 3500 End-to-end inference query time Performance Evaluation: Raven in Spark with GPU SELECT PREDICT(model, col1, …) FROM Hospital Raven + GPU outperforms ONNX runtime by up to ~8x for complex models ~8x
  • 14. 1 10 100 1,000 10,000 100,000 DT-depth5 DT- depth8 LR-.001 GB/RF-20est DT-depth5 DT- depth8 LR-.001 GB-20est hospital - 100M rows expedia - 100M rows End-to-end Time (sec) Log Scale End-to-end inference query time MADlib SQL Server (DOP1) Raven (DOP1) SQL Server (DOP16) Raven (DOP16) Performance Evaluation: Raven Plans in SQL Server Potential gains with Raven in SQL Server are significantly large! ~230x ~100x Best of Raven: • Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL • Gradient Boost (GB): Model Projection Pushdown
  • 15. Performance Evaluation: Raven in SQL Server with GPU Potential gains with Raven and GPU acceleration are significantly large! ~100x Batch size: • CPU: Minimum query time obtained with optimal choice of batch size (50K/100K rows). • GPU: 600K rows. 0 200 400 600 800 1000 1200 1400 depth3- 20est depth5- 60est depth4- 100est depth8- 100est depth8- 500est End-to-end Time (secs) Min. CPU-SKL GPU-HB ~2.6x hospital – 100M rows, GB models
  • 16. Demo
  • 17. Conclusion: in-DBMS model inference • Raven is the first step in a long journey of incorporating ML inference as a foundational extension of relational algebra and an integral part of SQL query optimizers and runtimes • Novel Raven optimizer with cross optimizations and operator transformations Ø Up to 13x performance improvements on Spark Ø Up to 330x performance improvements on SQL Server • Integration of Raven within Spark and SQL Server
  • 18. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
  • 20. Current state of affairs: In-application model inference Use case: hospital length-of-stay “Find pregnant patients that are expected to stay in the hospital more than a week” Security • Data leaves the DB • Model outside of the DB Performance • Data movement • Use of Python for data operations DBMS
  • 21. Raven: In-DBMS model inference INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; Static Analysis MQ: inference query Inference query: SQL + PREDICT (SQL Server 2017 syntax) to combine SQL operations with ML inference DBMS DBMS Raven model data SQL + ML
  • 22. Raven: In-DB model inference DBMS Raven Security • Data and models within the DB • Treat models as data User experience • Leverage maturity of RDBMS • Connectivity, tool integration Can in-DBMS ML inference match (or exceed?) the performance of state-of-the-art ML frameworks? Yes, by up to 230x!
  • 23. Cross-optimizations in practice l (name, model) AS ay”, ipeline import Pipeline eprocessing import StandardScaler ee import DecisionTreeClassifier = ion’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) nTreeClassifier())])”); (Data Scientist) oking model (Data Analyst) varbinary(max) = ( FROM scoring_models ame = ”duration_of_stay“ ); _info AS pi ests AS be ON pi.id = be.id l_tests AS pt ON be.id = pt.id length_of_stay DEL=@model, DATA=data AS d) stay Pred float) AS p t = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model y Run Cod Cross-IR optimizations and operator transformations: Ø Predicate-based model pruning Ø Model projection pushdown Ø Model splitting Ø Model inlining Ø NN translation Ø Standard DB optimizations Ø Compiler optimizations
  • 24. Raven overview INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Key ideas: 1. Novel cross-optimizations between SQL and ML operations 2. Combine high-performance ML inference engines with SQL Server
  • 25. Effect of cross optimizations 1 10 102 103 104 1K 10K 100K 1M Inference Time (ms) Log Scale Dataset Size RF (scikit-learn) RF-NN (CPU) RF-NN (GPU) 24.5x 15x 5.3x
  • 26. Execution modes In-process Deep integration of ONNX Runtime in SQL Server Out-of-process For queries/models not supported by our static analyzer sp_execute_external_script (Python, R, Java) Containerized For languages not supported by out- of-process execution
  • 27. In-process execution Native predict: execute the model in the same process as SQL Server Rudimentary support since SQL Server 2017 (five hardcoded models) Take advantage of state-of-the-art ML inference engines Compiler optimizations, Code generation, Hardware acceleration SQL Server + ONNX Runtime Some challenges Align schemata between DB and model Transform data to/from tensors (avoid copying) Cache inference sessions Allow for different ML engines INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; St Ana MQ: inference query
  • 28. Current status In-process predictions Ø Implementation in SQL Server 2019 Ø Public preview in Azure SQL DB Edge Ø Private preview in Azure SQL DW Out-of-process predictions Ø ONNX Runtime as an external language (ongoing)
  • 29. Benefits of deep integration 1 10 100 1K 10k 1K 10K 100K 1M 10M 1K 10K 100K 1M 10M Total Inference Time (ms) Log Scale Dataset Size Random Forest MLP ORT Raven Raven ext.