High Performance Distributed TensorFlow with GPUs and Kubernetes
2 likes1,762 views
The document discusses the implementation of high performance distributed TensorFlow in production using GPUs and Kubernetes, focusing on model optimization, validation, and serving. Key features include continuous insight into live production, various cloud deployment options, and tools for tuning and optimizing models post-training. It emphasizes the importance of safely deploying ML/AI models and offers strategies for managing and monitoring model performance in real-time.
![SERVING (POST-TRAIN) OPTIMIZATIONS
§ Prepare Model for Serving
§ Simplify Network, Reduce Size
§ Reduce Precision -> Fast Math
§ Some Tools
§ Graph Transform Tool (GTT)
§ tfcompile
After Training
After
Optimizing!
pipeline optimize --optimization-list=[‘quantize_weights’,‘tfcompile’]
--model-name=mnist
--model-tag=A
--model-path=./tensorflow/mnist/model
--model-inputs=[‘x’]
--model-outputs=[‘add’]
--output-path=./tensorflow/mnist/optimized_model
Linear
Regression
Model Size: 70MB –> 70K (!)](https://image.slidesharecdn.com/hpcadvisorycouncil-feb2018-chrisfregly-180225205039/85/High-Performance-Distributed-TensorFlow-with-GPUs-and-Kubernetes-13-320.jpg)
![DEPLOY MODELS SAFELY TO PROD
§ Deploy from CLI or Jupyter Notebook
§ Tear-Down and Rollback Models Quickly
§ Shadow Canary: Deploy to 20% Live Traffic
§ Split Canary: Deploy to 97-2-1% Live Traffic
pipeline predict-kube-start --model-name=mnist
--model-tag=BStart Cluster B
pipeline predict-kube-start --model-name=mnist
--model-tag=CStart Cluster C
pipeline predict-kube-start --model-name=mnist
--model-tag=AStart Cluster A
pipeline predict-kube-route --model-name=mnist
--model-split-tag-and-weight-dict='{"A":97, "B":2, "C”:1}'
--model-shadow-tag-list='[]'
Route Live Traffic](https://image.slidesharecdn.com/hpcadvisorycouncil-feb2018-chrisfregly-180225205039/85/High-Performance-Distributed-TensorFlow-with-GPUs-and-Kubernetes-19-320.jpg)
![LIVE, ADAPTIVE TRAFFIC ROUTING
§ A/B Tests
§ Inflexible and Boring
§ Multi-Armed Bandits
§ Adaptive and Exciting!
pipeline predict-kube-route --model-name=mnist
--model-split-tag-and-weight-dict='{"A":1, "B":2, "C”:97}’
--model-shadow-tag-list='[]'
Route Traffic
Dynamically](https://image.slidesharecdn.com/hpcadvisorycouncil-feb2018-chrisfregly-180225205039/85/High-Performance-Distributed-TensorFlow-with-GPUs-and-Kubernetes-25-320.jpg)
High Performance Distributed TensorFlow with GPUs and Kubernetes
- 1. HIGH PERFORMANCE DISTRIBUTED TENSORFLOW IN PRODUCTION WITH GPUS AND KUBERNETES HPC ADVISORY COUNCIL, FEB 2018 CHRIS FREGLY FOUNDER @ PIPELINE.AI
- 2. HIGH PERFORMANCE DISTRIBUTED TENSORFLOW IN PRODUCTION WITH GPUS AND KUBERNETES HPC ADVISORY COUNCIL, FEB 2018 CHRIS FREGLY FOUNDER @ PIPELINE.AI
- 3. KEY TAKE-AWAYS Optimize Your Models After Training Validate Models Online in Live Production (Safely!) Evaluate Model Performance Offline *and* Online Monitor and Tune Your Model Serving Runtime
- 4. INTRODUCTIONS: ME § Chris Fregly, Founder & Engineer @PipelineAI § Formerly Netflix, Databricks, IBM Spark Tech § Advanced Spark and TensorFlow Meetup § Please Join Our 60,000+ Global Members!! Contact Me [email protected] @cfregly Global Locations * San Francisco * Chicago * Austin * Washington DC * Dusseldorf * London
- 5. INTRODUCTIONS: YOU § Data Scientist, Data Engineer, Data Analyst, Data Curious § Want to Deploy ML/AI Models Rapidly and Safely § Need to Trace or Explain Model Predictions § Have a Decent Grasp of Computer Science Fundamentals
- 6. PIPELINE.AI IS 100% OPEN SOURCE § https://github.com/PipelineAI/pipeline/ § Please Star this GitHub Repo! § VC’s Value GitHub Stars @ $1,500 Each (?!) GitHub Repo Geo Heat Map: http://jrvis.com/red-dwarf/
- 7. PIPELINE.AI OVERVIEW 500,000Docker Downloads 60,000 Registered Users 60,000 Meetup Members 30,000 LinkedIn Followers 2,400 GitHub Stars 15 Enterprise Beta Users
- 8. RECENT PIPELINE.AI NEWS Sept 2017 Dec 2017 Jan 2018 PipelineAI Becomes Google ML/AI Expert Register to Install PipelineAI in Your Own Environment (Starting March 2018) http://pipeline.ai Try GPU Community Edition Today! http://community.pipeline.ai
- 9. WHY HEAVY FOCUS ON MODEL SERVING? Model Training Batch & Boring Offline in Research Lab Pipeline Ends at Training No Insight into Live Production Small Number of Data Scientists Optimizations Very Well-Known Real-Time & Exciting!! Online in Live Production Pipeline Extends into Production Continuous Insight into Live Production Huuuuuuge Number of Application Users Runtime Optimizations Not Yet Explored <<< Model Serving 100’s Training Jobs per Day 1,000,000’s Predictions per Sec
- 10. CLOUD-BASED MODEL SERVING OPTIONS § AWS SageMaker § Released Nov 2017 @ Re-invent § Custom Docker Images for Training/Serving (ie. PipelineAI Images) § Distributed TensorFlow Training through Estimator API § Traffic Splitting for A/B Model Testing § Google Cloud ML Engine § Mostly Command-Line Based § Driving TensorFlow Open Source API (ie. Estimator API) § Azure ML PipelineAI Supports Hybrid-Cloud Deployments
- 11. BUILD MODEL WITH THE RUNTIME § Package Model + Runtime into 1 Docker Image § Emphasizes Immutable Deployment and Infrastructure § Same Image Across All Environments § No Library or Dependency Surprises from Laptop to Production § Allows Tuning Model + Runtime Together pipeline predict-server-build --model-name=mnist --model-tag=A --model-type=tensorflow --model-runtime=tfserving --model-chip=gpu --model-path=./tensorflow/mnist/ Build Local Model Server A
- 12. TUNE MODEL + RUNTIME TOGETHER § Model Training Optimizations § Model Hyper-Parameters (ie. Learning Rate) § Reduced Precision (ie. FP16 Half Precision) § Model Serving (Post-Train) Optimizations § Quantize Model Weights + Activations From 32-bit to 8-bit § Fuse Neural Network Layers Together § Model Runtime Optimizations § Runtime Config: Request Batch Size, etc § Different Runtime: TensorFlow Serving CPU/GPU, Nvidia TensorRT
- 13. SERVING (POST-TRAIN) OPTIMIZATIONS § Prepare Model for Serving § Simplify Network, Reduce Size § Reduce Precision -> Fast Math § Some Tools § Graph Transform Tool (GTT) § tfcompile After Training After Optimizing! pipeline optimize --optimization-list=[‘quantize_weights’,‘tfcompile’] --model-name=mnist --model-tag=A --model-path=./tensorflow/mnist/model --model-inputs=[‘x’] --model-outputs=[‘add’] --output-path=./tensorflow/mnist/optimized_model Linear Regression Model Size: 70MB –> 70K (!)
- 14. NVIDIA TENSOR-RT RUNTIME § Post-Training Model Optimizations § Specific to Nvidia GPUs § GPU-Optimized Prediction Runtime § Alternative to TensorFlow Serving § PipelineAI Supports TensorRT!
- 15. TENSORFLOW LITE RUNTIME § Post-Training Model Optimizations § Currently Supports iOS and Android § On-Device Prediction Runtime § Low-Latency, Fast Startup § Selective Operator Loading § 70KB Min - 300KB Max Runtime Footprint § Supports Accelerators (GPU, TPU) § Falls Back to CPU without Accelerator § Java and C++ APIs
- 16. 3 DIFFERENT RUNTIMES, SAME MODEL pipeline predict-server-build --model-name=mnist --model-tag=C --model-type=tensorflow --model-runtime=tensorrt --model-chip=gpu --model-path=./tensorflow/mnist/ Build Local Model Server C pipeline predict-server-build --model-name=mnist --model-tag=A --model-type=tensorflow --model-runtime=tfserving --model-chip=cpu --model-path=./tensorflow/mnist/ Build Local Model Server A pipeline predict-server-build --model-name=mnist --model-tag=B --model-type=tensorflow --model-runtime=tfserving --model-chip=gpu --model-path=./tensorflow/mnist/ Build Local Model Server B Same Model, Diff Runtime
- 17. RUN A LOADTEST LOCALLY! § Perform Mini-Load Test on Local Model Server § Immediate, Local Prediction Performance Metrics § Compare to Previous Model + Runtime Variations § Gain Intuition Before Push to Prod pipeline predict-server-start --model-name=mnist --model-tag=A --memory-limit=2G pipeline predict-http-test --model-endpoint-url=http://localhost:8080 --test-request-path=test_request.json --test-request-concurrency=1000 Start Local LoadTest Start Local Model Servers
- 18. PUSH IMAGE TO DOCKER REGISTRY § Supports All Public + Private Docker Registries § DockerHub, Artifactory, Quay, AWS, Google, … § Or Self-Hosted, Private Docker Registry pipeline predict-server-push --model-name=mnist --model-tag=A --image-registry-url=<your-registry> --image-registry-repo=<your-repo> Push Images to Docker Registry
- 19. DEPLOY MODELS SAFELY TO PROD § Deploy from CLI or Jupyter Notebook § Tear-Down and Rollback Models Quickly § Shadow Canary: Deploy to 20% Live Traffic § Split Canary: Deploy to 97-2-1% Live Traffic pipeline predict-kube-start --model-name=mnist --model-tag=BStart Cluster B pipeline predict-kube-start --model-name=mnist --model-tag=CStart Cluster C pipeline predict-kube-start --model-name=mnist --model-tag=AStart Cluster A pipeline predict-kube-route --model-name=mnist --model-split-tag-and-weight-dict='{"A":97, "B":2, "C”:1}' --model-shadow-tag-list='[]' Route Live Traffic
- 20. COMPARE MODELS OFFLINE & ONLINE § Offline, Batch Metrics § Validation + Training Accuracy § CPU + GPU Utilization § Online, Live Prediction Values § Compare Relative Precision § Newly-Seen, Streaming Data § Online, Real-Time Metrics § Response Time, Throughput § Cost ($) Per Prediction
- 21. ENSEMBLE PREDICTION AUDIT TRAIL § Necessary for Model Explain-ability § Fine-Grained Request Tracing § Used for Model Ensembles
- 22. REAL-TIME PREDICTION STREAMS § Visually Compare Real-time Predictions Features and Inputs Predictions and Confidences Model B Model CModel A
- 23. PREDICTION PROFILING AND TUNING § Pinpoint Performance Bottlenecks § Fine-Grained Prediction Metrics § 3 Steps in Real-Time Prediction 1. transform_request() 2. predict() 3. transform_response()
- 24. SHIFT TRAFFIC TO MAX(REVENUE) § Shift Traffic to Winning Model with Multi-armed Bandits
- 25. LIVE, ADAPTIVE TRAFFIC ROUTING § A/B Tests § Inflexible and Boring § Multi-Armed Bandits § Adaptive and Exciting! pipeline predict-kube-route --model-name=mnist --model-split-tag-and-weight-dict='{"A":1, "B":2, "C”:97}’ --model-shadow-tag-list='[]' Route Traffic Dynamically
- 26. SHIFT TRAFFIC TO MIN(CLOUD CO$T) § Based on Cost ($) Per Prediction § Cost Changes Throughout Day § Lose AWS Spot Instances § Google Cloud Becomes Cheaper § Shift Across Clouds & On-Prem
- 27. PSEUDO-CONTINUOUS TRAINING § Identify and Fix Borderline (Unconfident) Predictions § Fix Predictions Along Class Boundaries § Facilitate ”Human in the Loop” § Retrain with Newly-Labeled Data § Game-ify the Labeling Process § Path to Crowd-Sourced Labeling
- 28. CONTINUOUS MODEL TRAINING § The Holy Grail of Machine Learning! § PipelineAI Supports Continuous Model Training! § Kafka, Kinesis § Spark Streaming, Flink § Storm, Heron
- 29. THANK YOU!! § Please Star this GitHub Repo! § All slides, code, notebooks, and Docker images here: https://github.com/PipelineAI/pipeline Contact Me [email protected] @cfregly