RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro
5 likes2,200 views
The document discusses the development of machine learning products using Flask and TensorFlow Serving, emphasizing challenges and solutions in deployment, iteration, and model management. It presents 'Racket', a minimalistic framework for model deployment, which automates versioning and provides a rich RESTful interface, enhancing integration for non-ML experts. Key takeaways highlight the importance of reducing friction between research and development and providing the right tools to benefit the entire organization.
![Takeaways
● To make great products
○ Do machine learning like the great engineer you are, not like the great machine learning expert
you aren’t.
[Google’s best practices for ML]
○ Provide DS with the right tooling, and the entire organization will benefit](https://image.slidesharecdn.com/restfulmachinelearningwithflaskandtensorflowserving-190104023422/85/RESTful-Machine-Learning-with-Flask-and-TensorFlow-Serving-Carlo-Mazzaferro-7-320.jpg)
RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro
- 1. RESTful Machine Learning with Flask and TensorFlow Serving Carlo Mazzaferro Data Scientist ByteCubed PyData DC 2018
- 2. Goals ● Explain some common problems that arise when developing machine learning products ● Describe some of the solutions that are leveraged to solve those problems, alongside with their limitations ● Open source a library
- 3. Background ● BS Biomed Eng/Bioinformatics ● DNA stuff ○ Lots of data (wrangling) ○ Data pipelines ○ Data management
- 4. Background ● BS Biomed Eng/Bioinformatics ● DNA stuff ○ Lots of data (wrangling) ○ Data pipelines ○ Data management ○ Data Science aspirant ● Currently ○ ML Engineering @ ByteCubed ○ Some frontend work
- 5. Background ● Currently ○ ML Engineering @ ByteCubed ○ Some frontend work ○ Putting models in the client’s hands ● Since always ○ Workflow optimization for ML/DS
- 6. Takeaways ● To make great products ○ Repeatability matters ○ Ease of deployment matters ○ Quick iteration is key ○ 80-20 rule Ultimately, reduce friction between R&D and the rest of the Software Practice
- 7. Takeaways ● To make great products ○ Do machine learning like the great engineer you are, not like the great machine learning expert you aren’t. [Google’s best practices for ML] ○ Provide DS with the right tooling, and the entire organization will benefit
- 8. TensorFlow Serving A high-performance serving system for machine learning models, designed for production environments Capabilities ● High-performance inference ● Model discovery ● RPC interface (gRPC and REST) ● Much more: tensorflow.org/serving
- 9. How? A high-performance serving system for machine learning models, designed for production environments Capabilities ● Keeping models loaded without needing to restore the dataflow graph and session ● gRPC interface for high-performance, low-latency inference ● Low-level control on how models are loaded ○ C++ API
- 10. ● Relies on a C++ API for low-level control ● Undocumented Python API ○ Small subset of features implemented ● gRPC API only ○ REST API (for inference only) introduced in August Limitations A high-performance serving system for machine learning models, designed for production environments
- 11. ● Pythonic API ✅ ● JSON + REST ✅ ● Test different models, architectures, configurations ✅ ● Track models over time ✅ Needs A high-performance serving system for machine learning models, designed for production environments
- 12. Current state Existing solutions ● Clipper.ai → great, but not very active (last commit in June) ● P/SaaS platforms (Algorithmia, Rekognition, etc.) ● Mlflow.org → R&D, tracking of model performance ● Custom solutions of varied quality Model deployment frameworks
- 13. Introducing racket Minimalistic framework for ML model deployment & management ● Access to most of the TensorFlow Serving functionality in a Pythonic way ● Model exploration and deployment, consolidated
- 14. racket Motivation ● Automation ○ Quick iteration ● Reduce boilerplate ○ Versioning & serving ● Exploration & visibility ○ Loading a different version ○ Ability to track & query model performance ● Flexibility ○ Multiple models can be loaded and are accessible through a simple API
- 15. racket Features ● Automated model version ● RESTful interface with rich capabilities ○ Loading a different model ○ Loading a different version ○ Ability to track & query model performance ● Automatic API documentation with Swagger ● Train, explore, and deploy with a single tool ● CLI access ● Static typing
- 16. ML For Humans Enabling integration with non-ml experts: Developer perspective ● REST access ○ Ease of integration ● Ease of deployment ○ Containers not just an afterthought ● Visibility: discover shapes of inputs needed, shapes of outputs +
- 17. ML For Humans Enabling integration with non-ml experts: Data Scientist perspective from racket import KerasLearner from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from tensorflow.keras.callbacks import TensorBoard class KerasModel(KerasLearner): VERSION = '1.1.1' MODEL_TYPE = 'regression' MODEL_NAME = 'keras-simple-regression' def build_model(self): optimizer = tf.train.RMSPropOptimizer(0.001) model = Sequential() model.add(Dense(24, input_dim=5, kernel_initializer='normal', activation='relu')) model.add(Dense(48, kernel_initializer='normal', activation='relu')) model.add(Dense(1, kernel_initializer='normal')) model.compile(loss='mean_absolute_error', optimizer=optimizer) return model def fit(self, x, y, x_val=None, y_val=None, epochs=2, batch_size=20): self.model.fit(x, y, epochs=epochs, verbose=0, validation_data=(x_val, y_val)) if __name__ == '__main__': X_train, X_test, y_train, y_test = train_test_split(x, y) kf = KerasModel() kf.fit(X_train, y_train, x_val=X_test, y_val=y_test) kf.store(autoload=True) ● Define model, rest is taken care of ● Scoring done automatically ○ Including user-defined/multiple metrics ● Access to previous runs’ data ● Multiple models & versions
- 18. Quick demo Demo link: https://asciinema.org/a/xxoebEfyu1bzO84hWWAams577 Docs: https://r-racket.readthedocs-hosted.com/en/latest/ Code: https://github.com/carlomazzaferro/racket
- 19. Moving Forward ● Needs ○ Support for batching requests ○ Better testing ○ Deployment guide ■ Kubernetes & co. ○ Feature parity on CLI vs REST access
- 20. Questions ?