Data science at scale with Kafka and Flink (Razorpay)
0 likes125 views
The document presents an overview of data science at scale, specifically focusing on the Mitra system developed by Shashank Agarwal at Razorpay. Key features include real-time predictions, dynamic model serving, and a robust rule engine utilizing Apache Flink for low-latency processing. It highlights the importance of model deployment strategies, tools like Kubeflow and MLflow, and the distinction between research and live models in achieving business goals.
Data science at scale with Kafka and Flink (Razorpay)
- 1. Data science at scale
- 2. Introduction Shashank Agarwal Associate Director, Engineering Founder, ThirdWatch (Acquired by Razorpay) www.linkedin.com/in/shashank734/ 2 @shashank734
- 3. Data Science @ Razorpay 3
- 5. Key Features of Mitra 5 ● Predict results within 200 milliseconds in the distributed environment ● Generate Hundreds of features on the fly during model serving ● Serve results from deployed ML models ● Dynamic rule engine on Flink streams ● Heavily use Flink’s in memory states for storing features and data ● Use CEP (Complex event processing) to handle async events
- 6. Core Engine 6 Data Source App Control App ● CEP ● Join Streams ● Features ● Enrichment ● ML serving ● Rule Engine ● Sink ● Database ● Model Serving Connector Async IO Data Lake & Model training app
- 7. Why Apache flink? 7 ● Low latency ● CEP (Complex event processing) and event time streams ● Flink’s in-memory states ● Async IO ● Automatic Checkpointing for fault tolerance. ● Active community support and active development. For example, Flink just introduced the most awaited features schema evolution and State TTL.
- 8. Model Deployment 8 On Small Scale On High Scale ● Network Load ● Resource allocation ● Limit traffic size
- 9. Model Serving Scenarios 9 ● Static model in-memory integration (classes, packages etc.) ● Static model Standalone service (Web Service, Socket connections etc) ● Online learning service with integrated training ● Parameter server with integrated model updates Tools : Kubeflow, Airflow, Mlflow, H2O
- 10. Some Takeaways 10 ● There is a difference between research and live models. ● Data richness increases the accuracy. ● Accuracy have different meanings for different businesses. ● Don’t over do, start with a simple approach and it’s not wrong to use business logic along with predictions. ● You should implement some scenario for A/B testing on Live data. ● Start productionize the models..
- 11. Thanks Ready for new challenge, We are Hiring : [email protected] @shashank734www.linkedin.com/in/shashank734