MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
1 like264 views
The document discusses the importance of MongoDB in transforming a pandas proof-of-concept into a scalable product for data science projects. It outlines the limitations of pandas, such as lack of persistence and indexing, and highlights MongoDB's benefits including schema-less design and horizontal scalability, which allow for efficient real-time and batch processing. Additionally, it addresses best practices for utilizing MongoDB in a data pipeline and emphasizes its role in enhancing speed and resource efficiency.
MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
- 1. MongoDB in Data Science How to convert a Pandas Proof-of-Concept to a scalable product and why MongoDB is the key to success !
- 2. Who I am Software Engineer Compiler Engineer Compiler Engineer LLVM contributor Software Engineer R/D Lead ML Engineer Backend Infrastructure Sr. ML Engineer
- 3. What will we learn ? ● Understand existing tools for delivering Data Science projects and when to use them. ● Why MongoDB could be crucial for your product and business ● How to easily productionize a Pandas Proof-of-Concept ● How to use MongoDB while being open to other technologies.
- 4. Motivation
- 5. Speed of inference Speed of development Key factors
- 7. Research Data Scientist Productionization Data/ML Engineer Speed of development Key factors
- 8. What is Pandas? Most popular Python framework for data manipulation and data wrangling in Data Science community.
- 9. What is Pandas? Most popular Python framework for data manipulation and data wrangling in Data Science community. Source: numpy.org, scipy.org, matplotlib.org, scikit-learn.org, pandas.pydata.org
- 10. Source: Stackoverflow post by David Robinson
- 11. Why use Pandas Dataframes ?
- 12. Why use Pandas Dataframes ?
- 13. Why use Pandas Dataframes ?
- 14. Why use Pandas Dataframes ?
- 15. Why use Pandas Dataframes ?
- 16. Drawbacks of Pandas ● Doesn’t have persistence layer ● Doesn’t support primary and secondary indexes ○ As a result, not efficient for querying ● Doesn’t support multi-threading
- 18. Productionization options Real time service Batch Job Slow Inference
- 19. Productionization options Real time service Batch Job Slow Inference Fast Inference
- 20. Real time service demo (recommendation) Event Store
- 21. Real time service demo (recommendation) Event Store Model Training Job
- 22. Real time service demo (recommendation) Event Store Model Training Job Model store
- 23. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 24. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 25. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 26. Real time service demo (recommendation) Event Store Feature Aggregation Model Inference Inference Service request respond
- 27. Real time service demo (recommendation)
- 28. Real time service demo (recommendation)
- 29. Real time service demo (recommendation)
- 30. Things to avoid ● Don’t forget to put indexes on your collection ● Don’t put indexes on every field ● Don’t read and write from the same replica
- 31. But… we generate a tons of user events! Is this solution going to work for us?
- 32. user events Consumer 1 Consumer 2 Consumer N MongoDB Postgres DFS Typical data pipeline
- 33. user events Consumer 1 Consumer 2 Consumer N MongoDB Postgres DFS Typical data pipeline
- 34. MongoDB TTL index Filters event_type ... Consumer Shrink down the amount of data
- 35. Real time service demo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 36. Training Job Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 39. MongoDB Connector Event Store Inference Job Inference as a batch job
- 41. Batch Job versus Real Time Service Real Time Service Batch Job Pros On demand (scales as needed) Easier to develop and maintain Cons Harder to develop and maintain Constantly utilizing resources
- 42. Benefits of MongoDB ● Schema-Less ● Horizontally scalable ● Available as PaaS from many vendors. ● Has a huge community ● Easier to hire people
- 43. Summary ● Allows to provide a real time experience ● Could help save expensive computational resources ● Provides a way to do real time as well as batch inference
- 44. We are hiring !!! careers.shopbonsai.ca
- 45. References ● https://stackoverflow.blog/2017/09/14/python-growing-quickly/ ● https://www.mongodb.com/products/spark-connector ● https://pandas.pydata.org/ ● https://scikit-learn.org/ ● https://matplotlib.org/ ● https://www.scipy.org/ ● https://www.numpy.org/ ● https://iconscout.com/icon/device-management-mobile-computer-seo-tool-analyze-7
- 46. Thanks !!!