SOLVING MLOPS FROM FIRST PRINCIPLES, DEAN PLEBAN, DagsHub
0 likes76 views
The document outlines a framework for solving MLOps challenges using first principles thinking. It emphasizes the importance of defining problems and parameters, researching solutions, and integrating findings for efficient data versioning and workflow management. Key themes include the transformative nature of MLOps, the importance of open-source solutions, and the need to evaluate tools systematically.
SOLVING MLOPS FROM FIRST PRINCIPLES, DEAN PLEBAN, DagsHub
- 1. DagsHub Solving MLOps from first principles November ‘21, Dean Pleban
- 2. What we’ll cover 1. First principles thinking and mental models – a brief introduction 2. MLOps buyer’s remorse 3. Assumptions about the problem 4. Assumptions about ideal solutions 5. Example 1: Data Versioning 6. Generalizing to a framework – CONFIDENTIAL –
- 3. About Me Dean Pleban Building tools for ML teamwork Strongly believe in open source Follow me: @DeanPlbn DeanPleban
- 4. Definitions Mental models First principles thinking
- 6. The ML world is changing – CONFIDENTIAL – OF TEAMS HAVE MODELS IN PRODUCTION 0% 10% 20% 30% 40% 50% 60% 70% 0 1-10 10-100 Over 100 # of models in production 80% Not just GAFAM… THE NEXT CHALLENGE IS SCALING FROM 1 TO 10 (OR 10 TO 100) (Poll with over 2000 votes)
- 7. MLOps Fatigue – Too many tools, manually synchronized – CONFIDENTIAL – MLOPS TOOLS TO CHOOSE FROM & INTEGRATE WITH 280
- 8. The signs Everything is manual Analysis Paralysis All or nothing Building everything in- house
- 9. Minimal Assumptions about the PROBLEM You are not Google / Facebook MLOps is still in early days Save time / future proof / production ready tradeoff
- 10. Minimal Assumptions about the SOLUTION Problem vs Feature focus Hard part starts when the first model goes to production Data scientists != developers and how this affects tooling Building on OSS makes sense for most cases
- 11. Example 1: Data Versioning I want to version my data My data is regularly changing and I want to revert back to an older version for disaster recovery / governance Step 1: Define the problem
- 12. Step 1: Define the problem Revert in case of bug Compare different versions Knowing which data is used where Add/modify data without breaking
- 13. Step 1: Define the problem • Do you actually suffer from “all the above”? • Prioritizing is important, separating must-have and nice-to- have
- 14. Example 1: Data Versioning The type of data you work with The type of data changes you expect What are the organizational constraints Who am I? Step 2: Define the problem parameters
- 15. Step 2: Define the problem parameters • Flexibility to anything is tempting, but answering each question differently will lead to very different tooling, so being specific is important • Organizational constraints are specifically critical, since they are many times the largest limitations on the tools to use. This also ties into modularity. E.g. does your org only work with Azure cloud tools? • This can also be the step where we define a “user story” or workflow that includes this problem – e.g. are we going to version the DB directly, or just the outputs of our queries?
- 16. Example 1: Data Versioning Step 3: Google the problem
- 17. Step 3: Google the problem • Specifically, budget a reasonable amount of time (at least 2-3 hours) to research existing solutions • Now that you’ve defined the problem, and not just features, search for those (as well as experimenting with problem parameters), this will give you more tools, that prioritize different problem aspects • Build out an info page so that other people in the org can review and add inputs • You will probably learn that you were searching for the wrong keywords • Read blogs and forum posts and see what TERMS people are using, and search again • Ask friends, use Reddit as a tool to discover keywords – describe your problem and people will recommend the tools and categories you need.
- 18. Step 3: Google the problem Reddit example Googling examples Example of a tool research output Recommend ed blogs
- 19. Example 1: Data Versioning Pre-technical evaluation Operating principles “Hello World” Kick the tires – mechanically Step 4: Evaluate solutions
- 20. Step 4: Evaluate solutions • Is there a hosted solution? • How much does it cost? • If I go for a hosted solution, how easy will it be to bring it in- house in the future, or customize it to my needs • How easy is it to get out of them • How easy is it to get out of them if they prove less useful
- 21. Step 4: Evaluate solutions Comparing 2 data versioning tools from a “face value” perspective Looking at the operating principles of DVC Get started tutorial Try to add a dataset with 10K images
- 22. Example 1: Data Versioning Start simple – 1 project, 1 user Define criteria for success, or don’t Review and extrapolate Step 5: Integrate
- 23. The 5 step process 1. Define the problem 2. Define the problem parameters 3. Google the problem 4. Evaluate solutions 5. Integrate
- 24. Thank You!