Experimentation to Industrialization: Implementing MLOps
0 likes454 views
The document outlines the implementation of MLOps by Thorogood Associates and presents a case study on establishing a global MLOps framework for a Fortune Global 500 consumer goods company. It emphasizes the importance of scalability, model reusability, operational efficiency, and ongoing improvements to enhance the value of machine learning and data science across the organization. Key takeaways include the necessity of a well-defined framework, collaboration among data science experts, and the use of integrated tools like Databricks and MLflow to optimize the MLOps process.
Experimentation to Industrialization: Implementing MLOps
- 1. Experimentation to Industrialization: Implementing MLOps Deb Lee & Al McEwan Thorogood Associates
- 2. About Us Deb Lee SENIOR CONSULTANT MLOps Practice Lead [email protected] Al McEwan PRINCIPAL CONSULTANT Solutions Architect, Databricks Champion, Global Head of Capability Development [email protected] Independent, Specialist Data & AI Consultancy US • UK • Singapore • Brazil • India Databricks Partner Since 2018 www.thorogood.com Data Science MLOps & DevOps Data Engineering Data Visualization
- 3. Agenda ØContext ØCase Study ØKey Learnings & Takeaways § People § Processes § Tools
- 4. Companies are investing heavily in ML and AI
- 5. Thinking must shift to embrace operationalization • Sandbox environments • Ad Hoc, Exploratory • Low Commitment • Familiar tool for the data scientist • Done locally or in non- integrated environments EXPERIMENTAL • Automated • Integrated • Reusable • Scalable • Understood and trusted • Cost efficient • Ongoing experiments OPERATIONAL • Sandbox environments • Ad Hoc, Exploratory • Low Commitment • Familiar tool for the data scientist • Done locally or in non- integrated environments EXPERIMENTAL • Automated • Integrated • Reusable • Scalable • Understood and trusted • Cost efficient • Ongoing experiments OPERATIONAL • Sandbox environments • Ad Hoc, Exploratory • Low Commitment • Familiar tool for the data scientist • Done locally or in non- integrated environments EXPERIMENTAL • Automated • Integrated • Reusable • Scalable • Understood and trusted • Cost efficient • Ongoing experiments OPERATIONAL MLOps
- 6. Key Benefits of MLOps SCALABILITY Ability to scale horizontally and vertically, consumption efficiencies from running data engineering and data science at-scale MODEL EVALUATION Maintain and monitor model quality using standardized & consolidated custom KPIs and model evaluation metrics FAST FEEDBACK LOOP Respond to business opportunities and changes quickly, incorporate enhancements to product on regular basis REUSABLE ASSETS Track, monitor, and identify reusable assets (registered models, datasets, pipelines) to increase efficiency & cost savings MODEL TRACEABILITY Create traceability & wider auditability using enterprise model registries, experiment tracking, and monitoring operations for greater observability AUTOMATED MODEL TRAINING Decrease manual dependencies using pipelines configured to kick off automated retraining based on defined triggers REPRODUCIBILITY Save time & create governance for product teams by using tools that enable reproducibility of experiments and model training VERSION SECURITY & COMPATABILITY Maintain security by using licensed packages on tested versions, keep OS versions of clusters up to date, keep all libraries and packages up to date
- 7. Case Study
- 8. Establishing a Global MLOps Framework Customer situation In order to stay ahead, the customer recognized that a global coordinated strategy and framework was needed to realize the benefits of MLOps Investment in experimentation that has proven valuable Data science teams work in focused business areas, following independent practices Fortune Global 500 Consumer Goods Company • 190 countries • 2.5 billion+ consumers daily • 400 brands
- 9. Establishing a Global MLOps Framework Thorogood’s approach Experimentation ML models operationalized MLOps guidance, recommendations & artefacts, project-tested Creation of reusable Code & Pipeline Accelerator templates
- 10. Establishing a Global MLOps Framework Framework impact REUSABILITY As more products are onboarded, a central function will improve reusability of existing assets and help consolidate models and approaches used across products. TIME & COST SAVINGS Reduce duplicative effort & apply responsible cloud consumption principles to all projects, receive cost efficiencies from consolidation of operations. SIMPLIFICATION A centralized function will maintain adherence to MLOps suggested standards to simplify toolsets used and improve ways of working for all teams. CONTINUOUS IMPROVEMENT The MLOps service will have dedicated teams for ongoing operations and one-off activities such as product enhancements & industrialization efforts. SCALABILITY Enable data science projects to scale up more quickly, rapidly realize a vision to unlock business value using data science in all areas of the organization. RELIABILITY Build greater trust and confidence from business users and data science teams by allowing them to realize the value of MLOps delivered using a consistent and high-quality methodology Customer’s Global MLOps Service
- 11. Key Takeaways & Learnings
- 12. Key Takeaways & Learnings People Processes Tools Data
- 13. People Real-world ML Systems Reference: “Hidden Technical Debt in Machine Learning Systems” by D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips 2015 Configuration Data Collection Feature Extraction ML Code Data Verification Machine Resource Management Analysis Tools Process Management Tools Serving Infrastructure Monitoring
- 14. People MLOps Requires Data Scientists who understand both Scale and Reproducibility ML Code: could be relatively small, but key to success Data Scientist skillset specialization Training in making code scalable, efficient and reproducible
- 15. People Blend of capabilities and skills needed depends on the engagement Machine Learning Engineering Data Science Data Engineering Data Visualization Solution Architect Program Management Scenario 1 Operationalization of a use case requiring: • Real-time model serving capabilities • Web application interface and backend • Creation of data engineering and data science pipelines • Scripted management and versioning of compute, datastore, datasets, pipelines Scenario 2 Continuous improvements to baseline monitoring operations requiring: • Create automatically refreshed monitoring dashboards • Enhance tracking of and reporting on drift and other scoring metrics alongside experiment tracking • Design for various target audiences: data scientists, ML support engineers, business users
- 16. Processes Key Takeaways & Learnings – Artefacts Created There are a number of moving parts and handshakes needed for a centralized MLOps service to function and teams to be in sync. Without a defined framework and process, it’s hard to be successful. QUESTIONNAIRE Used to qualify use cases & projects in the pipeline for onboarding to MLOps service ML TEST SCORE Measures the overall readiness of the ML system for production DECISION TREE For anyone embarking on a data science project, guide on tools to use considering training volumes, libraries, serving method, parallelization, retraining frequency PLAYBOOK Guidelines for experimentation and operationalization to streamline the MLOps process REPRODUCIBILITY CHECKLIST Requires code versioning, data versioning, model versioning in model registry, cluster configuration, environment specification Reference: “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction” by E.Breck et al. 2017
- 17. Tools Decision trees helping to guide tool selection at critical junctures How many models are being built? A large model spanning the entire business One model per dimension (i.e. per product) We recommend use of Spark’s MLLib if model is trained on a big dataset (>0.5GB) We recommended use of Spark’s MLLib if cross- validation scenarios exist Non-Spark options can be considered for smaller training datasets Non-Spark options can be considered for this scenario Training & Evaluation Orchestration Deployment Tracking Experimentation Initial Industrialization Model Monitoring & Enhancements Considerations: Decision Points: … … … … … … … …
- 18. Tools Databricks is Optimally Positioned to Support MLOps Databricks Spark: Optimized for large training data volumes per model Best-in-class and widely used for data science experiments Multi-Cloud ready: available on Azure, AWS, and GCP Unifies requisite data engineering & data science capabilities with in-built functions MLFlow provides a powerful platform to manage the ML lifecycle Integrated with serving and reporting technologies
- 19. How to get started Ø Assess your current state Ø Define your target state Ø Refine your approach to People, Tools and Processes Ø Educate yourself on the ‘art of the possible’ • Check out our MLOps Resource Hub for useful content at www.thorogood.com • Most importantly, please reach out to us with any questions or feedback on this topic CONTACT US Deb Lee [email protected] Al McEwan [email protected]
- 20. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.