Agility in Software 2.0 - Notebook Interfaces and MLOps with Buttresses and Rebars
0 likes•82 views
The document discusses the evolution of software development from traditional methods (Software 1.0) to Software 2.0, emphasizing the role of machine learning and data curation in creating smarter systems. It highlights the importance of agile methodologies and introduces computational notebooks for data science as a new paradigm for experimentation. Additionally, the document presents MLOps as a critical framework to standardize and streamline the management of machine learning lifecycles, ensuring reliable and efficient production systems.
Agility in Software 2.0 - Notebook Interfaces and MLOps with Buttresses and Rebars
- 1. Agility in Software 2.0 – Notebook Interfaces and MLOps with Buttresses and Rebars 6th Int’l. Conference on Lean and Agile Software Development January 22, 2022 RISE Research Institutes of Sweden CC BY-NC 2.0 K. Edblom Markus Borg @mrksbrg mrksbrg.com
- 2. Agile iterates fast. Data science moves faster! (CC CC BY-NC 2.0 K. Edblom
- 3. Who is Markus? • Development engineer, ABB 2007-2010 – Process automation – Editor and compiler development • PhD student, LundUniversity 2010-2015 – Requirements engineering and testing – Traceability, change impact analysis • Senior researcher, RISE 2015- – Software engineering for AI/ML
- 5. Also Markus… • Member of the board, Swedsoft – Influence decision makers – Write comment letters – Facilitate networking • Adjunct lecturer, Lund University – Teaching software engineering
- 6. Software 2.0 Computational Notebooks MLOps (CC BY-NY-ND 2.0 Flick: *Hajee)
- 9. ”a large portion of real-world problems have the property that it is significantly easier to collect the data than to explicitly write the program” https://medium.com/@karpathy/software-2-0-a64152b37c35 Andrej Karpathy Director of AI at Tesla 9
- 10. Karpathy’s Software 2.0 Software 1.0 • Humans write source code • Other humans comprehend the source code Software 2.0 • Humans curate data and specify goals • Backprop. and gradient descent produces millions of weights • Humans cannot comprehend mapping from input to output
- 11. “computers’ ability to learn without being explicitly programmed” - Arthur Samuel (1959) 11
- 13. Neural network YOLO (You Only Look Once) by Redmon et al. (2016) 13
- 14. CC BY-NC 2.0 Flickr: @andreas_komodromos
- 16. 16
- 17. 17 Established software quality assurance no longer sufficient
- 18. 18 MLware is different Software engineering practices throughout the lifecycle must evolve – New types of requirements (explainability, fairness, …) – New architectures (system, neural networks, …) – Configuration management (data, models, parameters, …) – Testing levels (data, model, …) – Operations (scaling, monitoring, retraining, …)
- 21. 21 Data science is not software engineering Established best practices might not apply Biggest difference is how experimental it is • Maximum agility needed to quickly reach insights CACE principle • “Changing Anything Changes Everything”
- 22. 22 Also different peopleware Data scientists are not software engineers • and maybe not software developers • perhaps not even computer scientists Data scientists master the art of taming data and train models Analogy: research on development of scientific software
- 23. 23
- 24. “Literate programming” - Donald E. Knuth (1984) 24 Mix source code and explanatory text
- 25. Computational notebooks Extended into “literate computing” with three types of cells • (Python) Source code • Explanatory text describing the code • Visual content Promotes interaction! Interpreter runs in the background Cells can be executed in any order Very agile, but very messy
- 26. 26
- 27. 27 Notebook collaboration pain points Concurrent editing is confusing Code management and refactoring Replicability is low Productization of a Notebook proof-of-concept is a big step CHI’20
- 28. 28 28 Very agile! Loads of tools!
- 29. 29
- 30. 30 30 Jakobsson and Henriksson (2021) https://lup.lub.lu.se/student-papers/search/publication/9066685 https://github.com/backtick-se/cowait
- 31. dataset versioning? dependency management? deployment? real-time performance? version control? repeatability? scalability? monitoring? feature store? model management? Data Features Training Model MLware sounds simple! 31
- 32. Sculley et al. (2015), Hidden Technical Debt in Machine Learning Systems In Proc. of the Advances in Neural Information Processing Systems 28
- 33. MLOps
- 34. 34
- 36. 36 “MLOps is the standardization and streamlining of machine learning life cycle management” As an engineering discipline, MLOps is a set of practices that combines Machine Learning, DevOps and Data Engineering, which aims to deploy and maintain ML systems in production reliably and efficiently. - Treveil et al., Introducing MLOps, O'Reilley Media, Inc., 2020.
- 37. (CC BY 2.0 Flick: Kuhnt) DevOps with ML specific additions • Experiment tracking • Model management 37
- 38. MLOps Please! How? Fully managed end-to-end solution by the hyperscalers Custom-built on-prem pipelines 38
- 40. 40 Data Validation Model Training Model Validation Packaging Deploy in Test Env. Deploy in Prod. Env. Training Deployment Model Monitoring Software Development ML Development Data Repos Code Repos ML Ops Release Management 40
- 41. Published April 21, 2021
- 42. High Risk Unacceptable Prohibited: Social Scoring, Public Facial Detection, User Manipulation, … Minimal Risk Business as usual: Video games, Camera Effects, Spamfilters, … Limited Risk Increased Transparency: Chatbots, Deep Fakes, Emotion Recognition, … Conformity Assessment: 1) X under product safety regulations 2) Education, employment, healthcare, immigration, justice, …
- 43. 43 High-risk AI providers must • Document internal rigorous engineering activities • Quality assurance, fairness, traceability, auditability, … • Pass independent conformance assessment • National Supervisory Authority • Monitoring to continuously check compliance If not? GDPR style punishment… • Up to 6% global annual turnover!
- 44. 44
- 45. Agile iterates fast. Data science moves faster! (CC CC BY-NC 2.0 K. Edblom
- 46. Agility in Software 2.0 – Notebook Interfaces and MLOps with Buttresses and Rebars 6th Int’l. Conference on Lean and Agile Software Development January 22, 2022 RISE Research Institutes of Sweden CC BY-NC 2.0 K. Edblom Markus Borg @mrksbrg mrksbrg.com