DataOps: An Agile Method for Data-Driven Organizations
7 likes2,449 views
The document discusses the concept of DataOps, an agile methodology that enhances data management and analytics across organizations, facilitating better collaboration and efficiency. It highlights the significance of real-time data processing, the importance of flexible data architectures like global data fabrics, and the advantages of using techniques such as the Rendezvous architecture for machine learning logistics. Additionally, the document emphasizes the value of integrating data-heavy roles into cross-functional teams to improve project outcomes and responsiveness to change.
![© 2018 MapR Technologies 4
How do you measure Earth’s oceans?
By NASA Goddard Space Flight Center from Greenbelt, MD, USA (Full Disk Image of Earth Captured August 24, 2011)
[CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons](https://image.slidesharecdn.com/dataopssessiontalkfinal-v2-adjstratasj2018-180329004423/85/DataOps-An-Agile-Method-for-Data-Driven-Organizations-4-320.jpg)
DataOps: An Agile Method for Data-Driven Organizations
- 1. © 2018 MapR Technologies 1 DataOps: An Agile Method for Data-driven Organizations Ellen Friedman, PhD Principal Technologist 7 March 2018 #StrataData
- 2. © 2018 MapR Technologies 2 Contact Information Ellen Friedman, PhD Principal Technologist, MapR Technologies Committer Apache Drill & Apache Mahout projects O’Reilly author Email [email protected] [email protected] Twitter @Ellen_Friedman #StrataData
- 3. © 2018 MapR Technologies 3 Big Data Applications Used Widely in Production Quoted from: New Vantage Partner Big Data Executive Surveys for 2016 & 2017 http://newvantage.com/wp-content/uploads/2016/01/Big-Data-Executive-Survey-2016-Findings-FINAL.pdf http://newvantage.com/wp-content/uploads/2017/01/Big-Data-Executive-Survey-2017-Executive-Summary.pdf
- 4. © 2018 MapR Technologies 4 How do you measure Earth’s oceans? By NASA Goddard Space Flight Center from Greenbelt, MD, USA (Full Disk Image of Earth Captured August 24, 2011) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
- 5. © 2018 MapR Technologies 5
- 6. © 2018 MapR Technologies 6 Changing How People Work with Data A 19th century big data story Matthew Fountain Maury extracted data from ship’s logs to build amazing charts for navigation
- 7. © 2018 MapR Technologies 7 Big data project: Maury’s Wind and Currents charts
- 8. © 2018 MapR Technologies 8 Big data project: Maury’s Wind and Currents charts At first, nobody was interested in them…
- 9. © 2018 MapR Technologies 9 Big data project: Maury’s Wind and Currents charts At first, nobody was interested in them… …until Captain Jackson shaved a month off the run from Baltimore to Rio de Janeiro Then everybody wanted one!
- 10. © 2018 MapR Technologies 10 © 2014 Ellen Friedman People with “vision” think with their eyes closed
- 11. © 2018 MapR Technologies 11 Aadhaar Project: Largest Biometric DB in the World • Unique 12 – digit number for each person in India • Proof of identity, authenticated anytime, anywhere • Runs on NoSQL database MapR-DB since 2014 Revolution: Changing a Society Photo credit PANOS, with permission 1.3 B people
- 12. © 2018 MapR Technologies 12 Changing Rhythm to How We Work with Data Utility providers using smart meters Collect data every 15 min instead of once a month
- 13. © 2018 MapR Technologies 13 Image © E Friedman Self-driving cars: Huge volume of sensor data Time value of data Analysis at the Speed of Life
- 14. © 2018 MapR Technologies 14 Changing Rhythm to How We Work with Data Apache Drill SQL query engine with schema discovery for data exploration May shorten prep time when running a new query from days/ weeks to hours Follow community on Twitter: @ApacheDrill
- 15. © 2018 MapR Technologies 15 We need a better fit to the way business happens
- 16. © 2018 MapR Technologies 16 A Better Fit • The way business happens • A dataflow (datafabric) that matches the shape of business • Technologies with capabilities to support flexibility and timely response across data centers • Organization at the human level matches as well
- 17. © 2018 MapR Technologies 17 Build a Global Data Fabric Flexibility & agility to respond as life changes
- 18. © 2018 MapR Technologies 18 Global Data Fabric • Comprehensive view of data • Breaks silos • Works with multi-tenancy • Computation (and data) where you want them • Fine-grained control over who has (and does not have) access
- 19. © 2018 MapR Technologies 19 A DataOps approach improves a project’s ability to stay on target & on time
- 20. © 2018 MapR Technologies 20 DataOps: Brings Flexibility & Focus Platform&network Operations Softwareengineering Architecture&planning Dataengineering Datascience Productmanagement DataOps Team B DataOps Team A Cross functional DataOps teams • Expands DevOps to include data-heavy roles • Organized around data-related goals • Better collaboration and communication between roles From Chap 2 of Machine Learning Logistics, by Ted Dunning & Ellen Friedman © 2018 (O’Reilly Media)
- 21. © 2018 MapR Technologies 21 DataOps Principles “DataOps teams seek to orchestrate data, tools, code and environments from beginning to end.” They “…measure performance of data analytics by the insights they deliver.” Thor Olavsrud interview with Ted Dunning & Ellen Friedman for CIO https://www.cio.com/article/3237694/analytics/what-is-dataops-data-operations-analytics.html
- 22. © 2018 MapR Technologies 22 Advantages of a DataOps Approach • Able to pivot & respond to real-world events as they happen • Improved efficiency and better use of people’s time • Faster time-to-value • A good fit to working with a global data fabric
- 23. © 2018 MapR Technologies 23 How do you keep people from feeling threatened by change?
- 24. © 2018 MapR Technologies 24 Don’t be threatening!
- 25. © 2018 MapR Technologies 25 Why Stream? Munich surfing wave Image © 2017 Ellen Friedman
- 26. © 2018 MapR Technologies 26 Stream transport supports microservices
- 27. © 2018 MapR Technologies 27 Stream Transport that Decouples Producers & Consumers P P P C C C Transport Processing Kafka / MapR Streams Good stream transport is persistent, performant & pervasive!
- 28. © 2018 MapR Technologies 28 More on Streaming Microservices • Chapter 3 of Streaming Architecture by Ted Dunning & Ellen Friedman © 2016 (O’Reilly Media) http://shop.oreilly.com/product/0636920049463.do • “Streaming Microservices” chapter by Ted Dunning & Ellen Friedman in Encyclopedia of Big Data Technologies, Sherif Sakr and Albert Zomaya, editors. In press 2018 (Springer International Publishing) • Chapter 4 in A Practical Guide to Microservices & Containers by James A. Scott © 2017 (MapR) https://mapr.com/ebooks/microservices-and-containers/title.html
- 29. © 2018 MapR Technologies 29 Legacy Applications How Does MapR Solve This? Big Data 1.0 Applications Next-Gen Applications MapR Converged Data Platform High Availability Real Time Unified Security Multi-tenancy Disaster Recovery Global Namespace Real-Time NoQL Database Stream TransportWeb-Scale Storage
- 30. © 2018 MapR Technologies 30 With MapR, Geo-Distributed Data Appears Local stream Data source Consumer
- 31. © 2018 MapR Technologies 31 With MapR, Geo-Distributed Data Appears Local stream stream Data source Consumer
- 32. © 2018 MapR Technologies 32 With MapR, Geo-distributed Data Appears Local stream stream Data source ConsumerGlobal Data Center Regional Data Center
- 33. © 2018 MapR Technologies 33 90% of the effort in successful machine learning isn’t the algorithm or the model… It’s the logistics
- 34. © 2018 MapR Technologies 34 Why? • Just getting the training data is hard • ! The myth of the unitary model • Model-to-model evaluation • Respond as the world changes: Agile roll out & roll back when deploy to production
- 35. © 2018 MapR Technologies 35 Metrics Metrics ResultsRendezvous Enter Rendezvous Architecture Scores ArchiveDecoy m1 m2 m3 Features / profiles InputRaw Rendezvous Architecture described in: - Machine Learning Logistics book by Ted Dunning & Ellen Friedman © 2018 (O’Reilly) - “Rendezvous Architecture” chapter in Encyclopedia of Big Data Technologies. Sherif Sakr and Albert Zomaya, editors. Springer International Publishing, in press 2018.
- 36. © 2018 MapR Technologies 36 Best thing about Rendezvous: Agile deployment • Many “good” models ready and waiting – Already running – Ready to deploy into production • To deploy a new model: just stop ignoring it
- 37. © 2018 MapR Technologies 37 Rendezvous to the Rescue: Better ML Logistics • Stream-1st architecture is a powerful approach with surprisingly widespread advantages – Innovative technologies emerging for streaming data • Microservices approach provides flexibility – Streaming supports microservices (if done right) • Containers remove surprises – Predictable environment for running models
- 38. © 2018 MapR Technologies 38 Preparation of Input (and Training) Data Model 1 Model 2 Model 3 request Raw Add external data Input Database The world Raw data may contain features you’ll want in future
- 39. © 2018 MapR Technologies 39 Raw data is gold!
- 40. © 2018 MapR Technologies 40 Decoy Model in the Rendezvous Architecture Input Scores Decoy Model 2 Model 3 Archive • Looks like a model, but it just archives inputs • Safe in a good streaming environment, less safe without good isolation
- 41. © 2018 MapR Technologies 41 Why do you need new models? Conditions may (will) change
- 42. © 2018 MapR Technologies 42 Advantages of Rendezvous Architecture Real model Result Canary Decoy Archive Input
- 43. © 2018 MapR Technologies 43 Rendezvous: Mainly for Decisioning Type Systems • Decisioning style machine learning – Looking for a “right answer” – Simpler than interactive machine learning (such as in self-driving car) • Examples include: – Fraud detection – Predictive analytics / market prediction – Churn prediction (as in telecommunications) – Yield optimization – Deep learning in form of speech or image recognition, in some cases
- 44. © 2018 MapR Technologies 44 Described in new book on ML management: Download free pdf via @MapR: https://mapr.com/ebook/machine-learning-logistics/ Includes a discussion of DataOps
- 45. © 2018 MapR Technologies 45 Example: Tensor Chicken Label training data Run the model Deploy model Gather training data Labeled image files Train model Update model Deep learning project by software engineer Ian Downard (see blog + @tensorchicken)
- 46. © 2018 MapR Technologies 46 DataOps: A Good Way to Adapt to Emerging Data Practices • Faster time-to-value & better ability to pivot – Better collaboration/communication across skill groups – Focused around data-related goals – More efficient use of team members’ time • A good fit to working with a data fabric • A good fit for a streaming microservices style
- 47. © 2018 MapR Technologies 47 Please support women in tech – help build girls’ dreams of what they can accomplish © Ellen Friedman 2015#womenintech #datawomen
- 48. © 2018 MapR Technologies 48 Related events at Strata • “Better Machine Learning Logistics with Rendezvous Architecture” talk by Ted Dunning Wed at 5:10pm • “Rendezvous Architecture” booth talk at MapR booth Thur at 11:30 am • Chat with us in the MapR booth
- 49. © 2018 MapR Technologies 49 Thank You !
- 50. © 2018 MapR Technologies 50 Additional Resources: Available Now O’Reilly report by Ted Dunning & Ellen Friedman © March 2017 Read free courtesy of MapR: https://mapr.com/geo-distribution-big-data-and-analytics/ O’Reilly book by Ted Dunning & Ellen Friedman © March 2016 Read free courtesy of MapR: https://mapr.com/streaming-architecture-using- apache-kafka-mapr-streams/
- 51. © 2018 MapR Technologies 51 Book signings at MapR booth • Wed afternoon break 3:35 pm – 4:15 pm • Thur morning break 10:30 am – 11:10 am Get a free copy of the book & meet the authors Ted Dunning & Ellen Friedman Or download free pdf via @MapR: https://mapr.com/ebook/machine-learning-logistics/
- 52. © 2018 MapR Technologies 52 Please tell me how DataOps works out for you. Ellen Friedman Twitter @Ellen_Friedman Email @[email protected] [email protected]