Downloaded 46 times
Uploaded byDatabricks
The Power of Unified Analytics with Ali Ghodsi
Databricks was founded in 2013 to bring Apache Spark to the enterprise. Their goal is to unify data science and engineering, which traditionally operate in silos. They have developed two main products to achieve this. Databricks Delta makes data lakes reliable and performant for AI use cases. MLflow unifies the workflows of data scientists and engineers by enabling them to track experiments, build reliable data pipelines, and deploy models into production. Overall, Databricks aims to make AI possible for more organizations by breaking down the barriers between data teams.
More Related Content
Similar to The Power of Unified Analytics with Ali Ghodsi
The Power of Unified Analytics with Ali Ghodsi
- 1. Databricks Unifying Data Science andEngineering 1
- 2. AMPLab funded bytech companies: The beginnings of Apache Spark at UC Berkeley ● Got a glimpse at their most impactful internal projects ● They were leveraging massive amounts of data ● Doing high impact machine learning/AI We wanted to democratize Data + AI
- 3. Apache Spark Streaming Spark SQL +DataFrames MLlib MachineLearning GraphX Graph Computation Spark Core API R SQL Python Scala Java
- 4. Databricks started in2013 Bring Apache Spark to the Enterprise
- 5. Only 1% ofenterprises successful with AI
- 6. Data Engineers Other 99%struggle due to organizational silos Data Scientists IT Line of Business
- 7. Databricks goal isto unify data science & engineering
- 8. Data is notready for AI 3 challenges created by data & AI divide 1 2 Data and AI technology silos 3 Data scientists and engineers are in silos
- 9. Data is notready for AI1
- 10. Massive data indata lakes Data Lake
- 11. Vision to doAI on that data Data Lake AI
- 12. Data Lake Data isnot ready for AI AI Inconsistent Data Lack of Schema Slow Performance and Costly
- 13. Databricks Delta Brings datareliability and performance to data lakes Fast Analytics +Data Reliability Blob Storage
- 14. Data Lake Databricks Delta:makes data ready for AI Data Reliability Schema Enforcement ACID Transactions Query Performance Very Fast at Scale Indexing (10-100x Faster) Reporting Machine Learning Alerting Dashboards Delta
- 15. Data and AItechnology silos2
- 16. Data & AItechnology silos Great for Data, but not AI Great for AI, but not for data TFServing TensorBoard Supporting and Deployment Libraries
- 17. Data & AItechnology silos Great for Data, but not AI Great for AI, but not for data TFServing TensorBoard Supporting and Deployment Libraries
- 18. 3 Data scientistsand engineers are in silos
- 19. Data scientists &engineers are in silos Data Engineers Data Scientists Challenging to track and reproduce experiments Build Model2 Have to ensure reliability, SLAs, and quality Deploy Model3 Data Prep Hard to make pipelines reliable 1
- 20. Databricks MLflow: unifies datascientists & engineers Data Engineers Data Scientists
- 21. Build reliable datapipelines Track the datasets Databricks Delta Track Experiments Reproduce experiments MLflow Project & Tracker Databricks Runtime for ML Deploy models in production, track their quality MLflow Serving Data Prep Deploy Model Build Model Databricks MLflow: unifies data scientists & engineers Data Engineers Data Scientists 1 3 2
- 22. Build reliable datapipelines Track the datasets Databricks Delta Track Experiments Reproduce experiments Databricks Runtime for ML MLflow Project & Tracker Deploy models in production, track their quality MLflow Serving Data Prep Deploy Model Build Model Databricks MLflow: unifies data scientists & engineers Data Engineers Data Scientists 1 3 2 Announcing: time travel +Delta
- 23. Confidential – forGartner briefing only
- 25. Databricks makes AIpossible for the other 99% by unifying data science and engineering
- 26. Databricks Unified Analyitcs demoby Michael Armbrust