Dataiku & Snowflake Meetup Berlin 2020
1 like399 views
The document is an introduction to Snowflake, a cloud data platform founded in 2012, highlighting its rapid growth, including a valuation of $12.4 billion and over 3,500 customers. It outlines Snowflake's architecture, benefits for data processing, and capabilities for various workloads including data warehousing, engineering, and analytics. Additionally, the document features a demo flow showcasing how to utilize Snowflake for data exploration and analysis.
Dataiku & Snowflake Meetup Berlin 2020
- 1. © 2020 Snowflake Inc. All Rights Reserved
- 2. © 2020 Snowflake Inc. All Rights Reserved Introduction to Snowflake Dataiku Berlin Meetup 25 February 2020 Harald Erb | Sr. Solutions Engineer
- 3. © 2020 Snowflake Inc. All Rights Reserved Quick Intro to Snowflake
- 4. © 2020 Snowflake Computing Inc. All Rights Reserved SNOWFLAKE TIMELINE 4 Founded in 2012 by industry veterans with over 120 database patents ~$1.5BN in venture capital funding from leading investors ~$12.4BN valuation First customers 2014, general availability 2015 1.800+ employees Over 3500+ customers today Queries processed in Snowflake per day: > 300 million Largest single table: > 68 trillion rows Largest number of tables single DB: > 200,000 Single customer most data: > 55PB Single customer most users: > 10,000 FUN FACTS Gartner and Forrester “Leader”
- 5. © 2020 Snowflake Inc. All Rights Reserved On Premises EDW 1st Gen Cloud EDW Data Lake, Hadoop Cloud Data Platform All Data All Users Fast Answers SQL Database Value of Data Time You can’t use yesterday’s technology to solve today’s data problems -- and definitely not tomorrow’s. JOURNEY TO A CLOUD DATA PLATFORM
- 6. © 2020 Snowflake Inc. All Rights Reserved A REAL-WORLD PROBLEM Data Warehouse Appliance Resource Usage Heatmap of weekly usage profile showing critical peak usage times, but also low Avg. CPU usage of 33% per week Typical 24h usage profile showing the two main workload groups “competing for Data Warehouse Appliance resources” And what about Analytics workloads? How can we support new Data Initatives?? Data Loading, ETL, Aggregation Reporting, BI Workload Groups over time
- 7. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE ARCHITECTURE 11 Scale Out Services Multi-Cluster Compute Centralized Storage Cloud Agnostic Layer
- 8. © 2020 Snowflake Inc. All Rights Reserved
- 9. © 2020 Snowflake Inc. All Rights Reserved Data Science Data Transformation Marketing Analytics / Reporting / BI XS S L M Data Load Structured & Semi-Structured Finance App Functional Architecture L Enabling one or multiple Data Teams/Projects to drive innovation…
- 10. © 2020 Snowflake Inc. All Rights Reserved Data Transformation Marketing Analytics / Reporting / BI XS S M L Data Load Structured & Semi-Structured Finance App L L XL Data Science Functional Architecture …they are even allowed to scale-up compute resources when needed…
- 11. © 2020 Snowflake Inc. All Rights Reserved Data Transformation XS S L Data Load Structured & Semi-Structured Finance App L L XL Data Science Functional Architecture Marketing Analytics / Reporting / BI M M … without slowing down other active users of the Cloud Data Platform.
- 12. © 2020 Snowflake Inc. All Rights Reserved Marketing Analytics / Reporting / BI M M Data Transformation XS S L Data Load Structured & Semi-Structured Finance App L Functional Architecture Secure Sharing & Collaboration Your Private Data Exchange Your Business Ecosystem Public Data Exchange Your Employees M
- 13. © 2020 Snowflake Inc. All Rights Reserved Marketing Analytics / Reporting / BI M M Data Transformation XS S L Data Load Structured & Semi-Structured Finance App L Secure Sharing & Collaboration Your Private Data Exchange Your Business Ecosystem Public Data Exchange Your Employees M Functional Architecture Clone M Test/Dev
- 14. © 2020 Snowflake Inc. All Rights Reserved 19 ONE PLATFORM, SHARED DATA, MANY WORKLOADS Data Warehouse Data Lake Data Engineering Data Exchange Data Applications Data Science Data Monetization Operational Reporting Ad Hoc Analysis Real-time Analytics OLTP Databases Enterprise Applications Third-Party Web/Log Data IoT Data Sources Data Consumers ETL,Streaming Live-Demo!
- 15. © 2020 Snowflake Inc. All Rights Reserved Demo
- 16. © 2019 Snowflake Computing Inc. All Rights Reserved 21 AWS PrivateLink AWS Direct Connect Frankfurt CDN S3 Endpoint Snowflake S3 bucket OCSP cache External Stage (COPY data) Internal Stage (GET data, Large Results) Customer On-premise environment Snowflake Driver/Clients TYPICAL SNOWFLAKE SETUP & DEMO FLOW Customer Cloud Snowflake Customer Account Data Exploration / Data Science Python/JupyterLab 4. Use Snowflake Python Connector to access / prepare Time series data 5. Train TS Model, predict future values and plot forecast along with historical data 6. Write the predictions back into new Snowflake table Customer “Data Lake” Snowflake Web UI 1. Provisioning of a Compute Cluster via SQL command! 2. Resize Compute Cluster and load data from external S3 Bucket 3. Analyze data using SQL and prepare a secure Database View for other users DEMO FLOW
- 17. © 2020 Snowflake Inc. All Rights Reserved
- 18. © 2020 Snowflake Inc. All Rights Reserved
- 19. © 2019 Snowflake Computing Inc. All Rights Reserved© 2020 Snowflake Inc. All Rights Reserved Connecting the dots
- 20. © 2020 Snowflake Computing Inc. All Rights Reserved SNOWFLAKE REFERENCE ARCHITECTURE FOR ANALYTICS 25
- 21. © 2020 Snowflake Computing Inc. All Rights Reserved 26 SNOWFLAKE: A SCALABLE + POWERFUL DATA PROCESSING BACKEND FOR DATAIKU!
- 22. © 2020 Snowflake Inc. All Rights Reserved THANK YOU