Building a Turbo-fast Data Warehousing Platform with Databricks
32 likes4,127 views
The document presents a comprehensive overview of building a high-performance data warehousing platform using Databricks, focusing on key aspects such as infrastructure, data ingestion, ETL processes, and performance optimizations. It highlights the advantages of Databricks over traditional data warehousing methods, emphasizing its speed, flexibility, and ease of use powered by Apache Spark. Additionally, the content addresses security measures, integration with AWS services, and real-world use cases of the platform.
Building a Turbo-fast Data Warehousing Platform with Databricks
- 1. Building a Turbo-fast Data Warehousing Platform with Databricks Parviz Deyhim
- 2. Agenda • Introduction to Databricks • Building a end-to-end Data warehouse platform • Infrastructure • Data ingest • ETL • Performance optimizations • Process & Visualize • Securing your platform • Conclusion
- 3. 3 Parviz Deyhim (Speaker) Parviz works with variety of different customers and helps them with adopting Apache Spark and architecting scalable data processing platform with Databricks Cloud. Previous to joining Databricks, Parviz worked at AWS as a big-data solutions architect. Denny Lee (Moderator) Denny is a Technology Evangelist with Databricks. Previous to joining Databricks, Denny worked as a Senior Director of Data Sciences Engineering at Concur and was part of the incubation team that built Hadoop on Windows and Azure (currently known as HDInsight). About the Speakers
- 5. We are Databricks, the company behind Spark • Founded by the creators of Apache Spark • Contributed ~75% of the Spark code in 2014 • Created Databricks cloud, a cloud-based big data platform on top of Spark to make big data simple
- 6. Typical big data project is far from ideal Weeks to prepare then explore data, and find insights Import and explore data Months to build, weeks to provision in existing Get a cluster up and running Months of re-engineering to deploy as an application Build and deploy data applications For each new project, it takes months until results
- 7. How Databricks powered by Spark helps our customers No infrastructure management Interactive workflow Collaboration across the organization Experiment to production instantly 100x faster than MapReduce Spark SQL + ML + Streaming + Graph processing Speed Flexibility Ease-of-use Unified
- 8. Databricks helps you to harness the power of Spark “Light switch” Spark clusters in the cloud 3rd Party Applications Interactive workspace with notebooks Production Pipeline Scheduler
- 9. Databricks Internal Data Warehouse Use Case
- 10. 10 Databricks Internal DWH Use Case Today: Collect logs from deployed customer clusters Our Goal: ○ Understand customers behavior ○ Create reports for various teams (e.g. customer success & support)
- 11. Stages Build & Maintain Infrastructure Data Ingest Process & Visualize Transform & Store
- 12. 12 Stages Build & Maintain Infrastructure Data Ingest Process & Visualize Transform & Store
- 13. 13 Challenges of Building a Data Warehouse Datacenter or Cloud? • Build/rent data center or use a public cloud offering? Picking the right resources • If datacenter: what server sizes and types? Storage? • In cloud: what instance size, how large of a disk/SSD to use? Deployment and Automation • How to automate the deployment process: • Chef, Puppet, Cloudformation and etc
- 14. 14 Maintenance • How to perform seamless upgrades? Securing the platform • How to encrypt datasets? • Controls, Policies, Audits Challenges of Building a Data Warehouse
- 15. 15 Databricks Hosted Platform Managed and automated hosted platform • Fully deployed on AWS • Create resources with a single click • Zero touch maintenance
- 16. 16 Compute Resources Automatic Instance Provisioning • R3.2xlarge instances • Use SSD for caching • No EBS • Deployed in major regions and more coming
- 17. 17 Networking: VPC Security & Isolation with AWS VPC
- 18. 18 Networking: Enhanced Networking High performance node to node connectivity with placement groups
- 19. 19 Integration with AWS services S3 Kinesis RDS Redshift ...
- 21. 21 Stages Build & Maintain Infrastructure Data Ingest Process & Visualize Transform & Store
- 22. 22 Customer Data Sources Customer have variety of different data sources Cloud storage: S3 Databases: MySQL, NoSQL APIs: Facebook, SalesForce and etc Often required to join datasets
- 23. 23 Traditional Approach Traditionally data warehouses require data to be copied Common Question: How do I move my datasets to Databricks?
- 24. 24 Traditional Approach Required to create a schema before data is copied
- 25. 25 Traditional Approach: Challenges Moving Data: • Very expensive and time consuming • Creates inconsistency as data gets updated Predefined Schema: • Challenging to change schema for different use-case
- 26. 26 Databricks Approach: Data Sources De-coupling compute from storage ● Leverage S3. No HDFS Read directly from data sources ● Eliminate the need to copy data Schema definition on read ● SparkSQL
- 27. 27 Spark Data Sources Support
- 28. 28 Databricks Use Case Different data sources • Customer metrics on S3 • Internal CRM Need a single view of our customers
- 29. 29 Databricks Use Case We use Spark to join datasets
- 30. 30 Databricks Demo 1. Reading data from external API 2. Reading usage logs data from S3 3. Joining usage and external datasets Link
- 31. 31 Stages Build & Maintain Infrastructure Data Ingest Process & Visualize Transform & Store
- 32. 32 Data Transformation Need to transform data before the join operation • Aggregation • Consolidation • Data cleansing
- 34. 34 ETL: Common Approaches Two common approaches • Offline • Streaming/real-time
- 35. 35 Extract & Transformation Offline ET • Data gets stored in raw format (as is) • Some recurring job perform ET on the dataset • New transformed dataset gets stored for later processing Advantage • Easy and quick to setup Disadvantages • Traditionally slow process
- 36. 36 Databricks Jobs Databricks Jobs • Schedule Production workflows using Notebooks or Jars • Create pipelines • Monitor results
- 39. 39 Performance Optimizations Storing data in parquet Partitioning dataset Spark caching • JVM • SSD
- 40. 40 Spark allows data to be stored in different data sources
- 41. 41 Parquet: Efficient columnar storage format for data warehousing use-cases
- 42. 42 Optimization: Parquet Columnar • Faster Scans Better compression Optimized for storage • Memory • Disk
- 43. Advantages • Fast memory access Disadvantages • GC pressure • No durability after JVM crash 43 Optimization: Caching (JVM) Spark caching: JVM
- 44. 44 Optimization: Caching (SSD) Spark caching: SSD Advantages • Survives JVM and instance crash Disadvantages • Much slower than JVM caching
- 45. 45 Databricks Use Case: Storing aggregate data in Parquet
- 46. 46 Databricks Use Case: Storing aggregate data in Parquet
- 48. 48 Stages Build & Maintain Infrastructure Data Ingest Process & Visualize Transform & Store
- 49. 49 Databricks Visualizations Notebook Visualizations 1. Built-in graphing capabilities 2. ggplot and matplotlib 3. D3 visualizations
- 50. 50 Databricks Visualizations Notebook Visualizations (DEMO) D3/SVG
- 53. 53 Secure Platform Encryption 1. In flight: SSL 2. At rest: S3 Encryption
- 54. 54 Secure Platform User Management: ACLs Notebooks read-write-execute Admin users
- 55. 55 Secure Platform On Our Roadmap S3 KMS encryption Single Sign On (SSO) AD/LDAP support
- 56. 56 Secure Platform On Our Roadmap IAM Roles for Spark nodes
- 57. Thank you