Rusty Waters: Elevating Lakehouses Beyond Spark
- 1. Rusty Waters Elevating Lakehouses Beyond Spark Satya Mandavilli Solution Architect S&P Global Commodity Insights Madhukar Nathala Associate Director S&P Global Commodity Insights
- 3. About Us • At S&P Global Commodity Insights, our complete view of global energy and commodities markets enables our customers to make decisions with conviction and create long-term, sustainable value. • Our team is at the forefront driving innovation and leveraging data to deliver valuable insights in the commodity market. • We combine cutting-edge technology with robust data strategies to empower our clients and enable informed decision-making.
- 4. Data Analyst Data Engineer Data Scientist AI/ML Engineer Data Governance Lead Data Steward
- 5. Our Platform 1k+ Active users 37k+ Tables Onboarded 75k+ Jobs daily 3k+ TBs of Processed Data
- 6. The Context FOCUS ON BUILDING & ENHANCING OUR UNIFIED, ENTERPRISE-GRADE LAKEHOUSE PLATFORM 01 A DYNAMIC COLLABORATION, DRIVING A TRANSFORMATIVE JOURNEY TO TACKLE DATA CHALLENGES. 02 FLEXIBILITY FOR USERS TO CHOOSE THEIR TOOLS AND TECHNOLOGIES 03
- 7. The Problem Spark may not be the optimal choice for every workflow Wait time for resources Smaller workflows becomes expensive Knowledge Gap
- 9. 90% of the workloads is 100MB or less Source: Big Data is Dead - MotherDuck Blog
- 10. The Solution  Native Rust library for Delta Lake, providing efficient and reliable data processing.  Enables data engineers to interact with Delta Lake without Apache Spark, reducing infrastructure costs and simplifying data pipelines through a standalone Rust API.
- 12. Example
- 13. The Effect  Cost per Month: $52  Resource Spin up Time: 6Mins  Job Run Time: 10Mins  Cost per Month: $7.3  Resource Spin up Time: 1.4Sec  Job Run Time: 4Mins Before (Spark + Notebook) After (Delta-RS + Lambda) Workflow: Processing 100MB dataset on an hourly schedule
- 14. The Benefits Easy to onboard Cost Savings High Performance Open Source
- 15.  Fixed with 0.19.0 Change Data Feed
- 16. Concurrent Writes  Multi-Cluster concurrent writes into Delta Lake tables in S3  Fixed with 0.23.0* * Conditional put support is only safe if you don’t have spark/DBR writers on the table
- 17. Closing Thoughts  No single engine is perfect for every data workload!  Avoid getting trapped in a Spark-only mindset; not every challenge is a Big Data problem.  With Rust, we gain advantages such as memory safety, efficient concurrency, and high performance, all of which enhance the Lakehouse capabilities.  Consider the potential savings in both costs and complexity for your Data Platform.  Focus on understanding your data and selecting the right engines for your workloads.