Delta Lake

For open table formats, new features and integrations can create exponentially more implementation work across projects. At Data + AI Summit, Holly Smith and Robert Pack will walk through how Kernel rewrites the source of truth, so projects that adopt it no longer need additional development for new features. They “just appear.” What started as a contribution to Delta could change how open table formats fundamentally integrate everywhere. 🌍 📍 San Francisco | June 15-18 🔗 Session details: https://lnkd.in/eiuUuF4A #DeltaLake #OpenSource #DataEngineering #DataAISummit

From 2017 to now, Delta Lake has grown significantly, with 40M+ downloads/month and powering the daily processing of hundreds of exabytes. Now the conversation shifts to what comes next. 👇 Join the Data + AI Summit (June 15-18) session “The Road to Delta 5.0” for a look at key shifts ahead: 🔹 Transitioning Delta into a catalog-first table format 🔹 Modernizing Delta on Spark with Data Source V2 APIs 🔹 Convergence of Delta and Iceberg formats 🔹 Harmonizing Delta Kernel across Java and Rust implementations 🔗 Session details: https://lnkd.in/e-rknKK2 #deltalake #opensource #dataaisummit

Back by popular demand! 📘 Delta Lake: The Definitive Guide book signing returns to Data + AI Summit 2026. If you are at Data + AI Summit, come meet the authors, say hi to the Delta Lake community, and pick up a signed copy while supplies last. 🗓️ Tuesday, June 16 🕝 2:00–2:30 PM 📍 Dev Lounge, Data + AI Summit Expo, Moscone Center Books tend to go quickly, so plan to stop by early. Hope to see you there! 👋 cc Denny Lee, Scott Haines, Tristen Wentling, Tyler Croy #DeltaLake #DataAISummit #OpenSource #DataEngineering

For years, Delta Lake and Apache Iceberg co-evolved as parallel standards. That era is ending and this session goes deep on the technical how of that convergence. At Data + AI Summit (June 15–18, San Francisco), Anoop Johnson and Ryan Blue (Databricks) will go deep on the technical how of this convergence. What they'll cover: 🔹 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 𝘃𝟰'𝘀 𝗮𝗱𝗮𝗽𝘁𝗶𝘃𝗲 𝗺𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝘁𝗿𝗲𝗲: what's changing in the redesign, and how the new structure enables single-file commits to significantly accelerate performance 🔹 𝗗𝗲𝗹𝘁𝗮 𝗟𝗮𝗸𝗲 𝟱.𝟬: adopts the Iceberg v4 metadata tree as its native content metadata 🔹 𝗢𝗻𝗲 𝗼𝗻-𝗱𝗶𝘀𝗸 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲: both Delta and Iceberg clients read and write directly, with no translation layers and no conversion overhead 🔗 Add it to your DAIS schedule: https://lnkd.in/eZgZS256 #DeltaLake #ApacheIceberg #DataAISummit #OpenLakehouse #DataEngineering

Catalog-Managed Tables changed the contract for Delta Lake: unified governance, enforceable constraints, and a path toward multi-table updates. The catalog is now the authoritative broker of commits—not the filesystem—and Delta connectors have to adapt. At Data + AI Summit (June 15–18, San Francisco), Scott Sandre will show how Catalog-Managed Tables support was designed in Delta Kernel. 👇 🔹 A unified, catalog-agnostic API for connectors to build against 🔹 Deep engine integrations for DuckDB, Delta-Spark, and Delta-Flink 🔹 Catalog complexity kept out of Kernel—the one right Delta abstraction 🔗 Session details: https://lnkd.in/ej3Rt7ks #DeltaLake #DeltaKernel #DataAISummit #Lakehouse

Data + AI Summit session: Tyler Croy (Scribd, Inc.) will introduce Virtual Delta Tables and the associated open source code designed for multimodal inference.👇 🔹 Incorporate original data from different sources into a singular logical structure for data scientists and ML engineers 🔹 A Delta Lake interface so the ecosystem from Databricks to DuckDB can be used in multimodal work 🔹 How far you can push the lakehouse architecture in a rapidly changing AI data landscape 🗓️ June 15-18 📍 San Francisco 🔗 Session details: https://lnkd.in/e_JrCfHq #DeltaLake #MultimodalAI #DataAndAISummit #Lakehouse

Headed to Data + AI Summit? Don't miss Your Guide to Open Table Formats — Delta, Iceberg, Best Practices, and What's Next! with Benjamin Mathew and Scott Sandre. This session covers Delta 5.0, Iceberg v4 Adaptive Metadata Tree, Unified Delta Kernel (GA), Catalog-Managed Commits, Auto CDF, FILE data type and VARIANT shredding, plus best practices for working across formats today. 👇 🗓️ June 15–18 📍 San Francisco 🔗 Add it to your agenda: https://lnkd.in/emjjX_dQ #DeltaLake #ApacheIceberg #DataAISummit #OpenSource #DataEngineering

Delta Lake and Apache Iceberg™ have converged on similar ideas—columnar metadata, manifest trees, deletion vectors—but two separate metadata structures still duplicate work. At Data + AI Summit (June 15–18, San Francisco), this session introduces the next major evolution of Delta Lake metadata. 👇 🔹 Unified metadata architecture: Delta Lake commits store content metadata directly in Iceberg v4’s adaptive metadata tree 🔹 Delta Lake gains efficient tree-structured manifests and Iceberg interoperability 🔹 Preserving the transactional guarantees that Delta users depend on 🔗 Session details: https://lnkd.in/ehuzk53w #DeltaLake #ApacheIceberg #DataandAISummit #Lakehouse

DuckDB's Delta and Unity Catalog extensions are no longer experimental. DuckDB can now INSERT into Delta tables, time-travel across versions, and read/write through Unity Catalog. 🔷 INSERT writes — INSERT via ATTACH ... (TYPE delta). Multiple INSERTs in one BEGIN/COMMIT = one Delta version. 🔷 Time travel — AT (VERSION => n), attach with VERSION n, or PIN_SNAPSHOT for a stable snapshot. 🔷 Unity Catalog — Catalog Managed Tables use Catalog Commits to keep UC in sync during concurrent writes. 🔷 Incremental snapshot loading — Faster time travel across nearby versions (nightly builds now, DuckDB v1.5.3 stable next). Delta + Unity Catalog + DuckDB: open storage, governance, fast analytical queries. Docker playground in the post. ⬇️ 🔗 Read more: https://lnkd.in/eCFajASh #DeltaLake #DuckDB #UnityCatalog #OpenSource #OpenLakehouse #Lakehouse

Headed to Data + AI Summit? Don't miss New Foundations of Delta Lake with Kernel and Spark's Data Source V2 with Rahul Potharaju and Tathagata Das. Since 2018, Delta has built the intelligence to deliver a high-performance, intuitive experience and command support on Spark DSv1 — work that set the standard for the last 8 years. Spark’s DSv2 APIs now take on more of that heavy lifting in the engine. This session explores how Delta is being updated to use DSv2 to build new foundations for the next decade of Delta. 👇 🗓️ June 15-18 📍 San Francisco 🔗 Session details: https://lnkd.in/e2dPZavQ #DeltaLake #ApacheSpark #DataAISummit #OpenSource #DataEngineering