SlideShare a Scribd company logo
Dustin Vannoy
Data Engineer
Cloud + Streaming
Azure Databricks with
Delta Lake
Dustin Vannoy
Data Engineering Consultant
Co-founder Data Engineering San Diego
/in/dustinvannoy
@dustinvannoy
dustin@dustinvannoy.com
Technologies
• Azure & AWS
• Spark
• Kafka
• Python
Modern Data Systems
• Data Lakes
• Analytics in Cloud
• Streaming
© Microsoft Azure + AI Conference All rights reserved.
Agenda
 Intro to Spark + Azure Databricks
 Delta Lake Overview
 Delta Lake in Action
 Schema Enforcement
 Time Travel
 MERGE, DELETE, OPTIMIZE
© Microsoft Azure + AI Conference All rights reserved.
Intro to Spark & Azure Databricks
Overview and Databricks workspace walk through
Why Spark?
Big data and the cloud
changed our mindset.
We want tools that
scale easily as data
size grows.
Spark is a leader in
data processing that
scales across many
machines. It can run
on Hadoop but is
faster and easier than
Map Reduce.
Benefit of horizontal scaling
Traditional Distributed (Parallel)
© Microsoft Azure + AI Conference All rights reserved.
What is Spark?
 Fast, general purpose engine for large-scale data processing
 Replaces MapReduce as Hadoop parallel programming API
 Many options:
 Yarn / Spark Cluster / Local
 Scala / Python / Java / R
 Spark Core / SQL / Streaming / ML / Graph
© Microsoft Azure + AI Conference All rights reserved.
Simple code, parallel compute
Spark consists of a programming API and execution engine
Worker Worker Worker Worker
Master
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.getOrCreate()
song_df = spark.read 
.option('sep','t') 
.option("inferSchema","true") 
.csv("/databricks-datasets/songs/data-001/part-0000*")
tempo_df = song_df.select(
col('_c4').alias('artist_name'),
col('_c14').alias('tempo'),
)
avg_tempo_df = tempo_df 
.groupBy('artist_name') 
.avg('tempo') 
.orderBy('avg(tempo)',ascending=False)
avg_tempo_df.show(truncate=False)
© Microsoft Azure + AI Conference All rights reserved.
Spark’s Strengths
 Data pipelines and analytics
 Batch or streaming
 SparkSQL
 Machine learning
 Uses memory to speed up processing
 Large community, many examples and tutorials
Demo
Databricks
Workspace
© Microsoft Azure + AI Conference All rights reserved.
Delta Lake Overview
Why use it and how to start
© Microsoft Azure + AI Conference All rights reserved.
Spark is powerful, but...
 Not ACID compliant – too easy to get corrupted data
 Schema mismatches – no validation on write
 Small files written, not efficient for reading
 Reads too much data (no indexes, only partitions)
© Microsoft Azure + AI Conference All rights reserved.
ACID
 Atomicity – all or nothing
 Consistency – data always in valid state
 Isolation – uncommitted operations don’t impact other reads/writes
 Durability – committed data is never lost
ACID compliance would give us ability to update and delete!
© Microsoft Azure + AI Conference All rights reserved.
Small File Problem
 Too much metadata
 Too many file open/close operations
 Compression not as effective
 Bad if using Map Reduce to read
We fix this with scheduled file compaction jobs, difficulty is avoiding
interference with new write operations
© Microsoft Azure + AI Conference All rights reserved.
Partitions
 Typically Spark reads all data in a table/directory before applying
filters
 Folder partitioning used to allow some filter push downs
 Limited to one fixed partition scheme to allow skipping reads
 Must use low cardinality columns for partitioning
We used to just add indexes and run statistics to improve seeks
Delta Lake Concepts
Reference: delta.io
© Microsoft Azure + AI Conference All rights reserved.
ACID Transactions
Atomicity, Consistency, and Isolation all improved
© Microsoft Azure + AI Conference All rights reserved.
Reminder: ACID
 Atomicity – all or nothing
 Consistency – data always in valid state
 Isolation – uncommitted operations don’t impact other reads/writes
 Durability – committed data is never lost
© Microsoft Azure + AI Conference All rights reserved.
ACID Transaction Support
“Serializable isolation levels
ensure that readers never
see inconsistent data”
- Delta Lake Documentation
© Microsoft Azure + AI Conference All rights reserved.
Schema Enforcement
How to use schema validation and schema merge
© Microsoft Azure + AI Conference All rights reserved.
Schema validation by default
 Delta defaults to validating schema
 Fails on mismatch
 Or, set schema merge option
© Microsoft Azure + AI Conference All rights reserved.
Time Travel
Data version history in Delta
© Microsoft Azure + AI Conference All rights reserved.
Delta Log
“The transaction log is the mechanism through which
Delta Lake is able to offer the guarantee of atomicity.”
Reference: Databricks Blog: Unpacking the Transaction Log
Demo
Delta
capabilities
© Microsoft Azure + AI Conference All rights reserved.
Final thoughts
Delta Lake delivers some powerful capabilities
© Microsoft Azure + AI Conference All rights reserved.
Delta Lake addresses
 ACID compliance
 Schema enforcement
 Compacting files
 Performance optimizations
© Microsoft Azure + AI Conference All rights reserved.
References
 Video - Simplify and Scale Data Engineering Pipelines with Delta Lake
- Amanda Moran
 Video - Building Data Intensive Application on Top of Delta Lakes
 Video - Why do we need Delta Lake for Spark? - Learning Journal
 Databricks Blog: Unpacking the Transaction Log
 Databricks Delta Lake - James Serra
 Databricks Delta Technical Guide - Jan 2019
 Productionizing Machine Learning with Delta Lake
© Microsoft Azure + AI Conference All rights reserved.
Please use EventsXD to fill out a session evaluation.
Thank you!

More Related Content

What's hot (20)

PDF
Getting Started with Delta Lake on Databricks
Knoldus Inc.
 
PDF
Hyperspace for Delta Lake
Databricks
 
PDF
Modernizing to a Cloud Data Architecture
Databricks
 
PDF
Spark with Delta Lake
Knoldus Inc.
 
PDF
Let’s get to know Snowflake
Knoldus Inc.
 
PDF
Azure Data Factory v2
inovex GmbH
 
PDF
Large Scale Lakehouse Implementation Using Structured Streaming
Databricks
 
PPT
An overview of snowflake
Sivakumar Ramar
 
PDF
Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...
Databricks
 
PPTX
Apache Spark overview
DataArt
 
PPTX
Azure Data Factory Data Flow
Mark Kromer
 
PPTX
Snowflake essentials
qureshihamid
 
PPTX
Databricks Fundamentals
Dalibor Wijas
 
PDF
Achieving Lakehouse Models with Spark 3.0
Databricks
 
PDF
Data Migration to Azure
Sanjay B. Bhakta
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PPTX
Microsoft Azure Data Factory Hands-On Lab Overview Slides
Mark Kromer
 
PDF
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
Spark Summit
 
PPTX
Introducing the Snowflake Computing Cloud Data Warehouse
Snowflake Computing
 
PDF
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
HostedbyConfluent
 
Getting Started with Delta Lake on Databricks
Knoldus Inc.
 
Hyperspace for Delta Lake
Databricks
 
Modernizing to a Cloud Data Architecture
Databricks
 
Spark with Delta Lake
Knoldus Inc.
 
Let’s get to know Snowflake
Knoldus Inc.
 
Azure Data Factory v2
inovex GmbH
 
Large Scale Lakehouse Implementation Using Structured Streaming
Databricks
 
An overview of snowflake
Sivakumar Ramar
 
Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...
Databricks
 
Apache Spark overview
DataArt
 
Azure Data Factory Data Flow
Mark Kromer
 
Snowflake essentials
qureshihamid
 
Databricks Fundamentals
Dalibor Wijas
 
Achieving Lakehouse Models with Spark 3.0
Databricks
 
Data Migration to Azure
Sanjay B. Bhakta
 
Understanding Query Plans and Spark UIs
Databricks
 
Microsoft Azure Data Factory Hands-On Lab Overview Slides
Mark Kromer
 
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
Spark Summit
 
Introducing the Snowflake Computing Cloud Data Warehouse
Snowflake Computing
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
HostedbyConfluent
 

Similar to Delta Lake with Azure Databricks (20)

PDF
Intro to databricks delta lake
Mykola Zerniuk
 
PDF
Making Apache Spark Better with Delta Lake
Databricks
 
PDF
What Is Delta Lake ???
✪Computants✪IBM_BP
 
PDF
Delta Lake: Open Source Reliability w/ Apache Spark
George Chow
 
PDF
Intro to Delta Lake
Databricks
 
PDF
Simplify and Scale Data Engineering Pipelines with Delta Lake
Databricks
 
PDF
Open Source Reliability for Data Lake with Apache Spark by Michael Armbrust
Data Con LA
 
PDF
Data Lakes with Azure Databricks
Data Con LA
 
PDF
Best Practices for Building Robust Data Platform with Apache Spark and Delta
Databricks
 
PDF
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
PDF
The delta architecture
Prakash Chockalingam
 
PDF
Delta Architecture
Paulo Gutierrez
 
PDF
DELTA LAKE IN FINTECH: ENHANCING DATA LAKE RELIABILITY WITH ACID TRANSACTIONS
International Research Journal of Modernization in Engineering Technology and Science
 
PDF
Building Reliable Data Lakes at Scale with Delta Lake
Databricks
 
PDF
Building Data Intensive Analytic Application on Top of Delta Lakes
Databricks
 
PPTX
4Developers 2018: Przetwarzanie Big Data w oparciu o architekturę Lambda na p...
PROIDEA
 
PDF
Lakehouse in Azure
Sergio Zenatti Filho
 
PDF
So You Want to Build a Data Lake?
David P. Moore
 
PPTX
databricks course | databricks online training
Accentfuture
 
PDF
Building Data Quality pipelines with Apache Spark and Delta Lake
Databricks
 
Intro to databricks delta lake
Mykola Zerniuk
 
Making Apache Spark Better with Delta Lake
Databricks
 
What Is Delta Lake ???
✪Computants✪IBM_BP
 
Delta Lake: Open Source Reliability w/ Apache Spark
George Chow
 
Intro to Delta Lake
Databricks
 
Simplify and Scale Data Engineering Pipelines with Delta Lake
Databricks
 
Open Source Reliability for Data Lake with Apache Spark by Michael Armbrust
Data Con LA
 
Data Lakes with Azure Databricks
Data Con LA
 
Best Practices for Building Robust Data Platform with Apache Spark and Delta
Databricks
 
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
The delta architecture
Prakash Chockalingam
 
Delta Architecture
Paulo Gutierrez
 
DELTA LAKE IN FINTECH: ENHANCING DATA LAKE RELIABILITY WITH ACID TRANSACTIONS
International Research Journal of Modernization in Engineering Technology and Science
 
Building Reliable Data Lakes at Scale with Delta Lake
Databricks
 
Building Data Intensive Analytic Application on Top of Delta Lakes
Databricks
 
4Developers 2018: Przetwarzanie Big Data w oparciu o architekturę Lambda na p...
PROIDEA
 
Lakehouse in Azure
Sergio Zenatti Filho
 
So You Want to Build a Data Lake?
David P. Moore
 
databricks course | databricks online training
Accentfuture
 
Building Data Quality pipelines with Apache Spark and Delta Lake
Databricks
 
Ad

More from Dustin Vannoy (6)

PDF
PASS 2024 - Best Practices for Development on Azure Databricks
Dustin Vannoy
 
PDF
Azure Data Platform Overview.pdf
Dustin Vannoy
 
PDF
DustinVannoy_DataPipelines_AzureDataConf_Dec22.pdf
Dustin Vannoy
 
PPTX
Getting Started with Spark Structured Streaming - Current 22
Dustin Vannoy
 
PPTX
Spark Streaming with Azure Databricks
Dustin Vannoy
 
PPTX
PASS_Summit_2019_Azure_Storage_Options_for_Analytics
Dustin Vannoy
 
PASS 2024 - Best Practices for Development on Azure Databricks
Dustin Vannoy
 
Azure Data Platform Overview.pdf
Dustin Vannoy
 
DustinVannoy_DataPipelines_AzureDataConf_Dec22.pdf
Dustin Vannoy
 
Getting Started with Spark Structured Streaming - Current 22
Dustin Vannoy
 
Spark Streaming with Azure Databricks
Dustin Vannoy
 
PASS_Summit_2019_Azure_Storage_Options_for_Analytics
Dustin Vannoy
 
Ad

Recently uploaded (20)

PPTX
Pipeline Automatic Leak Detection for Water Distribution Systems
Sione Palu
 
PPTX
Insurance-Analytics-Branch-Dashboard (1).pptx
trivenisapate02
 
PDF
Top Civil Engineer Canada Services111111
nengineeringfirms
 
PDF
apidays Munich 2025 - Integrate Your APIs into the New AI Marketplace, Senthi...
apidays
 
PDF
D9110.pdfdsfvsdfvsdfvsdfvfvfsvfsvffsdfvsdfvsd
minhn6673
 
PPTX
Nursing Shift Supervisor 24/7 in a week .pptx
amjadtanveer
 
PPTX
Customer Segmentation: Seeing the Trees and the Forest Simultaneously
Sione Palu
 
PDF
Blue Futuristic Cyber Security Presentation.pdf
tanvikhunt1003
 
PDF
An Uncut Conversation With Grok | PDF Document
Mike Hydes
 
PPTX
Data-Users-in-Database-Management-Systems (1).pptx
dharmik832021
 
PPTX
short term project on AI Driven Data Analytics
JMJCollegeComputerde
 
PDF
apidays Munich 2025 - The Physics of Requirement Sciences Through Application...
apidays
 
PPTX
short term internship project on Data visualization
JMJCollegeComputerde
 
PDF
blockchain123456789012345678901234567890
tanvikhunt1003
 
PPTX
The whitetiger novel review for collegeassignment.pptx
DhruvPatel754154
 
PDF
202501214233242351219 QASS Session 2.pdf
lauramejiamillan
 
PPTX
White Blue Simple Modern Enhancing Sales Strategy Presentation_20250724_21093...
RamNeymarjr
 
PPTX
HSE WEEKLY REPORT for dummies and lazzzzy.pptx
ahmedibrahim691723
 
PDF
717629748-Databricks-Certified-Data-Engineer-Professional-Dumps-by-Ball-21-03...
pedelli41
 
PPTX
Fluvial_Civilizations_Presentation (1).pptx
alisslovemendoza7
 
Pipeline Automatic Leak Detection for Water Distribution Systems
Sione Palu
 
Insurance-Analytics-Branch-Dashboard (1).pptx
trivenisapate02
 
Top Civil Engineer Canada Services111111
nengineeringfirms
 
apidays Munich 2025 - Integrate Your APIs into the New AI Marketplace, Senthi...
apidays
 
D9110.pdfdsfvsdfvsdfvsdfvfvfsvfsvffsdfvsdfvsd
minhn6673
 
Nursing Shift Supervisor 24/7 in a week .pptx
amjadtanveer
 
Customer Segmentation: Seeing the Trees and the Forest Simultaneously
Sione Palu
 
Blue Futuristic Cyber Security Presentation.pdf
tanvikhunt1003
 
An Uncut Conversation With Grok | PDF Document
Mike Hydes
 
Data-Users-in-Database-Management-Systems (1).pptx
dharmik832021
 
short term project on AI Driven Data Analytics
JMJCollegeComputerde
 
apidays Munich 2025 - The Physics of Requirement Sciences Through Application...
apidays
 
short term internship project on Data visualization
JMJCollegeComputerde
 
blockchain123456789012345678901234567890
tanvikhunt1003
 
The whitetiger novel review for collegeassignment.pptx
DhruvPatel754154
 
202501214233242351219 QASS Session 2.pdf
lauramejiamillan
 
White Blue Simple Modern Enhancing Sales Strategy Presentation_20250724_21093...
RamNeymarjr
 
HSE WEEKLY REPORT for dummies and lazzzzy.pptx
ahmedibrahim691723
 
717629748-Databricks-Certified-Data-Engineer-Professional-Dumps-by-Ball-21-03...
pedelli41
 
Fluvial_Civilizations_Presentation (1).pptx
alisslovemendoza7
 

Delta Lake with Azure Databricks

  • 1. Dustin Vannoy Data Engineer Cloud + Streaming Azure Databricks with Delta Lake
  • 2. Dustin Vannoy Data Engineering Consultant Co-founder Data Engineering San Diego /in/dustinvannoy @dustinvannoy [email protected] Technologies • Azure & AWS • Spark • Kafka • Python Modern Data Systems • Data Lakes • Analytics in Cloud • Streaming
  • 3. © Microsoft Azure + AI Conference All rights reserved. Agenda  Intro to Spark + Azure Databricks  Delta Lake Overview  Delta Lake in Action  Schema Enforcement  Time Travel  MERGE, DELETE, OPTIMIZE
  • 4. © Microsoft Azure + AI Conference All rights reserved. Intro to Spark & Azure Databricks Overview and Databricks workspace walk through
  • 5. Why Spark? Big data and the cloud changed our mindset. We want tools that scale easily as data size grows. Spark is a leader in data processing that scales across many machines. It can run on Hadoop but is faster and easier than Map Reduce.
  • 6. Benefit of horizontal scaling Traditional Distributed (Parallel)
  • 7. © Microsoft Azure + AI Conference All rights reserved. What is Spark?  Fast, general purpose engine for large-scale data processing  Replaces MapReduce as Hadoop parallel programming API  Many options:  Yarn / Spark Cluster / Local  Scala / Python / Java / R  Spark Core / SQL / Streaming / ML / Graph
  • 8. © Microsoft Azure + AI Conference All rights reserved. Simple code, parallel compute Spark consists of a programming API and execution engine Worker Worker Worker Worker Master from pyspark.sql import SparkSession from pyspark.sql.functions import col spark = SparkSession.builder.getOrCreate() song_df = spark.read .option('sep','t') .option("inferSchema","true") .csv("/databricks-datasets/songs/data-001/part-0000*") tempo_df = song_df.select( col('_c4').alias('artist_name'), col('_c14').alias('tempo'), ) avg_tempo_df = tempo_df .groupBy('artist_name') .avg('tempo') .orderBy('avg(tempo)',ascending=False) avg_tempo_df.show(truncate=False)
  • 9. © Microsoft Azure + AI Conference All rights reserved. Spark’s Strengths  Data pipelines and analytics  Batch or streaming  SparkSQL  Machine learning  Uses memory to speed up processing  Large community, many examples and tutorials
  • 11. © Microsoft Azure + AI Conference All rights reserved. Delta Lake Overview Why use it and how to start
  • 12. © Microsoft Azure + AI Conference All rights reserved. Spark is powerful, but...  Not ACID compliant – too easy to get corrupted data  Schema mismatches – no validation on write  Small files written, not efficient for reading  Reads too much data (no indexes, only partitions)
  • 13. © Microsoft Azure + AI Conference All rights reserved. ACID  Atomicity – all or nothing  Consistency – data always in valid state  Isolation – uncommitted operations don’t impact other reads/writes  Durability – committed data is never lost ACID compliance would give us ability to update and delete!
  • 14. © Microsoft Azure + AI Conference All rights reserved. Small File Problem  Too much metadata  Too many file open/close operations  Compression not as effective  Bad if using Map Reduce to read We fix this with scheduled file compaction jobs, difficulty is avoiding interference with new write operations
  • 15. © Microsoft Azure + AI Conference All rights reserved. Partitions  Typically Spark reads all data in a table/directory before applying filters  Folder partitioning used to allow some filter push downs  Limited to one fixed partition scheme to allow skipping reads  Must use low cardinality columns for partitioning We used to just add indexes and run statistics to improve seeks
  • 17. © Microsoft Azure + AI Conference All rights reserved. ACID Transactions Atomicity, Consistency, and Isolation all improved
  • 18. © Microsoft Azure + AI Conference All rights reserved. Reminder: ACID  Atomicity – all or nothing  Consistency – data always in valid state  Isolation – uncommitted operations don’t impact other reads/writes  Durability – committed data is never lost
  • 19. © Microsoft Azure + AI Conference All rights reserved. ACID Transaction Support “Serializable isolation levels ensure that readers never see inconsistent data” - Delta Lake Documentation
  • 20. © Microsoft Azure + AI Conference All rights reserved. Schema Enforcement How to use schema validation and schema merge
  • 21. © Microsoft Azure + AI Conference All rights reserved. Schema validation by default  Delta defaults to validating schema  Fails on mismatch  Or, set schema merge option
  • 22. © Microsoft Azure + AI Conference All rights reserved. Time Travel Data version history in Delta
  • 23. © Microsoft Azure + AI Conference All rights reserved. Delta Log “The transaction log is the mechanism through which Delta Lake is able to offer the guarantee of atomicity.” Reference: Databricks Blog: Unpacking the Transaction Log
  • 25. © Microsoft Azure + AI Conference All rights reserved. Final thoughts Delta Lake delivers some powerful capabilities
  • 26. © Microsoft Azure + AI Conference All rights reserved. Delta Lake addresses  ACID compliance  Schema enforcement  Compacting files  Performance optimizations
  • 27. © Microsoft Azure + AI Conference All rights reserved. References  Video - Simplify and Scale Data Engineering Pipelines with Delta Lake - Amanda Moran  Video - Building Data Intensive Application on Top of Delta Lakes  Video - Why do we need Delta Lake for Spark? - Learning Journal  Databricks Blog: Unpacking the Transaction Log  Databricks Delta Lake - James Serra  Databricks Delta Technical Guide - Jan 2019  Productionizing Machine Learning with Delta Lake
  • 28. © Microsoft Azure + AI Conference All rights reserved. Please use EventsXD to fill out a session evaluation. Thank you!

Editor's Notes

  • #2: With the shift to data lakes that use distributed file storage as the foundation, we have been missing the reliability that relational databases provides. Databricks Delta is a data management system focused on bringing more reliability and performance into our data lakes. It sits on top of existing storage and the API is very similar to reading and writing to files from Spark already. This session will present the overview of Delta Lake, why it may be a better option than standard data lake storage, and how you can use it from Azure Databricks. We will work through demos that showcase the key benefits of delta lake: 1. ACID transactions 2. Schema enforcement and evolution 3. Time travel (data versioning)
  • #7: Let’s think about the benefit of parallel processing, often referred to as distributed systems. The idea is actually very easy to understand. If we had a task such as counting all the people at a concert, you could have one person who is really good at counting do it and if the venue is small enough they will do just fine. But the job will be completed faster if you have many people counting and combining the results at the end. Sure there is a little more organization needed, but if you need to count the attendees at a Beyonce concert you could just hire a lot of people to do the job. And if one of them gets distracted by the music, you can send whoever finishes first in to take over counting that section. We call this capability “Horizontal Scaling” because if our data processing system is not powerful enough to do the work, we add more computers to help out rather than replacing the single server with a more powerful server. Distributed computing and parallel processing are not new concepts, few things in computing are, but what if you had an easy way to tell all the workers what to do without having to micro-manage to avoid two people counting the same section? That is where new programming models and frameworks have stepped in over the last 10 years and gave us the beloved buzz word ”Big Data”. Spark is not the only option here, but it has a lot of strengths and is often chosen over the traditional single machine processing options.
  • #8: A fast and general engine for large-scale data processing, uses memory to provide benefit Often replaces MapReduce as parallel programming api on Hadoop, the way it handles data (RDDs) provides one performance benefit and use of memory when possible provides another large performance benefit Can run on Hadoop (using Yarn) but also as a separate Spark cluster. Local is possible as well but reduces the performance benefits…I find its still a useful API though Run Java, Scala, Python, or R. If you don’t already know one of those languages really well, I recommend trying it in Python and Scala and pick whichever is easiest for you. Several modules for different use cases, similar api so you can swap between modes relatively easily. For example, we have both streaming and batch sources of some data and we reuse the rest of the spark processing transformations.
  • #9: In the day to day we will talk about writing Spark code and also refer to running the code on the Spark cluster. There are actually quite a few options for how to do either of these things, but a quick look at Spark code that uses Spark DataFrames in Python. And then whatever cluster we run it on will have a concept of a master node and worker nodes, as well as some storage that is often a hybrid of local storage on the workers plus a distributed file system like Hadoop’s HDFS, Amazon S3, or Azure Data Lake Storage. If you don’t follow all those terms, it’s ok. There is plenty of time to build up to those concepts after you start learning to write spark code and run it in a simple Spark environment. We will cover that in other videos.
  • #10: So we sort of get what Spark is, we saw a small code sample and discussed how a cluster exists to run the code on. Let’s go back to a higher level and talk about Spark’s strengths.
  • #11: Quick overview of important databricks workspace segments – Clusters, Tables, Notebooks Open create_parquet_tables notebook and run first few commands as examples of working without delta
  • #14: Atomicity – typical Spark save does not use locking and is not atomic so it could leave incomplete changes behind and corrupt data. Overwrite will remove data before loading new data, so typically not an issue. With append mode the default commiter should have atomicity but some of the faster commiters don’t gurantee atomicity. - Learning Journal, Delta Lake for Apache Spark video on YouTube Consistency – with typical Spark overwrite there is a time where no files exist and if failure happens at that point you are left in invalid state. Isolation – an operation that is in progress (not commited) should not impact the results of other reads or writes...do not want dirty reads. Typical database offers different levels of isolation but Spark doesn’t have specific option of commit such as read/commited and serializable. Task level and job level commits exist but lack of atomicity in write leaves this not fully working. Durability – typically not an issue, though lack of commit can lead to issues here as well
  • #15: Atomicity – typical Spark save does not use locking and is not atomic so it could leave incomplete changes behind and corrupt data. Overwrite will remove data before loading new data, so typically not an issue. With append mode the default commiter should have atomicity but some of the faster commiters don’t gurantee atomicity. - Learning Journal, Delta Lake for Apache Spark video on YouTube Consistency – with typical Spark overwrite there is a time where no files exist and if failure happens at that point you are left in invalid state. Isolation – an operation that is in progress (not commited) should not impact the results of other reads or writes...do not want dirty reads. Typical database offers different levels of isolation but Spark doesn’t have specific option of commit such as read/commited and serializable. Task level and job level commits exist but lack of atomicity in write leaves this not fully working. Durability – typically not an issue, though lack of commit can lead to issues here as well
  • #19: Atomicity – typical Spark save does not use locking and is not atomic so it could leave incomplete changes behind and corrupt data. Overwrite will remove data before loading new data, so typically not an issue. With append mode the default commiter should have atomicity but some of the faster commiters don’t gurantee atomicity. - Learning Journal, Delta Lake for Apache Spark video on YouTube Consistency – with typical Spark overwrite there is a time where no files exist and if failure happens at that point you are left in invalid state. Isolation – an operation that is in progress (not commited) should not impact the results of other reads or writes...do not want dirty reads. Typical database offers different levels of isolation but Spark doesn’t have specific option of commit such as read/commited and serializable. Task level and job level commits exist but lack of atomicity in write leaves this not fully working. Durability – typically not an issue, though lack of commit can lead to issues here as well
  • #24: Quote and image from Databricks blog post by Burak Yavuz, Michael Armbrust and Brenner Heintz -> https://databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html
  • #25: Demo notebook create_delta_tables Show bad data when running one set of writes from one source, then run from second source Same example with delta destination to show failure Same example but tweaked to allow schema merge Show transaction log files Demo of file where data was streamed in, show by timestamp and version