Accelerating Data Ingestion with Databricks Autoloader
2 likes2,213 views
The document discusses Databricks Autoloader, an optimized cloud file source for Apache Spark that enables efficient incremental data ingestion from cloud storage. It highlights the components, implementation methods, and schema evolution features, emphasizing the advantages of using Autoloader over existing ingestion patterns. Additionally, it provides insights into practical implementations, lessons learned, and best practices for managing event grids and streaming operations.
![Schema Metastore
_schemas
{“ID”:1,
“ProductName”:“Belt”}
{
"type": "struct",
"fields": [
{
"name": "ID",
"type": “string",
"nullable": true,
"metadata": {}
},
{
"name": "ProductName",
"type": “string",
"nullable": true,
"metadata": {}
}
]
}
0
On First Read](https://image.slidesharecdn.com/1060simonwhiteley-210614173625/85/Accelerating-Data-Ingestion-with-Databricks-Autoloader-30-320.jpg)
![Schema Metastore – DataType Inference
_schemas
{“ID”:1,
“ProductName”:“Belt”}
{
"type": "struct",
"fields": [
{
"name": "ID",
"type": “int",
"nullable": true,
"metadata": {}
},
{
"name": "ProductName",
"type": “string",
"nullable": true,
"metadata": {}
}
]
}
0
On First Read
.option(“cloudFiles.inferColumnTypes”,”True”)](https://image.slidesharecdn.com/1060simonwhiteley-210614173625/85/Accelerating-Data-Ingestion-with-Databricks-Autoloader-31-320.jpg)
![Schema Metastore – Schema Hints
_schemas
{“ID”:1,
“ProductName”:“Belt”}
{
"type": "struct",
"fields": [
{
"name": "ID",
"type": “long",
"nullable": true,
"metadata": {}
},
{
"name": "ProductName",
"type": “string",
"nullable": true,
"metadata": {}
}
]
}
0
On First Read
.option(“cloudFiles.schemaHints”,”ID long”)](https://image.slidesharecdn.com/1060simonwhiteley-210614173625/85/Accelerating-Data-Ingestion-with-Databricks-Autoloader-32-320.jpg)
Accelerating Data Ingestion with Databricks Autoloader
- 1. Accelerating Data Ingestion with Databricks Autoloader Simon Whiteley Director of Engineering, Advancing Analytics
- 2. Agenda ▪ Why Incremental is Hard ▪ Autoloader Components ▪ Implementation ▪ Evolution ▪ Lessons
- 3. Why Incremental is Hard
- 5. Incremental Ingestion ▪ Only Read New Files ▪ Don’t Miss Files ▪ Trigger Immediately ▪ Repeatable Pattern ▪ Fast over large directories ?
- 6. Existing Patterns – 1) ETL Metadata etl batch read {“lastRead”:”2021/05/26”} Contents: • /2021/05/24/file 1 • /2021/05/25/file 2 • /2021/05/26/file 3 • /2021/05/27/file 4 .load(f“/{loadDate}/”
- 7. Existing Patterns – 2) Spark File Streaming file stream read Contents: • File 1 • File 2 • File 3 • File 4 Checkpoint: • File 1 • File 2 • File 3
- 8. Existing Patterns – 3) DIY triggered batch read Blob File Trigger Logic App Azure Function Databricks Job API
- 9. Incremental Ingestion Approaches Approach Good At Bad At Metadata ETL Repeatable Not immediate, requires polling File Streaming Repeatable Immediate Slows down over large directories DIY Architecture Immediate Triggering Not Repeatable
- 11. Prakash Chockalingam Databricks Engineering Blog Auto Loader is an optimized cloud file source for Apache Spark that loads data continuously and efficiently from cloud storage as new data arrives.
- 12. What is Autoloader? Essentially, Autoloader combines our three approaches of: • Storing Metadata about what has been read • Using Structured Streaming for immediate processing • Utilising Cloud-Native Components to optimise identifying arriving files There are two parts to the Autoloader job: • CloudFiles DataReader • CloudNotification Services (optional)
- 13. Cloudfiles Reader Blob Storage Blob Storage Queue {“fileAdded”:”/landing/file 4.json” • File 1.json • File 2.json • File 3.json • File 4.json Dataframe Check Files in Queue Read specific files from source
- 14. CloudFiles DataReader df = ( spark .readStream .format(“cloudfiles”) .option(“cloudfiles.format”,”json”) .option(“cloudfiles.useNotifications”,”true”) .schema(mySchema) .load(“/mnt/landing/”) ) Tells Spark to use Autoloader Tells Autoloader to expect JSON files Should Autoloader use the Notification Queue
- 15. Cloud Notification Services - Azure Blob Storage Event Grid Topic Event Grid Subscription Blob Storage Queue Event Grid Subscription Blob Storage Queue Event Grid Subscription Blob Storage Queue
- 16. Cloud Notification Services - Azure Blob Storage New File Arrives, Triggers Event Topic Subscription checks message filters, inserts into queue {fileAdded:“/file 4/”}
- 17. NotificationServices Config cloudFiles .useNotifications – Directory Listing VS Notification Queue .queueName – Use an Existing Queue .connectionString – Queue Storage Connection .subscriptionId .resourceGroup .tenantId .clientId .clientSecret Service Principal for Queue Creation
- 18. Implementing Autoloader ▪ Setup Steps ▪ Reading New Files ▪ A Basic ETL Setup
- 21. Low Frequency Streams Autoloader One File Per Day 24/7 Cluster
- 22. Low Frequency Streams Autoloader One File Per Day 1/7 Cluster df .writeStream .trigger(once=True) .save(path) Autoloader can be combined with trigger.Once – each run finds only files not processed since last run
- 24. Delta Merge Autoloader df .writeStream .foreachBatch(runThis) .save(path) def runThis(df, batchId): (df .write .save(path) )
- 25. Delta Implementation ▪ Batch ETL Pattern ▪ Merge Statements ▪ Logging State
- 26. Evolving Schemas New Features since Databricks Runtime 8.2
- 27. What is Schema Evolution? {“ID”:1,“ProductName”:“Belt”} {“ID”:2,“ProductName”:“T-Shirt”,”Size”:”XL”} {“ID”:3,“ProductName”:“Shirt”,“Size”:“14”, “Care”:{ “DryClean”: “Yes”, “MachineWash”:“Don’t you dare” } }
- 28. How do we handle Evolution? 1. Fail the Stream 2. Manually Intervene 3. Automatically Evolve In order to manage schema evolution, we need to know: • What the schema is expected to be • What the schema is now • How we want to handle any changes in schema
- 29. Schema Inference In Databricks 8.2 Onwards – simply don’t provide a Schema to enable Schema Inference. This infers the schema once when the stream is started and stores it as metadata. cloudfiles .schemaLocation – where to store the schema .inferColumnTypes – sample data to infer types .schemaHints – manually specify data types for certain columns
- 30. Schema Metastore _schemas {“ID”:1, “ProductName”:“Belt”} { "type": "struct", "fields": [ { "name": "ID", "type": “string", "nullable": true, "metadata": {} }, { "name": "ProductName", "type": “string", "nullable": true, "metadata": {} } ] } 0 On First Read
- 31. Schema Metastore – DataType Inference _schemas {“ID”:1, “ProductName”:“Belt”} { "type": "struct", "fields": [ { "name": "ID", "type": “int", "nullable": true, "metadata": {} }, { "name": "ProductName", "type": “string", "nullable": true, "metadata": {} } ] } 0 On First Read .option(“cloudFiles.inferColumnTypes”,”True”)
- 32. Schema Metastore – Schema Hints _schemas {“ID”:1, “ProductName”:“Belt”} { "type": "struct", "fields": [ { "name": "ID", "type": “long", "nullable": true, "metadata": {} }, { "name": "ProductName", "type": “string", "nullable": true, "metadata": {} } ] } 0 On First Read .option(“cloudFiles.schemaHints”,”ID long”)
- 33. Schema Evolution cloudFiles.schemaEvolutionMode • addNewColumns – Fail the job, update the schema metastore • failOnNewColumns – Fail the job, no updates made • rescue – Do not fail, pull all unexpected data into _rescued_data To allow for schema evolution, we can include a schema evolution mode option:
- 34. Evolution Reminder {“ID”:1,“ProductName”:“Belt”} {“ID”:2,“ProductName”:“T-Shirt”,”Size”:”XL”} {“ID”:3,“ProductName”:“Shirt”,“Size”:“14”, “Care”:{ “DryClean”: “Yes”, “MachineWash”:“Don’t you dare” } } 1 2 3
- 35. Schema Evolution - Rescue 1 2 3 ID Product Name _rescued_data 1 Belt ID Product Name _rescued_data 2 T-Shirt {“Size”:”XL”} ID Product Name _rescued_data 3 Shirt {“Size”:”14”,”Care”:{“DryC…
- 36. Schema Evolution – Add New Columns _schemas {“ID”:2, “ProductName”:“T-Shirt”, “Size”:”XL”} 0 On Arrival 2 1 { "type": "struct", "fields": [ { "name": "ID", "type": “string", }, { "name": "ProductName", "type": “string", } , { "name": “Size", "type": “string", }…
- 37. Schema Evolution ▪ Inference & The Schema Metastore ▪ Schema Hints ▪ Schema Evolution
- 38. Lessons from an Autoloader Life
- 39. Autoloader Lessons ▪ EventGrid Quotas & Settings ▪ Streaming Best Practices ▪ Batching Best Practices
- 41. EventGrid Quota Lessons • You can have 500 files from a single storage account using the system topic • Deleting checkpoint will reset the stream ID and create a new Subscription/Queue, leaving an orphan set • Use the CloudNotification Libraries to manage this more closely with custom topics
- 42. Streaming Optimisation • MaxBytesPerTrigger / MaxFilesPerTrigger Manages the size of the streaming microbatch • FetchParallelism Manages the workload on your queue
- 43. Batch Lessons – Look for Lost Messages Default 7 days!
- 44. Databricks Autoloader ▪ Reduces complexity of ingesting files ▪ Has some quirks in implementing ETL processes ▪ Growing number of schema evolution features
- 46. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.