Build Real-Time Applications with Databricks Streaming
1 like1,218 views
This document discusses using Databricks, Spark, and Power BI for real-time data streaming. It describes a use case of a fire department needing real-time reporting of equipment locations, personnel statuses, and active incidents. The solution involves ingesting event data using Azure Event Hubs, processing the stream using Databricks and Spark Structured Streaming, storing the results in Delta Lake, and visualizing the data in Power BI dashboards. It then demonstrates the architecture by walking through creating Delta tables, streaming from Event Hubs to Delta Lake, and running a sample event simulator.
Build Real-Time Applications with Databricks Streaming
- 1. Real-Time Data Streaming with Databricks, Spark and Power BI Bennie Haelen Principal Architect ā Insight Digital Innovation
- 2. Use Case Description ā¢ Large Metropolitan Fire Department ā¢ Implemented a MDW architecture on Azure ā¢ Based upon the Insight repeatable MDW framework architecture Legend RAW Ins-swdi-lens-aas Azure Automation Ins-swdi-lens-lapp PL_MT_raw2stage PL_processAAS Dataflow Workflow PL_DATA_ORA_2_ADLS_FULL DROPZONE CSV file 1 2 4 7 8 9 Power BI 5 PL_MT_stage2mdw PL_DATA_mdw2asql 6 Ins-swdi-lens-asql 3 Ins-swdi-lens-adf RAW/Archive STAGE MDW Oracle .parquet Workspace Folders Storage Acct ins-swdi-lens-adls Databricks Hive Databases Key Vaults Ins-swdi-lens-email-lapp
- 3. Use Case Extension ā¢ Need to add a real-time reporting channel ā¢ Up-to-date location & status of equipment ā¢ Location & status of firefighters, EMT personnel ā¢ List of active incidents within the city ā¢ Near real-time Visualization ā¢ Automatically updating dashboard ā¢ Map with automatic updates of locations and incidents ā¢ Used by fire chiefs to make real-time move-up decisions ā¢ Pre-emptively Move-up equipment & resources
- 4. Use Case Analysis ā¢ Forwarding of events through the Azure Cloud ā¢ ESB exposes a Web Sockets interface ā¢ Azure function reads events from ESB through WebSockets interface ā¢ Function forwards the events to the Azure cloud ā¢ Function is hosted in a Web Application Central FD Database Ingest data from the various event sources Change Data Capture Triggered with each transactional operation Enterprise Service Bus CDC Ingest & forward events to consumers Solution ā¢ Create Cloud ingest ā¢ Real Time Stream processing ā¢ Performant ACID Data Store ā¢ Real-Time Visualization `
- 5. Architectural Requirements ā¢ Ingest Event Stream ā¢ High ingestion rate (1000+ events per second) ā¢ Need high-performance, fault tolerant service ā¢ Stream Events, perform domain-specific conversions ā¢ Need real-time streaming analytics ā¢ Stored Processed Data in high-performant data store ā¢ Keyed access to the data ā¢ Ability to perform UPSERT operations ā¢ Visualize the data in a real-time dashboard ā¢ Updates triggered by data changes in the underlying data store
- 6. Solution Architecture Ingestion Channel Azure Event Hubs Event Processing Databricks with Spark Structured Streaming Real-Time Data Store Databricks Delta Lake Visualization Power BI Service Dashboard Ingest Event Stream ā¢ High ingestion rate (1000+ events per second) ā¢ Need high-performance, fault tolerant service Azure Event Hubs ā¢ Microsoft real-time data ingestion engine ā¢ Can ingest millions of events/second ā¢ Kafka compatibility Process Stream ā¢ Continuous Processing ā¢ Real time ingestion ā¢ Micro-batch processing Databricks on Azure ā¢ Spark Structured Streaming ā¢ Fault-tolerant Stream processing engine ā¢ Kafka compatibility Real-Time Storage ā¢ Keyed Access to Data ā¢ Ability to perform UPSERTS ā¢ Simple SQL-based access Delta Lake ā¢ ACID Transactions ā¢ High Scalability Real-Time Visualization ā¢ Simple Integration ā¢ Updates through Data Triggers ā¢ Direct Query into Data Source Microsoft Power BI ā¢ Direct Query against Delta Lake ā¢ Real-time dashboarding facilities ā¢ Updates trigger through data changes or push datasets
- 7. Demo Architecture ā¢ nb-create-unitStatusTable notebook Invokes the generic CreateDeltaTable with the appropriate parameters to create our UnitStatus table ā¢ nb-create-delta-table notebook Generic notebook which creates a Delta table ā¢ nb-eventhub-spark-streaming notebook reads the events from Event Hubs and invokes the foreachBatch sink function implemented in nb- unitstatus-event-processor notebook ā¢ nb-unitstatus-event-processor Processes the events, performs the transformations, and finally updates our UnitStatusTable Units-eh Event Hub C# .NET Console Application nb-eventhub-spark- streaming Databricks Notebook nb-unitstatus- event-processor Delta Table old_stream_fd. unit_status Databricks Notebook nb-create-unit- status-table Databricks Notebook nb-create-delta- table Create Delta Table unit_status UPSERTS Power BI Premium Power BI Report Streaming- demo.eventsimulator Databricks Notebook
- 8. Demo - Organization Creation of Delta Lake Table Implementation Resources Walk Through Spark Streaming Notebook Stream Processor Function Demo Run Event Simulator
- 9. Demo 1 ā Infrastructure Walkthrough
- 10. Demo 2 ā Code Walkthrough
- 11. Demo 3 ā Sample Run
- 12. Summary ā¢ The need for large scale real-time stream processing become more evident every day ā¢ Provide organizations with the ability to respond quickly to a dynamic business climate ā¢ Spark Structured Streaming makes it easy to add a real- time channel ā¢ Simple extensions on top of Spark SQL
- 13. Feedback Your feedback is important to us. Donāt forget to rate and review the sessions.