Big data streaming with Apache Spark on Azure
Download as PPTX, PDF1 like411 views
The document discusses the integration of big data streaming with microservices and analytics using an event bus architecture, emphasizing the independence and scalability of microservices. It presents scenarios for generating annual sales reports and detecting anomalies using machine learning and data lakes, along with tools like Apache Spark for data processing. Additionally, it provides insights into deploying these solutions with Azure services, including Event Hubs and HDInsight.
Big data streaming with Apache Spark on Azure
- 1. Big data streaming Willem Meints
- 3. Event bus Micro services • Multiple smaller services that scale independedely • Each service his own data store • Data flows between services through the event bus
- 5. Data analytics challenges with microservices • A complete picture is there, but spread over a vast landscape • Most data doesn’t come in a database • Data changes rapidly
- 7. Scenario 1: Get a annual sales report • The goal is to get a complete picture of the situation • Data based on business events
- 10. Scenario 2: Detect anomalies • The goal is to detect anomalies on the website and prevent abuse • Machine learning needed to detect the anomalies • Data based on the data lake
- 11. Click stream collector Event bus Data analytics Data Lake
- 12. Click stream collector Event bus Data analytics Data Lake Model
- 13. Analytics tools
- 14. vs
- 16. Event bus Data processing tool Distributed database Alerting Dashboarding Flow control logic Cluster Manager
- 17. The Azure based solution
- 18. Azure Event Hub HDInsight Azure Data Lake Alerting Dashboarding Azure App Services Cluster Manager
- 19. Demo A short introduction into Apache Spark
- 20. Spark SQL Spark Streaming Machine Learning GraphX Apache Spark Core
- 22. Resilient Distributed Data Sets Resilient Distributed Dataset Partition Record Record Partition Record Record
- 23. Stream Batches Processed data Streams with Spark
- 24. Stream Batches Processed data Streams with Spark Lists of RDDs
- 25. Demo Deploying Spark to Azure using HDInsight
- 26. Azure Event Hubs • Capable of streaming large volumes of data • SDK available in many languages • Ruby • Python • Java/Scala • C# • Apache Spark
- 27. Hoe werkt een Azure Event Hub? Partition Partition Partition Consumer group Consumer group
- 28. Demo Using Azure Event Hub with Spark
- 29. Tips for going in production • When using streams, always have n+1 worker nodes • More partitions = more speed • Longer intervals is slower, but sometimes better
- 30. Thanks! Willem Meints Technical Evangelist/Microsoft MVP @willem_meints