NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
1 like177 views
This document discusses analyzing StackExchange data with Azure Data Lake. It introduces Azure Data Lake Store as an enterprise-grade data lake and Azure Data Lake Analytics as a serverless big data analytics service. It demonstrates using these services to acquire, aggregate, analyze and visualize StackExchange data. Key features of Data Lake Store and Analytics are explained, including security, pricing, and extensibility options.
NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
- 1. Analyzing StackExchange data with Azure Data Lake Tom Kerkhove Azure Consultant
- 2. Tom Kerkhove Azure Consultant @ Codit Microsoft Azure MVP & Advisor “Integration of Things” whitepaper (https://bit.ly/azure-iot) Nice to meet you blog.tomkerkhove.be @TomKerkhove tomkerkhove
- 3. Agenda • Introduction to Azure Data Lake • What is Azure Data Lake Store? • What is Azure Data Lake Analytics? 3
- 4. 4
- 6. Let’s go open-source, right?! ➔ Comes with a few challenges for C#/SQL professional ➔ New languages to learn & maintain ➔ Rapidly evolving ecosystem ➔ Cluster management ➔ Typically linux machines
- 7. Analyzing Big Data in Azure
- 8. ➔ WebHDFS compatible ➔ Any size ➔ Any format as-is ➔ Write-once-read-many ➔ Enterprise-grade security ➔ Thé big data store in Azure Azure Data Lake Store
- 10. Characteristics ➔ Data Warehousing ➔ Structured data ➔ Defined set of schemas ➔ Requires Extract-Transform- Load (ETL) before storing ➔ Known for some of us ➔ Exploratory analysis is hard because of transforming the data Data Warehousing vs Data Lakes ➔ Data Lakes ➔ Raw data (unstructured/semi-structured/structured) ➔ “Dump” all your data in the lake ➔ Data scientists will interpret data from the lake ➔ Without metadata, turns in a data swamp pretty fast
- 11. Martin Fowler on Data Lake & Data Warehouses: https://bit.ly/martin-fowler-data-lake
- 12. Security ➔ Roled-based Access Control (RBAC) ➔ Grant user/groups access to folder/file (https://bit.ly/data-lake-store-acls) ➔ Firewall (off by default) ➔ Encryption at rest ➔ Keys managed by Microsoft ➔ Bring-your-own-key with Azure Key Vault
- 13. ➔ ~$0,032/GB stored per month ➔ Transaction costs ➔ ~$0,043 per 1M write transactions ➔ ~$0,0034 per 1M read transactions ➔ 1 transaction is block of up to 128 kB ➔ Regular Egress fees ➔ Monthly commitment packages ➔ Save up to 33% Pricing
- 14. Azure Data Lake Store vs Blob Storage No Limitations Store whatever you want in any format Security Built-in Azure Active Directory support Pricing More expensive than Storage GRS Redundancy It’s there but no control over it Built for Scale Optimized for high- scale reads Integration With Data Factory, Data Catalog & HDInsight Full comparison on https://bit.ly/adls-vs-storage
- 15. Demo – Data Lake Store 15
- 16. Meet StackExchange ➔ Over 280 websites ➔ 150+ GB of open-source data ➔ Different kinds of data ➔ Posts ➔ Users ➔ Votes ➔ ... ➔ A big data sample data set
- 17. What Are We Going To Do? • Download the original data set Acquiring The Data • Upload data set to Azure • Determine what service to use Moving The Data • Merging data from each site into one file • Conversion from XML to CSV Aggregating The Data • Run business logic on it • Attempt to gain knowledge from it Analyzing The Data • Visualize what we’ve learned Visualizing The Data
- 18. How is it setup?
- 19. Azure Data Lake Analytics ➔ Run analytics jobs on managed clusters ➔ No maintenance ~ Serverless ➔ Written in U-SQL ➔ SQL Syntax ➔ Extensibility in C# ➔ Easily scaled with Analytics Units ➔ Pay for processing time only
- 20. ➔ Built-in partitioned tables ➔ Query data where it lives ➔ No need to prepare data ➔ One query that runs on multiple data stores ➔ Use the correct data store for the job Data Sources
- 21. Writing U-SQL scripts Extract from data source by using built-in or custom extractors. Transform / Analyse the data using SQL-syntax, in-line C# or C# method calls Output the result to a data source by using built-in or custom extractors
- 22. ➔ C# Expressions ➔ User-Defined Functions (UDF) ➔ User-Defined Operations (UDO) ➔ User-Defined Aggregators (UDAGG) Extensibility
- 23. ➔ User-Defined Extractors ➔ User-Defined Processors ➔ Take one row and produce one row ➔ Pass-through versus transforming ➔ User-Defined Reducers ➔ Take n rows and produce 1 row ➔ User-Defined Outputters ➔ User-Defined Appliers ➔ Take one row and produce 0 to n rows ➔ Used with OUTER/CROSS APPLY ➔ User-Defined Combiners ➔ Combines rowsets (like a user- defined join) User-Defined Operations (UDO)
- 24. Metadata Model
- 25. U-SQL Batch Job Execution Lifetime Michael Rys on “Tuning & Optimizing U-SQL” https://bit.ly/tuning-optimizing-u-sql
- 27. Job States
- 28. ➔ Roled-based Access Control (RBAC) ➔ Firewall (Off by default) ➔ Access control on service catalog ➔ Access control on a per-database level Security
- 29. ➔ Account-level limitations ➔ Maximum of AUs ➔ Maximum of concurrent job ➔ Days to retain queries ➔ Job-level limitations ➔ Maximum of AUs ➔ Maximum priority ➔ Granted per user and/or group Resource Management
- 30. Demo – Data Lake Analytics 30
- 31. ➔ Store Explorer ➔ Browse store ➔ Download complete / subset of file ➔ Preview ➔ Only in Visual Studio ➔ Job Visualizer ➔ Determine bottlenecks by using heatmaps ➔ Playback jobs based on telemetry ➔ Query optimization ➔ Job Profiler Azure Data Lake tools for Visual Studio
- 32. ➔ Integration with Source control ➔ Unit Testing extensibility ➔ Local execution ➔ Simulate Data Lake Store ➔ Run & debug jobs Azure Data Lake tools for Visual Studio (Code)
- 33. ➔ Billed for processing time, not per job ➔ Billed per second ➔ $1,687 per hour per Analytics Unit ➔ ~ $0,028 per minute ➔ Monthly commitment packages ➔ Save up to 74% Pricing
- 34. Operations Data Lake Store Data Lake Analytics Available Graphs • Storage Utilization • Read & Write • Ingress & Egress • Job status • Used # of AU time Available Metrics • Data Read & Write • Read & Write Requests • Total Storage • Job status • Used # of AU time Support for alerts Yes Built-in & custom Log Analytics queries (Requires Audit logs) Support for Audit Logs Yes Yes Support for Request Logs Yes Yes
- 35. ➔ Integrate with your data pipelines in Azure Data Factory ➔ Move data from Azure Data Lake Store to other store ➔ Move data to Azure Data Lake Store ➔ Run U-SQL jobs within pipeline ➔ Integration with Azure Data Catalog ➔ Register your Azure Data Lake Store assets Integration with Azure Services
- 36. ➔ Azure Data Architecture Guide (https://docs.microsoft.com/en-us/azure/architecture/data-guide/) ➔ “Mastering Azure Analytics” by Zoiner Tejada (https://bit.ly/mastering-azure-analytics) ➔ MVA “Introducing Azure Data Lake” (https://bit.ly/intro-to-azure-data-lake) ➔ Azure Data Lake GitHub Repo (https://azure.github.io/AzureDataLake/) ➔ U-SQL Documentation (https://usql.io) Learn more!
- 37. ➔ Big Data is not just a hype so get ready ➔ Azure Data Lake Store ➔ Analyse today & explore tomorrow ➔ Beware of the data swamps ➔ Data Lake Analytics ➔ Serverless ➔ Re-use existing skills ➔ Pay for what we use ➔ Big Data in Azure? Use Azure Data Lake! Summary
- 38. 38
Editor's Notes
- #8: HDI – Managed cluster service, Open-source technology, Runs on Windows or Linux Store – Unlimited Storage, WebHDFS Analytics - Managed job service, U-SQL batch-processing Based on MSFT Cosmos Cortana, Bing, Xbox Live, etc.
- #12: Analogy with fishing – Go fishing in lake, but it in your warehouse. Lake becomes swamp, fish dies
- #15: No Limitations – Store is unlimited, storage is limited to 100 accounts in a subscription, 500 TB each Security –AAD vs SAS or Name/Key auth Pricing – ADLS is more expensive Redundancy – No control over redundancy Built for Scale – Optimized for high reads and analytics, scales with the reads, high volume of small writes Real-time analytics