![U-SQL Script Structure
Script :=
Statement_List.
Statement_List :=
{ [Statement] ';' }.
Statement :=
Use_Statement
| If_Else_Statement
| Declare_Variable_Statement
| Reference_Assembly_Statement
| Deploy_Resource_Statement
| DDL_Statement
| Query_Statement
| Procedure_Call
| Import_Package_Statement
| DML_Statement
| Output_Statement.](https://image.slidesharecdn.com/aibigdataconferenceeugenepolonichkoazuredatalake-171107113032/85/Ai-big-dataconference_eugene_polonichko_azure-data-lake-16-320.jpg)
Ai big dataconference_eugene_polonichko_azure data lake
- 1. Azure Data Lake: What is it? Why is it? Where is it? EUGENE POLONICHKO DATA PLATFORM MVP BIDWH ARCHITECT
- 2. About me Eugene Polonichko has over 7 years of experience with SQL Server. He mainly focused on BI projects (SSAS, SSIS, PowerBI, Cognos, Informatica PowerCenter, Pentaho, Tableau). Eugene is a passionate speaker and SQL community volunteer presenting regularly at PASS SQL Saturday events and local user groups around Ukraine and Europe. Eugene is PASS Chapter Leader and he has a status MVP Data Platform https://www.linkedin.com/in/eugenepolonichko/ https://twitter.com/EvgenPolonichko
- 3. Agenda What is Data Lake? Architecture of Azure Data Lake Azure Data Lake Store Overview of Azure Data Lake Store Compare For big data processing Azure Data Lake Analytics U-SQL Concepts U-SQL Script Structure Extractors U-SQL Jobs U-SQL catalog Monitoring and performance U-SQL jobs Data Lake Analytics pricing
- 4. Data Lake
- 5. Data Lake
- 6. Architecture of Azure Data Lake
- 7. Azure Data Lake Stores Azure Data Lake Store is a hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics. The Azure Data Lake store is an Apache Hadoop file system compatible with Hadoop Distributed File System (HDFS) Can be accessed from Hadoop (available with HDInsight cluster) using the WebHDFS-compatible REST APIs
- 8. Azure Data Lake Stores Use Cases Store social media posts, log files, sensor data Store corporate data such as relational databases (as flat files)
- 9. Data Lake Storage vs Azure Storage Optimized storage for big data analytics workloads General purpose object store for a wide variety of storage scenarios Batch, interactive, streaming analytics, log files and etc Any type of text or binary data, such as application back end, account contains folders, which in turn contains data stored as files Storage account has containers Optimized performance for parallel analytics workloads. High Throughput and IOPS. Not optimized for analytics workloads
- 12. DEMO
- 13. Azure Data Lake Analytics Azure Data Lake Analytics is an on-demand analytics job service to simplify big data analytics. You can focus on writing, running, and managing jobs rather than on operating distributed infrastructure. Dynamic scaling Develop faster, debug, and optimize smarter using familiar tools Affordable and cost effective Works with all your Azure Data U-SQL: simple and familiar, powerful, and extensible
- 15. Concepts Retrieve data from stored locations in rowset format Transform the rowset(s) Transform the rowset(s)
- 16. U-SQL Script Structure Script := Statement_List. Statement_List := { [Statement] ';' }. Statement := Use_Statement | If_Else_Statement | Declare_Variable_Statement | Reference_Assembly_Statement | Deploy_Resource_Statement | DDL_Statement | Query_Statement | Procedure_Call | Import_Package_Statement | DML_Statement | Output_Statement.
- 18. U-SQL Built-in Extractors: Extractors.Text() : Extractors.Csv() Extractors.Tsv() Extractors
- 20. U-SQL Jobs ADLAUs Azure Data Lake Analytics Unit Parallelism N = N ADLAUs 1 ADLAU ~= A VM with 2 cores and 6 GB of memory
- 21. U-SQL Jobs
- 23. DEMO
- 26. DEMO
- 27. Pricing
- 29. Questions?
- 30. Thank you