Day 1 - Technical Bootcamp azure synapse analytics

Editor's Notes

• #6: Hand it over to Lino
• #12: Need to explain and why needed Check with Gaurav
• #18: formatting
• #26: What specific approach would you say is the most efficient way for moving flat file data from the ingest storage locations to the data lake? Follow the pattern of landing data in the data lake first, then ingest from the flat files into relational tables within the data warehouse. Then create pipelines that extract the source data and store in Azure Data Lake Store Gen2 as Parquet files. What storage service would you recommend to use? They should use Azure Data Lake Store (ADLS) Gen2 (Azure Storage with hierarchical file systems).
• #27: How would you recommend to structure the folder to manage the data at the various levels of refinement? They should use Azure Data Lake Store (ADLS) Gen2 (Azure Storage with hierarchical file systems). In ADLS, it is a best practice to have a dedicated Storage Account for production, and a separate Storage Account for dev and test workloads. This will ensure that dev or test workloads never interfere with production. One common folder structure is to organize the data in separate folders by degree of refinement. For example a bronze folder contains the raw data, silver contains the cleaned, prepared and integrated data and gold contains data ready to support analytics, which might include final refinements such as pre-computed aggregates.
• #31: When it comes to ingesting raw data in batch from new data sources, what data formats are supported by Synapse? CSV, Parquet, ORC, JSON How do you ingest streaming data? Collect messages in Event Hub or IoT Hub and process them with Stream Analytics. Azure offers purpose-built stream ingestion services such as Azure Kafka and Azure Event Hubs that are robust, proven and performant. (Preview) Azure Synapse will also support native stream ingestion through integration with Azure Stream Analytics. When it comes to storing refined versions of the data for possible querying, what data format would you recommend they use? Why? Parquet. There is industry alignment around the Parquet format for sharing data at the storage layer (e.g., across Hadoop, Databricks, and SQL engine scenarios). Parquet is a high-performance, column oriented format optimized for big data scenarios.
• #33: https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-sql-data-warehouse#polybase-troubleshooting <- a good resource to share out as well Lino 68
• #34: What should you use for the fastest loading of staging tables? A heap table. If you are loading data only to stage it before running more transformations, loading the table to heap table is much faster than loading the data to a clustered columnstore table. A temporary table. Loading data to a temporary table loads faster than loading a table to permanent storage.
• #35: How should you configure table distribution for tables created from the raw input that might be useful, or tables used for staging? Consider round-robin distribution. A round-robin distributed table distributes table rows evenly across all distributions. The assignment of rows to distributions is random. Unlike hash-distributed tables, rows with equal values are not guaranteed to be assigned to the same distribution. As a result, the system sometimes needs to invoke a data movement operation to better organize your data before it can resolve a query. This extra step can slow down your queries. For example, joining a round-robin table usually requires reshuffling the rows, which is a performance hit. Consider using the round-robin distribution for your table in the following scenarios: When getting started as a simple starting point since it is the default If there is no obvious joining key If there is not good candidate column for hash distributing the table If the table does not share a common join key with other tables If the join is less significant than other joins in the query When the table is a temporary staging table
• #38: FALSE – only Polybase requires CONTROL permission
• #62: Important Points: No need to pre-create the table in the SQL Pool to write to, it will be created if it does not exist. sqlanalytics only works in Scala language cells. To read with other languages like Python, use Spark to register a temporary view and then query the view using Spark.SQL(“select * from viewname”) To write with other languages, create a view for table you want write, then query from that view in Scala. Then write.
• #69: Statistics synchronous option to be asynchronous Recalculated when there are changes over 500 rows or 20% of rows are updated By default, auto update statistics are configured to be synchronous but can be configured to be asynchronous operations Statistics are updated opportunistically when queries are run.
• #70: CSV support added [Cost management for serverless SQL pool - Azure Synapse Analytics | Microsoft Docs] When statistics are created for a Parquet column, only the relevant column is read from files. When statistics are created for a CSV column, whole files are read and parsed.
• #75: 1,048,576 – this is why we recommend at least 60 million rows for a CCI table. Azure Synapse automatically distributes data into 60 distribution, and each distribution needs at least 1 million rows for good rowgroup compression (or each partition, if your table is partitioned).
• #83: Answer: PERFORMANCE for both! A well designed and optimized Azure Synapse can blow away the competition but falling prey to common performance pitfalls can hurt our chances.
• #85: Their downstream reports are used by many users, which often means the same query is being executed repeatedly against data that does not change that often. What can WWI to improve the performance of these types of queries? How does this approach work when the underlying data changes? They should consider result-set caching. Cache the results of a query in provisioned Azure Synapse SQL Pool storage. This enables interactive response times for repetitive queries against tables with infrequent data changes. The result-set cache persists even if SQL pool is paused and resumed later. Query cache is invalidated and refreshed when underlying table data or query code changes. Result cache is evicted regularly based on a time-aware least recently used algorithm (TLRU).
• #86: Result-set caching The maximum size of the result-set cache is 1TB Query results are persisted for a maximum of 48 hours but can be evicted earlier to save space based on the least recently used result Disabled on DW by default unless turned on at a session level or the entire database level Additional storage costs are incurred by caching query result sets Check the is_result_set_caching column in the sys.databases DMV to show the result-set caching setting for a database Users can tell if a query was executed with a result cache hit or miss by querying sys.pdw_request_steps for commands where value is like ‘%DWResultCacheDb%’
• #87: Result-set caching The maximum size of the result-set cache is 1TB Query results are persisted for a maximum of 48 hours but can be evicted earlier to save space based on the least recently used result Disabled on DW by default unless turned on at a session level or the entire database level Additional storage costs are incurred by caching query result sets Check the is_result_set_caching column in the sys.databases DMV to show the result-set caching setting for a database Users can tell if a query was executed with a result cache hit or miss by querying sys.pdw_request_steps for commands where value is like ‘%DWResultCacheDb%’
• #88: Current Limitations: If MIN/MAX aggregates are used in the SELECT list, the indexed view will automatically be disabled when UPDATE and DELETE occur in the referenced base tables. Run ALTER INDEX with REBUILD to re-enable the indexed view Only INNER JOIN is supported Only HASH and ROUND_ROBIN distributions are supported Only CLUSTERED COLUMNSTORE INDEX is supported ALTER VIEW is not supported
• #95: A distributed table appears as a single table, but the rows are actually stored across 60 distributions. The rows are distributed with a hash or round-robin algorithm. Hash distributed A hash-distributed table distributes table rows across the Compute nodes by using a deterministic hash function to assign each row to one distribution. Since identical values always hash to the same distribution, the data warehouse has built-in knowledge of the row locations. SQL Data Warehouse uses this knowledge to minimize data movement during queries, which improves query performance. Hash-distributed tables work well for large fact tables in a star schema. They can have very large numbers of rows and still achieve high performance. There are, of course, some design considerations that help you to get the performance the distributed system is designed to provide. Choosing a good distribution column is one such consideration that is described in this article. Consider using a hash-distributed table when: The table size on disk is more than 2 GB. The table has frequent insert, update, and delete operations. Round-robin distributed A round-robin distributed table distributes table rows evenly across all distributions. The assignment of rows to distributions is random. Unlike hash-distributed tables, rows with equal values are not guaranteed to be assigned to the same distribution. As a result, the system sometimes needs to invoke a data movement operation to better organize your data before it can resolve a query. This extra step can slow down your queries. For example, joining a round-robin table usually requires reshuffling the rows, which is a performance hit. Consider using the round-robin distribution for your table in the following scenarios: When getting started as a simple starting point since it is the default If there is no obvious joining key If there is not good candidate column for hash distributing the table If the table does not share a common join key with other tables If the join is less significant than other joins in the query When the table is a temporary staging table Replicated Tables A replicated table has a full copy of the table accessible on each Compute node. Replicating a table removes the need to transfer data among Compute nodes before a join or aggregation. Since the table has multiple copies, replicated tables work best when the table size is less than 2 GB compressed. Replicated tables work well for small dimension tables in a star schema. Dimension tables are usually of a size that makes it feasible to store and maintain multiple copies. Dimensions store descriptive data that changes slowly, such as customer name and address, and product details. The slowly changing nature of the data leads to fewer rebuilds of the replicated table. Consider using a replicated table when: The table size on disk is less than 2 GB, regardless of the number of rows. To find the size of a table, you can use the DBCC PDW_SHOWSPACEUSED command: DBCC PDW_SHOWSPACEUSED('ReplTableCandidate'). The table is used in joins that would otherwise require data movement. When joining tables that are not distributed on the same column, such as a hash-distributed table to a round-robin table, data movement is required to complete the query. If one of the tables is small, consider a replicated table. We recommend using replicated tables instead of round-robin tables in most cases. To view data movement operations in query plans, use sys.dm_pdw_request_steps. The BroadcastMoveOperation is the typical data movement operation that can be eliminated by using a replicated table. Source: https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-tables-distribute https://docs.microsoft.com/en-us/azure/sql-data-warehouse/design-guidance-for-replicated-tables
• #96: What are table partitions? Table partitions enable you to divide your data into smaller groups of data. In most cases, table partitions are created on a date column. Partitioning is supported on all SQL Data Warehouse table types; including clustered columnstore, clustered index, and heap. Partitioning is also supported on all distribution types, including both hash or round robin distributed. Partitioning can benefit data maintenance and query performance. Whether it benefits both or just one is dependent on how data is loaded and whether the same column can be used for both purposes, since partitioning can only be done on one column. Benefits to loads The primary benefit of partitioning in SQL Data Warehouse is to improve the efficiency and performance of loading data by use of partition deletion, switching and merging. In most cases data is partitioned on a date column that is closely tied to the order in which the data is loaded into the database. One of the greatest benefits of using partitions to maintain data it the avoidance of transaction logging. While simply inserting, updating, or deleting data can be the most straightforward approach, with a little thought and effort, using partitioning during your load process can substantially improve performance. Partition switching can be used to quickly remove or replace a section of a table. For example, a sales fact table might contain just data for the past 36 months. At the end of every month, the oldest month of sales data is deleted from the table. This data could be deleted by using a delete statement to delete the data for the oldest month. However, deleting a large amount of data row-by-row with a delete statement can take too much time, as well as create the risk of large transactions that take a long time to rollback if something goes wrong. A more optimal approach is to drop the oldest partition of data. Where deleting the individual rows could take hours, deleting an entire partition could take seconds. Benefits to queries Partitioning can also be used to improve query performance. A query that applies a filter to partitioned data can limit the scan to only the qualifying partitions. This method of filtering can avoid a full table scan and only scan a smaller subset of data. With the introduction of clustered columnstore indexes, the predicate elimination performance benefits are less beneficial, but in some cases there can be a benefit to queries. For example, if the sales fact table is partitioned into 36 months using the sales date field, then queries that filter on the sale date can skip searching in partitions that don’t match the filter. Sizing partitions While partitioning can be used to improve performance some scenarios, creating a table with too many partitions can hurt performance under some circumstances. These concerns are especially true for clustered columnstore tables. For partitioning to be helpful, it is important to understand when to use partitioning and the number of partitions to create. There is no hard fast rule as to how many partitions are too many, it depends on your data and how many partitions you loading simultaneously. A successful partitioning scheme usually has tens to hundreds of partitions, not thousands. When creating partitions on clustered columnstore tables, it is important to consider how many rows belong to each partition. For optimal compression and performance of clustered columnstore tables, a minimum of 1 million rows per distribution and partition is needed. Before partitions are created, SQL Data Warehouse already divides each table into 60 distributed databases. Any partitioning added to a table is in addition to the distributions created behind the scenes. Using this example, if the sales fact table contained 36 monthly partitions, and given that SQL Data Warehouse has 60 distributions, then the sales fact table should contain 60 million rows per month, or 2.1 billion rows when all months are populated. If a table contains fewer than the recommended minimum number of rows per partition, consider using fewer partitions in order to increase the number of rows per partition. For more information, see the Indexing article, which includes queries that can assess the quality of cluster columnstore indexes. Source: https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-tables-partition
• #99: Answer: Because the risky part of the load is moving data from external storage into the SQL Pool – loading into a staging table (round robin/heap) ensures that the riskiest part of the load happens the fastest!
• #101: What are the typical issues they should look out for with regards to distributed table design for the following scenarios? Their smallest fact table exceeds several GB’s and by their nature experiences frequent inserts. They should use a hash distribution. A hash-distributed table distributes table rows across the Compute nodes by using a deterministic hash function to assign each row to one distribution. Since identical values always hash to the same distribution, the data warehouse has built-in knowledge of the row locations. SQL Data Warehouse uses this knowledge to minimize data movement during queries, which improves query performance. Hash-distributed tables work well for large fact tables in a star schema. They can have very large numbers of rows and still achieve high performance. Consider using a hash-distributed table when: The table size on disk is more than 2 GB. The table has frequent insert, update, and delete operations. As they develop the data warehouse, the WWI data team identified some tables created from the raw input that might be useful, but they don’t currently join to other tables and they are not sure of the best columns they should use for distributing the data. They should consider round-robin distribution. A round-robin distributed table distributes table rows evenly across all distributions. The assignment of rows to distributions is random. Unlike hash-distributed tables, rows with equal values are not guaranteed to be assigned to the same distribution. As a result, the system sometimes needs to invoke a data movement operation to better organize your data before it can resolve a query. This extra step can slow down your queries. For example, joining a round-robin table usually requires reshuffling the rows, which is a performance hit. Consider using the round-robin distribution for your table in the following scenarios: When getting started as a simple starting point since it is the default If there is no obvious joining key If there is not good candidate column for hash distributing the table If the table does not share a common join key with other tables If the join is less significant than other joins in the query When the table is a temporary staging table Their data engineers sometimes use temporary staging tables in their data preparation. They should use a round-robin distributed table. They have lookup tables that range from several hundred MBs to 1.5 GBs. They should consider using replicated tables. A replicated table has a full copy of the table accessible on each Compute node. Replicating a table removes the need to transfer data among Compute nodes before a join or aggregation. Since the table has multiple copies, replicated tables work best when the table size is less than 2 GB compressed.
• #102: Statistics synchronous option to be asynchronous Recalculated when there are changes over 500 rows or 20% of rows are updated By default, auto update statistics are configured to be synchronous but can be configured to be asynchronous operations Statistics are updated opportunistically when queries are run.
• #106: Clustered Columnstore, Clustered index (and non-clustered index), Heap
• #108: Clustered columnstore indexes By default, SQL Data Warehouse creates a clustered columnstore index when no index options are specified on a table. Clustered columnstore tables offer both the highest level of data compression as well as the best overall query performance. Clustered columnstore tables will generally outperform clustered index or heap tables and are usually the best choice for large tables. For these reasons, clustered columnstore is the best place to start when you are unsure of how to index your table. There are a few scenarios where clustered columnstore may not be a good option: Columnstore tables do not support varchar(max), nvarchar(max) and varbinary(max). Consider heap or clustered index instead. Columnstore tables may be less efficient for transient data. Consider heap and perhaps even temporary tables. Small tables with less than 100 million rows. Consider heap tables. Clustered and nonclustered indexes Clustered indexes may outperform clustered columnstore tables when a single row needs to be quickly retrieved. For queries where a single or very few row lookup is required to performance with extreme speed, consider a cluster index or nonclustered secondary index. The disadvantage to using a clustered index is that only queries that benefit are the ones that use a highly selective filter on the clustered index column. To improve filter on other columns a nonclustered index can be added to other columns. However, each index which is added to a table adds both space and processing time to loads. Heap tables When you are temporarily landing data on SQL Data Warehouse, you may find that using a heap table makes the overall process faster. This is because loads to heaps are faster than to index tables and in some cases the subsequent read can be done from cache. If you are loading data only to stage it before running more transformations, loading the table to heap table is much faster than loading the data to a clustered columnstore table. In addition, loading data to a temporary table loads faster than loading a table to permanent storage. For small lookup tables, less than 100 million rows, often heap tables make sense. Cluster columnstore tables begin to achieve optimal compression once there is more than 100 million rows. Source: https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-tables-index
• #109: Format slide
• #110: Formatting
• #112: Their sales transaction dataset exceeds a billion rows. For their downstream reporting queries, they need to be able to join, project and filter these rows in no longer than 10s of seconds. WWI is concerned their data is just too big to do this. What specific indexing techniques should they use to reach this kind of performance for their fact tables? Why? Clustered Columnstore Indexes. As they offer the highest level of data compression and best overall query performance, columnstore indexes are usually the best choice for large tables such as fact tables. Would you recommend the same approach for tables they have with less than 100 million rows? No. For "small" tables with less than 100 million rows, they should consider Heap tables. How should they configure indexes on their smaller lookup tables (e.g., those that contain store names and addresses)? They should consider using Heap tables. For small lookup tables, less than 100 million rows, often heap tables make sense. Cluster columnstore tables begin to achieve optimal compression once there is more than 100 million rows. What would you suggest for their larger lookup tables that are used just for point lookups that retrieve only a single row? How could they makes these more flexible so that queries filtering against different sets of columns would still yield efficient lookups? Use clustered indexes. Clustered indexes may outperform clustered columnstore tables when a single row needs to be quickly retrieved. For queries where a single or a very few number of rows to lookup is required to perform with extreme speed, consider a clustered index or non-clustered secondary index. The disadvantage to using a clustered index is that the only queries that benefit are the ones that use a highly selective filter on the clustered index column. To improve filter performance on other columns, a non-clustered index can be added to other columns. However, be aware that each index which is added to a table adds both space and processing time to data loads. What should they use for the fastest loading of staging tables? A heap table. If you are loading data only to stage it before running more transformations, loading the table to heap table is much faster than loading the data to a clustered columnstore table. A temporary table. Loading data to a temporary table loads faster than loading a table to permanent storage.