U-SQL Query Execution and Performance Tuning

U-SQL Query Execution and Performance Tuning

• 1.
  U-SQL Query Executionand Performance Tuning
• 2.
  • • • • • •
• 3.
  • Job ExecutionExperience and Investigations Query Execution Stage Graph Dryad crash course Job Metrics Resource Planning • Partitioning Analysis Analyze the critical path Heat Map Critical Path Data Skew • Tuning / Optimizations Data Partitioning Partition Elimination Predicate Pushing Column Pruning Some Data Hints UDOs can be evil INSERT optimizations U-SQL Query Execution and Performance Tuning
• 5.
   Automatic "in-lining" optimized out-of- the-box  Per job parallelization  visibility into execution  Heatmap to identify bottlenecks
• 6.
  Preparing Queued Running Finalizing Ended (Succeeded, Failed, Cancelled) New Compiling Queued Scheduling Starting Running Ended Whatyou see in the UX Underlying Job State The script is being compiled by the Compiler Service All jobs enter the queue. Are there enough ADLAUs to start the job? If yes, then allocate those ADLAUs for the job The U-SQL runtime is now executing the code on 1 or more ADLAUs or finalizing the outputs The job has concluded.
• 7.
  U-SQL C# user code C++ systemcode Algebra other files (system files, deployed resources) managed dll Unmanaged dll Input script Compilation output (in job folder) Compiler & Optimizer Files Meta Data Service Deployed to vertices
• 8.
  Some fixed amountof work Each square is called a “vertex” Each vertex represents a fraction of the work
• 9.
  U-SQL Query Execution Physicalplans vs. Dryad stage graph…
• 10.
  252 Pieces ofwork AVG Vertex execution time 4.3 Billion rows Data Read & Written
• 11.
  U-SQL Query Execution Dryadas an art form…
• 12.
  13 U-SQL Query Execution Redefinitionof big-data…
• 13.
  14 U-SQL Query Execution Redefinitionof big-data…
• 15.
  16 U-SQL Performance Analysis Analyzethe critical path, heat maps, playback, and runtime metrics on every vertex…
• 17.
    data maybe distributed such that all rows that match a certain key go to a single vertex  imbalanced execution, vertex time out. 0 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 30,000,000 35,000,000 40,000,000 California NewYork Illinois Ohio Michigan NewJersey Washington Arizona Tennessee Maryland Minnesota Alabama Louisiana Oregon Connecticut Mississippi Kansas Nevada Nebraska Idaho Maine RhodeIsland Delaware Alaska DistrictofColumbia Wyoming Population by State
• 18.
      @rows = SELECT Gender,AGG<MyAgg>(…) AS Result FROM @HugeInput GROUP BY Gender; Gender==Male Gender==Female @HugeInput Vertex 0 Vertex 1
• 19.
  1 2 34 5 6 7 8 36 1 2 3 4 5 6 7 8 6 15 15 36
• 20.
  U-SQL Partitioning duringProcessing Data Skew
• 21.
  U-SQL Partitioning Data Skew– Recursive Reducer // Metrics per domain @Metric = REDUCE @Impressions ON UrlDomain USING new Bing.TopNReducer(count:10) ; // … Inherent Data Skew [SqlUserDefinedReducer(IsRecursive = true)] public class TopNReducer : IReducer { public override IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output) { // Compute TOP(N) per group // … } } Recursive • Allow multi-stage aggregation trees • Requires same schema (input => output) • Requires associativity: • R(x, y) = R( R(x), R(y) ) • Default = non-recursive • User code has to honor recursive semantics www.bing.com brought to a single vertex
• 22.
  U-SQL Partitioning duringProcessing Partitioning – Combiner Modes // Existing Watson hits @DeDup = COMBINE @Exceptions AS L WITH @WatsonBuckets AS R ON L.AppId WITH R.AppId USING new Windows.WatsonDedupCombiner() ; // … [SqlUserDefinedCombiner(Mode = CombinerMode.Right)] public class WatsonDedupCombiner : ICombiner { public override IEnumerable<IRow> Combine(IRowset left, IRowset right, IUpdatableRow output) { // DeDup against existing Call Stacks // … } } CombinerMode • Allow parallelism even within a partition public enum CombinerMode { Inner, /// Inner join - both row level Left, /// Left Group Join - left row level Right, /// Right Group Join - right row level Full /// Full Group Join - none row level • Default = Full Group Join • User code has to honor row level semantics Row Level Combiner X IDother columns X Q Q F Q X X Z F F X LEFT X ID other columns Q Q F Q X Z F F X RIGHT F Z M1 M2 M3 M4 Enables Broadcast JOIN
• 23.
               
• 24.
            Partition Scheme Whento use? HASH(keys) DIRECT HASH Exact control of hash bucket RANGE(keys) Keeps ranges together ROUND ROBIN To get equal distribution (if others give skew)
• 26.
  // Unstructured Files(24 hours daily log impressions) @Impressions = EXTRACT ClientId int, Market string, OS string, ... FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif" FROM @"wasb://ads@wcentralus/2015/10/30/{Market:*}_{*}.nif" ; // … // Filter to by Market @US = SELECT * FROM @Impressions WHERE Market == "en" ; U-SQL Optimizations Partition Elimination – Unstructured Files Partition Elimination • Even with unstructured files! • Leverage Virtual Columns (Named) • Avoid unnamed {*} • WHERE predicates on named virtual columns • That binds the PE range during compilation time • Named virtual columns without predicate = error • Design directories/files with PE in mind • Design for elimination early in the tree, not in the leaves Extracts all files in the folder Post filter = pay I/O cost to drop most data PE pushes this predicate to the EXTRACT EXTRACT now only reads “en” files! en_10.0.nif en_8.1.nif de_10.0.nif jp_7.0.nif de_8.1.nif ../2015/10/30/ …
• 27.
  // TABLE(s) -Structured Files (24 hours daily log impressions) CREATE TABLE Impressions (Day DateTime, Market string, ClientId int, ... INDEX IX CLUSTERED(Market, ClientId) PARTITIONED BY BUCKETS (Day) HASH(Market, ClientId) INTO 100 ); DECLARE @today DateTime = DateTime.Parse("2015/10/30"); // Market = Vertical Partitioning ALTER TABLE Impressions ADD PARTITION (@today); // … // Daily INSERT(s) INSERT INTO Impressions(Market, ClientId) PARTITION(@today) SELECT * FROM @Q ; // … // Both levels are elimination (H+V) @Impressions = SELECT * FROM dbo.Impressions WHERE Market == "en" AND Day == @today ; U-SQL Optimizations Partition Elimination – TABLE(s) Partition Elimination • Horizontal and vertical partitioning • Horizontal is traditional within file (range, hash, robin) • Vertical is across files (bucketing) • Immutable file system • Design according to your access patterns Enumerate all partitions filtering for today 30.ss 30.1.ss 29.ss 28.ss 29.1.ss Impressions … deen jp de PE across files + within each file
• 28.
  @Inpressions = SELECT *FROM searchDM.SML.PageView(@start, @end) AS PageView OPTION(LOWDISTINCTNESS=Query) ; // Q1(A,B) @Sessions = SELECT ClientId, Query, SUM(PageClicks) AS Clicks FROM @Impressions GROUP BY Query, ClientId ; // Q2(B) @Display = SELECT * FROM @Sessions INNER JOIN @Campaigns ON @Sessions.Query == @Campaigns.Query ; U-SQL Optimizations Partitioning – Minimize (re)partitions Input must be partitioned on: (Query) Input must be partitioned on: (Query) or (ClientId) or (Query, ClientId) Optimizer wants to partition only once But Query could be skewed Data Partitioning • Re-Partitioning is very expensive • Many U-SQL operators can handle multiple partitioning choices • Optimizer bases decision upon estimations Wrong statistics may result in worse query performance
• 29.
  // Unstructured (24hours daily log impressions) @Huge = EXTRACT ClientId int, ... FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif" ; // Small subset (ie: ForgetMe opt out) @Small = SELECT * FROM @Huge WHERE Bing.ForgetMe(x,y,z) OPTION(ROWCOUNT=500) ; // Result (not enough info to determine simple Broadcast join) @Remove = SELECT * FROM Bing.Sessions INNER JOIN @Small ON Sessions.Client == @Small.Client ; U-SQL Optimizations Partitioning - Cardinality Broadcast JOIN right? Broadcast is now a candidate. Wrong statistics may result in worse query performance Optimizer has no stats this is small...
• 31.
  // Bing impressions @Impressions= SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ; // Compute sessions @Sessions = REDUCE @Impressions ON Client, Market READONLY Market USING new Bing.SessionReducer(range : 30) ; // Users metrics @Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" ; // … Microsoft Confidential U-SQL Optimizations Predicate pushing – UDO pass-through columns
• 32.
  // Bing impressions @Impressions= SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ; // Compute page views @Impressions = PROCESS @Impressions READONLY Market PRODUCE Client, Market, Header string USING new Bing.HtmlProcessor() ; @Sessions = REDUCE @Impressions ON Client, Market READONLY Market USING new Bing.SessionReducer(range : 30) ; // Users metrics @Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" ; Microsoft Confidential U-SQL Optimizations Predicate pushing – UDO row level processors public abstract class IProcessor : IUserDefinedOperator { /// <summary/> public abstract IRow Process(IRow input, IUpdatableRow output); } public abstract class IReducer : IUserDefinedOperator { /// <summary/> public abstract IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output); }
• 33.
  // Bing impressions @Impressions= SELECT Client, Market, Html FROM searchDM.SML.PageView(@start, @end) AS PageView ; // Compute page views @Impressions = PROCESS @Impressions PRODUCE Client, Market, Header string USING new Bing.HtmlProcessor() ; // Users metrics @Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" && Header.Contains("microsoft.com") AND Header.Contains("microsoft.com") ; U-SQL Optimizations Predicate pushing – relational vs. C# semantics
• 35.
  // Bing impressions @Impressions= SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ; // Compute page views @Impressions = PROCESS @Impressions PRODUCE * REQUIRED ClientId, HtmlContent(Header, Footer) USING new Bing.HtmlProcessor() ; // Users metrics @Metrics = SELECT ClientId, Market, Header FROM @Sessions WHERE Market == "en-us" ; U-SQL Optimizations Column Pruning and dependencies C H M C H M C H M Column Pruning • Minimize I/O (data shuffling) • Minimize CPU (complex processing, html) • Requires dependency knowledge: • R(D*) = Input ( Output ) • Default no pruning • User code has to honor reduced columns A B C D E F G J KH I … M … 1000
• 36.
  • Use SELECTwith UDFs instead of PROCESS • Use User-defined Aggregators instead of REDUCE • Hint Cardinality if you use CROSS APPLY and it does chose the wrong plan • Avoid ORDER BY unless needed (OUTPUT, “Top N Rows”) • Learn to use Windowing Functions (OVER expression) • Use SQL.MAP and SQL.ARRAY instead of C# Dictionary and array
• 37.
  Multiple INSERTs intosame table • Generates separate file per insert in physical storage: • Can lead to performance degradation • Recommendations: • Try to avoid small inserts • Rebuild table after frequent insertions with: ALTER TABLE T REBUILD;
• 38.
  • • • • • •
• 39.
  Additional Resources  Blogs andcommunity page:  http://usql.io  http://blogs.msdn.com/b/visualstudio/  http://azure.microsoft.com/en-us/blog/topics/big-data/  https://channel9.msdn.com/Search?term=U-SQL#ch9Search  Documentation and articles:  http://aka.ms/usql_reference  https://azure.microsoft.com/en- us/documentation/services/data-lake-analytics/  https://msdn.microsoft.com/en-us/magazine/mt614251  ADL forums and feedback  http://aka.ms/adlfeedback  https://social.msdn.microsoft.com/Forums/azure/en- US/home?forum=AzureDataLake  http://stackoverflow.com/questions/tagged/u-sql