Optimising Queries - Series 1 Query Optimiser Architecture
Download as PPTX, PDF0 likes245 views
The document discusses the architecture of SQL Server's query optimizer, emphasizing its importance in maintaining server performance through efficient query execution. It outlines the phases of query processing, such as parsing, compilation, optimization, and execution, as well as strategies for optimizing queries and data access plans. Key topics include auto-parameterization, avoiding recompilation of queries, and techniques for diagnosing performance issues.
Optimising Queries - Series 1 Query Optimiser Architecture
- 1. www.dageop.com Optimising Queries ® OQ-01 Query Optimiser Architecture DR. SUBRAMANI PARAMASIVAM (MANI)
- 2. About me Dr. SubraMANI Paramasivam PhD., MCT, MCSE, MCITP, MCP, MCTS, MCSA CEO, Principal Consultant & Trainer @ DAGEOP (UK) Email: [email protected] Blog: http://dataap.org/blog Follow Us https://www.facebook.com/pages/YOUR-SQL-MAN-LTD/ http://www.youtube.com/user/YourSQLMAN https://twitter.com/dageop https://uk.linkedin.com/in/dageop Proud Sponsor • SQLBits • SQL Saturdays • MCT Summit • SQL Server Geeks Summit • Data Awareness Programme • Dageop’s Data Day ® www.DataAP.org SPEAKER
- 3. Contents • OQ-01 Query Optimiser Architecture • Phases • Strategies • Data access plans • Auto-parameterisation • Avoiding recompilation of queries www.dageop.com Optimizing Queries
- 4. OQ-01 Query Optimiser Architecture www.dageop.com Optimizing Queries
- 6. Optimising Queries • Optimising Query is very important task to maintain the server resources. • Most of the time server is hit by performance issues. • Clean the SQL Server Cache. • Well written T-SQL Queries will help to optimise. www.dageop.com Optimizing Queries
- 8. Introduction to Query Optimiser Architecture 2 Major Components in SQL Server •SQL Server Database Engine • Reading data between disk and memory •Relational Engine (Query Processor) • Accepts queries, analyse & executes the plan. www.dageop.com Optimizing Queries
- 9. Introduction to Query Optimiser Architecture • Analyse Execution Plans (Cannot consider every plan) • Estimate the cost of each plan • Select the plan with low cost www.dageop.com Optimizing Queries Plan1 Plan2 Plan3 Plan4 Plan5 Plan6 Plan7 Plan1 – 80% Plan2 – 60% Plan3 – 10% Plan5 – 90% Plan6 – 70% Plan3 – 10% Analyse Estimate Select COST X 2 Plan & itself
- 10. BETTER UNDERSTANDING of Query Optimiser is a must for both DBA & DEVELOPERS. Main Challenges are Cardinality & Cost Estimations www.dageop.com Optimizing Queries
- 11. Query Optimiser • 1st Part - Searching or enumerating candidate plans • 2nd Part - Estimates the cost of each physical operator in that plan (I/O, CPU, and memory) • This cost estimation depends mostly on the algorithm used by the physical operator, as well as the estimated number of records that will need to be processed (Cardinality Estimation). PRIMARY WAY to interact with Query Optimizer is through EXECUTION PLANS www.dageop.com Optimizing Queries
- 12. Query Optimiser • Execution Plan • Trees consisting of a number of physical operators • Physical operators • Index Scan • Hash Aggregate • result operator (root element in the plan) • Nested Loops Join • Merge Join • Hash Join • Algorithms • Can be saved as .sqlplan (Can be viewed in SSMS) www.dageop.com Optimizing Queries
- 13. Query Optimiser • Operators • Logical • Relational algebraic operation • Conceptually describes what operation needs to be performed • Physical • Implement the operation described by logical operators • Access columns, rows, index, table, views, calculations, aggregations, data integrity checks • 3 methods • Init() – Initializes a connection • GetNext() – 0 to many calls to get data • Close() – Some cleanup and close the connection. • Query Optimizer chooses efficient physical operator based on logical. www.dageop.com Optimizing Queries
- 14. Query Optimiser www.dageop.com Optimizing Queries Logical Physical Logical & Physical Bitmap Create Assert Aggregate Branch Repartition Bitmap Clustered Index Scan Cache Clustered Index Delete Clustered Index Scan Distinct Clustered Index Insert Clustered Index Update Distinct Sort Clustered Index Merge Collapse Distribute Streams Hash Match Compute Scalar Eager Spool Merge Join Concatenation Flow Distinct Nested Loops Cursor Full Outer Join Nonclustered Index Delete Inserted Scan Gather Streams Index Insert Log Row Scan Inner Join Index Spool Merge Interval Insert Nonclustered Index Update Index Scan Lazy Spool Online Index Insert Index Seek Left Anti Semi Join Parallelism Parameter Table Scan Left Outer Join RID Lookup Remote Delete Left Semi Join Stream Aggregate Remote Index Scan Partial Aggregate Table Delete Remote Index Seek Repartition Streams Table Insert Remote Insert Right Anti Semi Join Table Merge Remote Query Right Outer Join Table Spool Remote Scan Right Semi Join Table Update Remote Update Row Count Spool Segment Segment Repartition Sequence Union Sequence Project Update Sort Split Switch Table Scan Table-valued Function Top Window Spool
- 15. Query Optimiser • Invalid Plans • Removal of an index • Removal of a constraint, • Significant changes made to the contents of the database • SQL Server under memory pressure • Changing configuration options - max degree of parallelism • Discard the plan cache and new optimization generated www.dageop.com Optimizing Queries
- 16. Query Optimiser • Highly complex pieces of software • Even 30 years of research, still has technical challenges. HINTS • override the operations of the Query Optimizer • caution and only as a last option • Influence Query Optimizer to use certain plans www.dageop.com Optimizing Queries
- 17. Query Optimiser - Interaction www.dageop.com Optimizing Queries Query Optimizer Execution Plans Trees, Algorithm, Physical operator ACTUAL ESTIMATED Graphics, Text, XML format
- 20. Phases • Once we execute the SQL Statement, it will follow certain procedures from SQL Statement to Query Result www.dageop.com Optimizing Queries Parsing Query Compilation Query Optimization Query Execution
- 21. Phases • If the Query is already in the plan cache then it will not generate any new execution plan for that query. • Parsing: • The Query’s syntax will be validated and query is transformed in a tree. Checks the objects for its existence which are used in the query. • After the query validated, the final tree is formed. • Query Compilation: • Query Tree will be compiled here. www.dageop.com Optimizing Queries
- 22. Phases • Query Optimization • The query optimizer takes as input the compiled query tree generated in the previous step and investigates several access strategies before it decides how to process the given query. • It will analyse the query to find most efficient execution plan. • Query Execution • After the execution plan is generated, it is permanently stored and executed • Note: For Some statements, parsing and optimization can be avoided if the database engines know that there is only one viable plan. This is called trivial plan optimization. www.dageop.com Optimizing Queries
- 24. Strategies • As a DBA, we need to choose right strategy for each SQL statement like scripts should be well written before executing the SQL statement. • Statistics will be used to understand the performance of each query. • Enable Set Statistics IO before query run. It will display the following information • How many scans were performed • How many logical reads (reads in cache) were performed • How many physical reads (reads on disk) were performed • How many pages were placed in the cache in anticipation of future reads (read-ahead reads) www.dageop.com Optimizing Queries
- 25. Strategies • We can understand the query performance from statistics IO, If the query is good then the logical reads of the query result should be lower and few physical reads and scans. • Enable Set Statistics Time, it will display the execution time of the query. It purely depends on the total activity of the server. • Enable show execution plan, to see how the query performed. • These are things will helpful for choosing the right strategy. www.dageop.com Optimizing Queries
- 26. www.dageop.com Optimizing Queries Strategies Choose the right strategies based on below information from SQL Server
- 28. Data Access Plans www.dageop.com Optimizing Queries
- 29. Data Access Plans • If we need to optimize the data access plans then we need to start from creating files for database, Index on the tables and T SQL Statements. • Steps: • Organize the file groups and files • Apply partitioning in big tables • Create the appropriate indexes/covering indexes • Defragment the indexes • Identify inefficient TSQL • Diagnose performance problems www.dageop.com Optimizing Queries
- 30. Data Access Plans • Organize the file groups and files • Initially two files will be created while created a database (.mdf & .ldf). • .mdf file : Primary data file for each database. All system objects will be stored in this file, including the user defined objects if .ndf file is not there. • .ndf file: These are secondary data files, these are optional. These files will have user created objects. • .ldf file: These are the transaction log files. This could be one or more files for single database. • File Group: • Database files are logically grouped for better performance and improved administration on large databases. When a new SQL Server database is created, the primary file group is created and the primary data file is included in the primary file group. Also, the primary group is marked as the default group. www.dageop.com Optimizing Queries
- 31. Data Access Plans • To obtain the performance of the data access, Primary file group must be separate and it should be only for system objects. • Need to create one more file called secondary data file for user defined objects. • Separating the system objects will improve the performance enhance the ability to access tables in cases of serious data failures. • For frequently accessed tables containing indexes, put the tables and the indexes in separate file groups. This would enable reading the index and table data faster www.dageop.com Optimizing Queries
- 32. Data Access Plans • Apply partitioning in big tables • Table partitioning is nothing but splitting the large table into multiple small tables so that queries have to scan less amount for data retrieving. • Consider partitioning big fat tables into different file groups where each file inside the file group is spread into separate physical disks (so that the table spans across different files in different physical disks). This would enable the database engine to read/write data operations faster. • Partitioning is very much needed for history tables. • 2 types • Physical • Logical www.dageop.com Optimizing Queries
- 33. Data Access Plans • Create the appropriate indexes/covering indexes • Create Non-Clustered index on frequently used columns • Create index on column which is used for joining the tables • Index for foreign key columns • Create covering index for particular columns which are using frequently. • Defragment the indexes • Once index has created , it should maintain properly to avoid defragmentation. • Maintaining the index will lead to performance gain. www.dageop.com Optimizing Queries
- 34. Data Access Plans • Identify inefficient TSQL • Don’t use * in the select statements. Mention column names while retrieving the data. It will improve the performance of the query • Avoid Deadlocks between two objects. • Diagnose the performance issue • SQL Server has many tools to Monitor and diagnose the issues. • Accessing the data will be more easier. www.dageop.com Optimizing Queries
- 37. Auto-Parameterisation • SQL Server Query Optimizer might decide to parameterize some of the queries. • In this case the specific parameter will not make any impact on the execution plan. It will return the same execution plan. • In SQL Server 2005 forced parameterization has been introduced and is disabled by default and can be enabled in database level. • To differentiate from forced parameterization, auto-parameterization is also referred as simple parameterization. www.dageop.com Optimizing Queries
- 39. Avoiding Recompilation of Queries www.dageop.com Optimizing Queries
- 40. Avoiding Recompilation of Queries • In SQL Server 2005 recompiles the stored procedures, only the statement that causes recompilation is compiled, rather than the entire procedure. • Recompilation will occur in following ways, • On Schema change of objects • On Change of the SET options • On statistics change of tables. www.dageop.com Optimizing Queries
- 41. Avoiding Recompilation of Queries • On Schema Change of Objects • Adding and dropping column, constraints, index, indexed view and trigger. • On change of the SET options • When executing the stored procedure, the compiled plan is created and it will store the environment setting of a connection (SET OPTION) . • Recompilation will occur, if the stored procedure run on different environment and with different SET option then it will not use the existing plan which it is created first time. • On statistics change of tables • SQL server maintains a modification counter for each table and index. • If the counter values exceed the defined threshold, the previously create compiled plans is considered stale plan and new plan will be created. www.dageop.com Optimizing Queries
- 42. Avoiding Recompilation of Queries • Temporary table modification counter threshold is 6. Stored procedure will be recompiled when stored procedure create a temp table insert 6 or more rows into this table. • For Permanent table the counter threshold is 500. • We can increase the temp table counter threshold to 500 as same as permanent table. • Use table variable instead of Temporary table. www.dageop.com Optimizing Queries
- 44. Review Query Optimiser Architecture Phases Strategies Data access plans Auto-parameterisation Avoiding recompilation of queries www.dageop.com Optimizing Queries
- 45. Q & A www.dageop.com Optimizing Queries
- 46. ® www.dageop.com