SQL Server Deep Drive
Download as PPTX, PDF•1 like•166 views
This document provides an overview of SQL Server internals including: - The query processing pipeline including parsing, optimizing, and executing queries. - How the optimizer evaluates and chooses the most efficient query execution plan. - The role of indexes, statistics, and parallelism in query optimization. - Transaction logging and the different SQL Server recovery models. - Key SQL Server memory structures like the buffer pool and query plan cache. - Threading and scheduling within the SQL Server query processor.
SQL Server Deep Drive
- 1. SQL Server Deep Dive IT Talk Denis Reznik Data Architect at Intapp, Inc. Microsoft Data Platform MVP
- 2. About Me • Kyiv, Ukraine • Data Architect at Intapp, Inc. • Microsoft Data Platform MVP • PASS Regional Mentor, CEE • Ukrainian Data Community Kyiv Co-Founder • Co-author of “SQL Server MVP Deep Dives vol. 2”
- 3. How We See SQL Server 3 | Some Magic Query Result Set
- 5. SQL Server Network Interface
- 6. SQL Server Protocols • Shared Memory • TCP/IP • Named Pipes • VIA (Deprecated)
- 7. TDS Endpoints TCP/IP Shared Memory Named Pipes VIA DAC TCP/IP Note: TDS – Tabular Data Stream
- 8. Connection • Connection can be Succeed or Not • https://www.connectionstrings.com/ • Connection string contains settings for Connection Timeout (or Timeout) • Connection Timeout just limit the time for establishing a connection • Does not limit the query execution time • sys.dm_exec_connections • sys.dm_exec_sessions • sys.dm_exec_requests • sys.dm_os_tasks • … down to the bottom of SQLOS 8 |
- 10. Parser Algebraizer Optimizer Executor Query Processing Plan Cache
- 11. Algebraizing • Define types for every node in ParseTree • Check that objects are exist and available in current execution context • Extract literals from the query • e.g. WHERE User_Id = 1, 1 is a literal • Returns BoundTree (QueryTree) • Logical representation of the query
- 12. • Goal – get the optimal execution plan • “Optimal” = Plan with the smallest cost • Cost = sum_of_all_operators_costs(I/O + CPU + memory_consumption) Optimizer Query Tree Pre-Optimization Search for trivial plan Load statistics (simple) Simplification – syntax transformation Optimization
- 13. Heap 1 .. 100 100 .. 1k 5K .. 6K 1K .. 5K 6K .. 7K 15K .. 21K 12K .. 15K 10K .. 11K 21K .. 22K 22K .. 41K 9K .. 10K 41K .. 51K 7K .. 8K 8K .. 9K 71K .. 1M 51K .. 71K 1M .. 2M 2M .. 3M
- 14. Clustered Index … … 1 .. 1M 1 .. 2K 2K+1 .. 4K 1M-2K .. 1M 1 .. 300 301 .. 800 801 .. 1,5K 1,5K+1 .. 2K
- 15. Index Seek … … 1 .. 1M 1 .. 2K 2K+1 .. 4K 1M-2K .. 1M 1 .. 300 301 .. 800 801 .. 1,5K 1,5K+1 .. 2K SELECT * FROM Users WHERE Id = 523
- 16. Index Scan … … 1 .. 1M 1 .. 2K 2K+1 .. 4K 1M-2K .. 1M 1 .. 300 301 .. 800 801 .. 1,5K 1,5K+1 .. 2K SELECT * FROM Users
- 17. Index Range Scan … … 1 .. 1M 1 .. 2K 2K+1 .. 4K 1M-2K .. 1M 1 .. 300 301 .. 800 801 .. 1,5K 1,5K+1 .. 2K SELECT * FROM Users WHERE Id BETWEEN 700 AND 1700
- 18. Non-Clustered Index … A .. Z A .. C C .. K X .. Z … 1 .. 1M 1 .. 2K 2K+1 .. 4K 1M-2K .. 1M SELECT * FROM Users WHERE Name = 'Donald Trump' 1 .. 2K 2K .. 4K 1M-2K .. 1M Clustered Index (Id) Non-Clustered Index (Name) Heap
- 19. Optimizer Phase 0 Load Statistics Exploration – Plan Alternatives Estimated cost < 0.2 Return Transaction Processing Plan Optimization
- 20. Statistics 500 1000 10 1200 800 1 800 2000 2800 4500 5400 SELECT * FROM Users WHERE Id BETWEEN 2100 AND 2500 SELECT * FROM Users WHERE Id BETWEEN 200 AND 5000
- 21. Query Plan Alternatives SELECT * FROM Users u INNER JOIN Posts p ON u.Id = p.OwnerUserId WHERE u.DisplayName = 'John Snow' Users Posts Posts Users
- 22. Query Plan Alternatives • 1 Table – 1 option • 2 Tables – 2 options • 3 Tables – 6 options • 4 Tables – 24 options • … • 10 Tables – 3628800 options • 1 Table – 1! • 2 Tables – 2! • 3 Tables – 3! • 4 Tables – 4! • … • 10 Tables – 10!
- 23. Optimizer Phase 1 Next set of query rules Estimate parallel plan (estimation processed twice) • Max Degree of Parallelism • Cost Threshold for Parallelism Estimated cost < 1.0 Returns Quick Plan Optimization
- 24. Parallel Query Execution • Amdal’s Law Thread 1 Thread 2 Thread 3 Thread 4 1s2s
- 25. Optimizer Phase 2 Full Set of Optimization Rules Indexed views Returns Full Query Plan • All planned checks were done • Good Enough Plan was Found • Timeout • Memory Limit Exceeded Optimization
- 26. Executing • Execute query using its Query Plan • Request data from the Storage Engine
- 27. Parameter Sniffing Exec SomeProc @p Optimizer SQL Server Cache Procedure cache@p Plan is cached for the first value of Procedure cache
- 29. Storage Engine
- 30. Buffer Pool Data Cache Database File Access Methods Buffer Manager Data Selection ? What if I load 2GB value from the disk and read this data using NOLOCK?
- 31. Data Modification Buffer Pool Data Cache Database File Access Methods Buffer Manager Transaction Log Log Manager Log Buffer ? ? Background ProcessesLock Manager ? 3 cases
- 32. Transaction Log Autogrow Log Backup or CHECKPOINT
- 33. FULL Recovery Model • Log all transactions • Database can be restored to any point of time • Highly recommended for critical data • Transaction truncation occurs after log backup
- 34. BULK-LOGGED Model • Log all transactions • Minimally-Logged operations are logged in another way • Database can’t be restored to any point of time • Can be temporary interchanged with FULL recovery model to do Minimally-Logged operations more effectively • Transaction truncation occurs after log backup
- 35. SIMPLE Recovery Model • Log all transactions like in BULK-LOGGED recovery model • Transaction truncation occurs after CHECKPOINT
- 36. DEMO Log Shrink
- 37. Lock Types - Shared S S X
- 38. Lock Types - Exclusive X X S
- 39. Lock Types - Update U U S SX
- 40. Lock Types – Intent locks S ISIS
- 41. ID City 1 Kyiv 2 Kharkiv 3 Dnipropetrovsk BEGIN TRAN UPDATE Users SET CityId = 2 WHERE Id = 4 UPDATE City SET Name = 'Dnipro' WHERE Id = 3 ID User City_Id 1 Tony Stark 2 2 John Smith 2 3 Clark Kent 2 4 Ivan Vanko 3 Classic Deadlock X BEGIN TRAN UPDATE City SET Name = 'Dnipro' WHERE Id = 3 UPDATE Users SET CityId = 2 WHERE Id = 4 X DEADLOCK!
- 42. DEMO Deadlock
- 43. SQLOS
- 44. CPU – Cooperative Multitasking 0 CPUsQueries 1 2 3 Time Quantum Length for each Scheduler is 4 ms
- 45. Windows Memory Management 0001 0010 0101 0000 1001 0101 0011 1010 1001 1110 0010 0110 0001 0010 0101 0000 1001 0101 0011 1010 1001 Physical Memory (RAM) Page File VirtualMemory Virtual Memory Manager SQL Server Browser Word
- 46. Data Cache Query Cache Locks Memory Memory Grants Some Small Caches External Memory Buffer Pool SQL Server Memory Consumers
- 47. Network Bandwidth TCP Packages TCP Packages TCP Packages Bandwidth TCP Packages
- 48. Query Execution Thread Pool Running Suspended Runnable Sleeping Scheduler Logical CPU Worker (Thread) Connection
- 50. Summary 50 |
- 51. Thank You! IT Talk Denis Reznik Twitter: @denisreznik Email: [email protected] Blog: http://reznik.uneta.com.ua Facebook: https://www.facebook.com/denis.reznik.5 LinkedIn: http://ua.linkedin.com/pub/denis-reznik/3/502/234