Oracle Database In-Memory and the Query Optimizer
6 likes2,331 views
The document discusses the enhancements in Oracle Database related to in-memory storage and the query optimizer. It covers topics such as in-memory column stores, cost models, scanning techniques, and query transformations specifically designed to optimize performance for complex queries. The presentation highlights new features, control mechanisms, and the implications of using in-memory capabilities in Oracle databases.
Oracle Database In-Memory and the Query Optimizer
- 1. 2014 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA Oracle Database In-Memory and the Query Optimizer UKOUG Tech14, Liverpool (GB) Christian Antognini 11 December 2014 Oracle Database In-Memory and the Query Optimizer 1
- 2. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 2 @ChrisAntognini ■Senior principal consultant, trainer and partner at Trivadis in Zurich (CH) ■[email protected] ■http://antognini.ch ■Focus: get the most out of Oracle Database ■Logical and physical database design ■Query optimizer ■Application performance management ■Author of Troubleshooting Oracle Performance (Apress, 2008/2014) ■OakTable Network, Oracle ACE Director
- 3. 2014 © Trivadis 1.In-Memory Column Store 2.In-Memory Aggregation 11 December 2014 Oracle Database In-Memory and the Query Optimizer 3 AGENDA
- 4. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer In-Memory Column Store 4
- 5. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 5 Architecture Overview
- 6. 2014 © Trivadis ■Data is populated into the IMCS in chunks called IM Column Unit (IMCU) 11 December 2014 Oracle Database In-Memory and the Query Optimizer 6 IM Column Unit … C1 C2 C3 C1 C2 C3 C1 C2 C3
- 7. 2014 © Trivadis ■They are similar to Exadata storage indexes ■Do not store information about NULL values ■They are automatically created and maintained for each column inside the IMCS ■They are only stored in memory ■They keep track of the minimum and maximum values in each IMCU ■Their aim is to avoid accessing IMCUs that do not contain relevant data 11 December 2014 Oracle Database In-Memory and the Query Optimizer 7 IM Storage Indexes
- 8. 2014 © Trivadis ■The cost model has been enhanced to be IM aware ■It requires IM statistics ■The total cost is a combination of both non-IM costs and IM costs ■Enhancements are disabled if OPTIMIZER_INMEMORY_AWARE = FALSE or OPTIMIZER_FEATURES_ENABLE < 12.1.0.2 ■IMCS is not disabled ■Costs are based on the former cost model 11 December 2014 Oracle Database In-Memory and the Query Optimizer 8 IM-Aware Query Optimizer
- 9. 2014 © Trivadis ■The following statistics are used by the query optimizer: ■Number of IMCUs ■Number of blocks with data into the IMCUs ■Number of rows stored into the IMCUs ■Fraction of the rows stored into the IMCUs -Value between 0 and 1 ■Number of rows stored into the transaction journal ■IM statistics are computed on the fly during hard parses ■A cursors is not invalidated when the IM statistics it is based on changes 11 December 2014 Oracle Database In-Memory and the Query Optimizer 9 IM Statistics
- 10. 2014 © Trivadis ■Two operations are related to IM scans: ■TABLE ACCESS INMEMORY FULL ■MAT_VIEW ACCESS INMEMORY FULL ■Also used in case of Exadata storage ■IM scans are selected by the query optimizer regardless of whether data is actually stored into the IMCS ■The current behavior seems buggy to me 11 December 2014 Oracle Database In-Memory and the Query Optimizer 10 IM Scans
- 11. 2014 © Trivadis ■Pruning based on storage indexes ■Pruning based on compression metadata ■Dictionary-based compression algorithms store metadata that contain a list of distinct values for each column within IMCU ■Evaluation against compressed data is possible ■Except for CAPACITY LOW/HIGH ■Evaluation takes advantage of SIMD ■Execution plans provide a new type of predicate: INMEMORY ■Also used in case of predicates offloaded to Exadata storage 11 December 2014 Oracle Database In-Memory and the Query Optimizer 11 Predicate Evaluation Taking Place During IM Scans
- 12. 2014 © Trivadis ■The query optimizer can generate predicates that are evaluated during IM scans ■The generated predicates are used as explicit predicates 11 December 2014 Oracle Database In-Memory and the Query Optimizer 12 Generation of Implied Predicates
- 13. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 13 IM Scan Example SELECT * FROM t WHERE id = 1 OR id > 1 AND n < 42 ------------------------------------------- | Id | Operation | Name | ------------------------------------------- | 0 | SELECT STATEMENT | | |* 1 | TABLE ACCESS INMEMORY FULL| T | ------------------------------------------- 1 - inmemory("ID">=1 AND ("N"<42 AND "ID">1 OR "ID"=1)) filter("N"<42 AND "ID">1 OR "ID"=1) Implied predicate
- 14. 2014 © Trivadis ■The INMEMORY_QUERY initialization parameter enables or disables IM scans at the session or system level ■IM scans can also be controlled through the (NO_)INMEMORY hints ■The hints override the INMEMORY_QUERY parameter 11 December 2014 Oracle Database In-Memory and the Query Optimizer 14 Controlling IM Scans
- 15. 2014 © Trivadis ■Based on Bloom filters ■Similar concept than parallel joins in previous versions, but executed serially ■Filter created by the left input ■Filter used by the right input ■Only available for hash joins ■Only the probe input have to be “IM-enabled” ■The join is used to filter the false positives ■IM joins can be controlled through the (NO_)PX_JOIN_FILTER hints 11 December 2014 Oracle Database In-Memory and the Query Optimizer 15 IM Joins
- 16. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 16 IM Join Example SELECT * FROM t1, t2 WHERE t1.id = t2.id AND t1.n < 1E3 ------------------------------------------------ | Id | Operation | Name | ------------------------------------------------ | 0 | SELECT STATEMENT | | |* 1 | HASH JOIN | | | 2 | JOIN FILTER CREATE | :BF0000 | |* 3 | TABLE ACCESS FULL | T1 | | 4 | JOIN FILTER USE | :BF0000 | |* 5 | TABLE ACCESS INMEMORY FULL| T2 | ------------------------------------------------ 1 - access("T1"."ID"="T2"."ID") 3 - filter("T1"."N"<1E3) 5 - inmemory(SYS_OP_BLOOM_FILTER(:BF0000,"T2"."ID")) filter(SYS_OP_BLOOM_FILTER(:BF0000,"T2"."ID"))
- 17. 2014 © Trivadis ■Table expansion is a cost-based transformation introduced in 11.2 to leverage partially unusable indexes ■It has been made IM aware ■Useful in case not all partitions are “IM-enabled” ■It can be controlled through the (NO_)EXPAND_TABLE hints 11 December 2014 Oracle Database In-Memory and the Query Optimizer 17 IM Table Expansion
- 18. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 18 IM Table Expansion Example SELECT count(d) FROM t WHERE d BETWEEN :b1 AND :b2 --------------------------------------------------------------------------- | Operation | Name | Pstart| Pstop | --------------------------------------------------------------------------- | SELECT STATEMENT | | | | | SORT AGGREGATE | | | | | VIEW | VW_TE_1 | | | | UNION-ALL | | | | | PARTITION RANGE SINGLE | | 11 | 11 | | TABLE ACCESS INMEMORY FULL | T | 11 | 11 | | PARTITION RANGE SINGLE | | 12 | 12 | | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| T | 12 | 12 | | INDEX RANGE SCAN | I | 12 | 12 | --------------------------------------------------------------------------- The IM scan is only enabled for the partition 4
- 19. 2014 © Trivadis ■The plan hash value is independent from the kind of FTS ■TABLE ACCESS FULL ■TABLE ACCESS STORAGE FULL ■TABLE ACCESS INMEMORY FULL ■Features like SQL plan baselines do not break when IMCS is enabled ■The “best” execution plan can change 11 December 2014 Oracle Database In-Memory and the Query Optimizer 19 Plan Hash Value
- 20. 2014 © Trivadis ■_INMEMORY_QUERY_SCAN controls IM scans (TRUE) ■_OPTIMIZER_INMEMORY_ACCESS_PATH controls whether the query optimizer costs for IM (TRUE) ■_OPTIMIZER_INMEMORY_BLOOM_FILTER controls IM joins (TRUE) ■_OPTIMIZER_INMEMORY_GEN_PUSHABLE_PREDS controls the generation of implied predicates (TRUE) ■_OPTIMIZER_INMEMORY_TABLE_EXPANSION controls IM table expansion (TRUE) 11 December 2014 Oracle Database In-Memory and the Query Optimizer 20 Undocumented Parameters
- 21. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer In-Memory Aggregation 21
- 22. 2014 © Trivadis ■To optimize queries against a star schema, the query optimizer should do the following: ■Start evaluating each dimension table that has restrictions on it ■Assemble a list with the resulting dimension keys ■Use this list to extract the matching rows from the fact table ■This approach cannot be implemented with regular joins ■The query optimizer can join only two data sets at one time ■Joining two dimension tables leads to a Cartesian product ■To solve this problem, Oracle Database implements two query transformations: ■Star transformation ■Vector transformation (new in 12.1.0.2 – requires the In-Memory option) 11 December 2014 Oracle Database In-Memory and the Query Optimizer 22 The Star Schema Challenge
- 23. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 23 Star Transformation vs. Vector Transformation Selectivity Elapsed Time
- 24. 2014 © Trivadis ■In the documentation Oracle refers to it as In-Memory Aggregation ■It isn’t only about aggregation ■It requires the In-Memory option because it takes advantage of some of its features ■INMEMORY_SIZE must be greater than 0 ■The query optimizer considers it when ■equijoins are used ■an aggregate function is applied to a column of the fact table ■the query contains a GROUP BY clause -ROLLUP, CUBE and GROUPING SETS are not yet supported 11 December 2014 Oracle Database In-Memory and the Query Optimizer 24 Vector Transformation (1)
- 25. 2014 © Trivadis ■It’s a cost-based query transformation ■It can be controlled through the (NO_)VECTOR_TRANSFORM hints ■As with the star transformation, sometimes is not possible to force it ■It introduces new row source operations ■KEY VECTOR CREATE ■KEY VECTOR USE ■VECTOR GROUP BY 11 December 2014 Oracle Database In-Memory and the Query Optimizer 25 Vector Transformation (2)
- 26. 2014 © Trivadis 1.For every dimension table with a filter on it the following operations take place ■Access table and filter data ■Create key vector ■Aggregate data ■Create temporary table 2.Access the fact table through a FTS and filter data by applying the key vectors ■On Exadata the filter is not yet offloaded 3.Aggregate data with either a vector or hash aggregation 4.Join data from the fact table to temporary tables 5.Join dimension tables without a filter on them 11 December 2014 Oracle Database In-Memory and the Query Optimizer 26 Vector Transformation – Processing Steps
- 27. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 27 Example SELECT c.customer_class, sum(o.order_total) FROM orders o, customers c, warehouses w WHERE o.customer_id = c.customer_id AND o.warehouse_id = w.warehouse_id AND c.dob BETWEEN :b1 AND :b2 AND w.warehouse_name BETWEEN :b3 AND :b4 GROUP BY c.customer_class customers orders warehouses
- 28. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 28 Execution Plan – Access Dimension Tables, Create Key Vectors, and Create Temporary Tables ------------------------------------------------------------------------- | Id | Operation | Name | ------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | 1 | TEMP TABLE TRANSFORMATION | | | 2 | LOAD AS SELECT | SYS_TEMP_0FD9DE69C_28565764 | | 3 | VECTOR GROUP BY | | | 4 | KEY VECTOR CREATE BUFFERED | :KV0000 | |* 5 | TABLE ACCESS STORAGE FULL | CUSTOMERS | | 6 | LOAD AS SELECT | SYS_TEMP_0FD9DE69D_28565764 | | 7 | VECTOR GROUP BY | | | 8 | HASH GROUP BY | | | 9 | KEY VECTOR CREATE BUFFERED | :KV0001 | |* 10 | TABLE ACCESS STORAGE FULL | WAREHOUSES | ...
- 29. 2014 © Trivadis 11 December 2014 Oracle Database In-Memory and the Query Optimizer 29 Execution Plan – Access Fact Table by Applying Key Vectors, and Join Temporary Tables ... | 11 | HASH GROUP BY | | |* 12 | HASH JOIN | | | 13 | MERGE JOIN CARTESIAN | | | 14 | TABLE ACCESS STORAGE FULL | SYS_TEMP_0FD9DE69D_28565764 | | 15 | BUFFER SORT | | | 16 | TABLE ACCESS STORAGE FULL | SYS_TEMP_0FD9DE69C_28565764 | | 17 | VIEW | VW_VT_72AE2D8F | | 18 | VECTOR GROUP BY | | | 19 | HASH GROUP BY | | | 20 | KEY VECTOR USE | :KV0001 | | 21 | KEY VECTOR USE | :KV0000 | |* 22 | TABLE ACCESS STORAGE FULL| ORDERS | ------------------------------------------------------------------------- ... 22 - filter(SYS_OP_KEY_VECTOR_FILTER("O"."CUSTOMER_ID",:KV0000) AND SYS_OP_KEY_VECTOR_FILTER("O"."WAREHOUSE_ID",:KV0001))
- 30. 2014 © Trivadis Core Messages ■In-Memory Column Store ■The query optimizer is IM aware ■In-Memory Aggregation ■Very interesting concept, complements the star transformation 11 December 2014 Oracle Database In-Memory and the Query Optimizer 30
- 31. 2014 © Trivadis Questions and answers ... BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA Christian Antognini Principal Senior Consultant [email protected] 11 December 2014 Oracle Database In-Memory and the Query Optimizer 31