Oracle in-Memory Column Store for BI
2 likes1,361 views
The document discusses the Oracle In-Memory Column Store, emphasizing its role in enhancing analytic query performance within both OLTP and BI databases. It outlines the architecture of in-memory column storage, including the advantages of dual-format support for analytics while maintaining existing row stores and the associated background population processes. Key features include optimization for compression, storage indexing, and asynchronous updates to facilitate efficient querying without significant overhead on transactional workloads.
Oracle in-Memory Column Store for BI
- 1. Oracle In-Memory Column Store Infrastructure at your Service. Oracle In-Memory Column Store for BI
- 2. Oracle In-Memory Column Store 05.10.2015 Infrastructure at your Service. About me Page 2 Franck Pachot Senior consultant Oracle Technology Leader Mobile +41 79 963 27 22 [email protected] www.dbi-services.com
- 3. Oracle In-Memory Column Store Experts At Your Service > 40 specialists in IT infrastructure > Certified, experienced, passionate Based In Switzerland > 100% self-financed Swiss company > Over CHF 6 mio. turnover Leading In Infrastructure Services > More than 100 customers in CH, D, & F > Over 35 SLAs dbi FlexService contracted 05.10.2015 dbi services Who we are Page 3
- 4. Oracle In-Memory Column Store 1. Analytic queries 2. Row store and Column store 3. In-Memory population 4. Performance 5. How to use it 05.10.2015 Agenda Page 4
- 5. Oracle In-Memory Column Store Datawarehouse > Ad-hoc queries aggregating data from lot of rows > You offload it to another database (designed for full scans, joins, sorts,…) > You design it for reporting (STAR schema, bitmap indexes, compression) > Queries reading lot of rows filtered by few dimensions are analytic queries OLTP > 3NF data model with primary keys and foreign keys > You insert rows (customer information, order lines,…) > You retrieve rows (by primary key, navigating through joins) > Queries not accessing from PK or selective index are analytic queries Where are your analytic queries? 05.10.2015 Page 5 Analytic Queries
- 6. Oracle In-Memory Column Store On your Order Entry application > You want to highlight good customers (more than n ordered in prev. days) > You want to show other articles related with the order On your CRM, ERP > You want to search by multicriteria screen > You want to calculate some real-time indicators Real-time Business Intelligence …are you executing BI queries on your OLTP database? Examples in OLTP 05.10.2015 Page 6 Analytic Queries
- 7. Oracle In-Memory Column Store In Oracle, you don’t need to offload reporting: it’s only better Oracle READS do not lock anything > It’s the Multi Versioning Consistency Model > You can QUERY on the operational OLTP database > Access through secondary indexes partitioning, full scan But it’s still better to offload to a BI database > Adding too many indexes slows DML > Bitmap indexes are not suited for DML > Memory sizing is different (High SGA for OLTP, High PGA for reporting) > Availability requirements are not the same > Keep more historical data that is purged on operation database Where are you doing analytic queries? 05.10.2015 Page 7 Analytic Queries
- 8. Oracle In-Memory Column Store In-Memory Index Access Full Table Scan Analytic queries > It’s not: ‘give me all columns for few specific rows’ > It’s: ’give some columns from all rows that match filter on few columns’ How are you doing analytic queries? 05.10.2015 Page 8 Analytic Queries SELECT Some columns Exadata SmartScan •Projection Offloading Covering indexes READ All Rows Table Full Scan •Multiblock read •Parallel Query Index Fast Full Scan FILTER Few columns Exadata SmartScan •Predicate Offloading •Storage index Index access •Covering index •Bitmap index Full Table Scan on In-Memory Column Store
- 9. Oracle In-Memory Column Store 1. Analytic queries 2. Row store and Column store 3. In-Memory population 4. Performance 5. How to use it 05.10.2015 Agenda Page 10
- 10. Oracle In-Memory Column Store Full scan 05.10.2015 Page 11 Row store and Column store EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO 7369 SMITH CLERK 7902 17-DEC-80 800 20 7499 ALLEN SALESMAN 7698 20-FEB-81 1600 300 30 7521 WARD SALESMAN 7698 22-FEB-81 1250 500 30 7566 JONES MANAGER 7839 02-APR-81 2975 20 7654 MARTIN SALESMAN 7698 28-SEP-81 1250 1400 30 7698 BLAKE MANAGER 7839 01-MAY-81 2850 30 7782 CLARK MANAGER 7839 09-JUN-81 2450 10 7788 SCOTT ANALYST 7566 19-APR-87 3000 20 7839 KING PRESIDENT 17-NOV-81 5000 10 7844 TURNER SALESMAN 7698 08-SEP-81 1500 0 30 7876 ADAMS CLERK 7788 23-MAY-87 1100 20 7900 JAMES CLERK 7698 03-DEC-81 950 30 7902 FORD ANALYST 7566 03-DEC-81 3000 20 7934 MILLER CLERK 7782 23-JAN-82 1300 10 SELECT ENAME FROM EMP WHERE SAL>=3000 > We read all blocks, with all column values
- 11. Oracle In-Memory Column Store Index scan 05.10.2015 Page 13 Row store and Column store EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO 7369 SMITH CLERK 7902 17-DEC-80 800 20 7499 ALLEN SALESMAN 7698 20-FEB-81 1600 300 30 7521 WARD SALESMAN 7698 22-FEB-81 1250 500 30 7566 JONES MANAGER 7839 02-APR-81 2975 20 7654 MARTIN SALESMAN 7698 28-SEP-81 1250 1400 30 7698 BLAKE MANAGER 7839 01-MAY-81 2850 30 7782 CLARK MANAGER 7839 09-JUN-81 2450 10 7788 SCOTT ANALYST 7566 19-APR-87 3000 20 7839 KING PRESIDENT 17-NOV-81 5000 10 7844 TURNER SALESMAN 7698 08-SEP-81 1500 0 30 7876 ADAMS CLERK 7788 23-MAY-87 1100 20 7900 JAMES CLERK 7698 03-DEC-81 950 30 7902 FORD ANALYST 7566 03-DEC-81 3000 20 7934 MILLER CLERK 7782 23-JAN-82 1300 10 SELECT ENAME FROM EMP WHERE SAL>=3000 > We read full blocks, with all column values SAL 800 950 1100 1250 1250 1300 1500 1600 2450 2850 2975 3000 3000 5000 SAL 800 950 1100 1250 1250 1300 1500 1600 2450 2850 2975 3000 3000 5000 SAL 800 950 1100 1250 1250 1300 1500 1600 2450 2850 2975 3000 3000 5000 SAL 800 950 1100 1250 1250 1300 1500 1600 2450 2850 2975 3000 3000 5000
- 12. Oracle In-Memory Column Store In-Memory scan 05.10.2015 Page 15 Row store and Column store SELECT ENAME FROM EMP WHERE SAL>=3000 > We read and process vectors of columns EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO 7369 SMITH CLERK 7902 17-DEC-80 800 20 7499 ALLEN SALESMAN 7698 20-FEB-81 1600 300 30 7521 WARD SALESMAN 7698 22-FEB-81 1250 500 30 7566 JONES MANAGER 7839 02-APR-81 2975 20 7654 MARTIN SALESMAN 7698 28-SEP-81 1250 1400 30 7698 BLAKE MANAGER 7839 01-MAY-81 2850 30 7782 CLARK MANAGER 7839 09-JUN-81 2450 10 7788 SCOTT ANALYST 7566 19-APR-87 3000 20 7839 KING PRESIDENT 17-NOV-81 5000 10 7844 TURNER SALESMAN 7698 08-SEP-81 1500 0 30 7876 ADAMS CLERK 7788 23-MAY-87 1100 20 7900 JAMES CLERK 7698 03-DEC-81 950 30 7902 FORD ANALYST 7566 03-DEC-81 3000 20 7934 MILLER CLERK 7782 23-JAN-82 1300 10
- 13. Oracle In-Memory Column Store Column Store is only In Memory > No persistence of the column store > Data is already persisted in the row store > Main reasons: > don’t have to design new block format, > manage recovery, etc. In-Memory is focused on analytic queries > No index, No IOT, no out-of-line LOB,… In-Memory is transparent for application > The DBA assigns physical memory to the column store > The data modeler defines which tables are populated in memory (and when – on first read or at instance startup, with priority) > The CBO chooses the right store to access > No need for denormalization, new indexes, materialized views,… Oracle choices about In-Memory implementation 05.10.2015 Page 18 Row store and Column store
- 14. Oracle In-Memory Column Store Oracle 12.1.0.2 introduces new columnar format > New format does NOT replace existing row format > Standard memory pools keeps rows format and Oracle block size structure Dual format architecture 05.10.2015 Page 19 Row store and Column store Table EMP EMPNO ENAME JOB 7902 FORD ANALYST … … … EMPNO ENAME JOB 7902 FORD ANALYST 7903 ALLAN CLERK 7904 BLAKE SALESMAN … … … EMP – Buffer cache EMPNO 7902 7903 7904 … JOB ANALYST CLERK SALESMAN … ENAME FORD ALLAN BLAKE … EMP – Column Store population update
- 15. Oracle In-Memory Column Store Row store is still there > Well proven mecanisms for: > Buffer cache algorithms, cache fusion RAC > ACID: transaction tables, ITL, global cache, … > ACID: undo, multi versioning concurency system, locks > ACID: redo log, log writer, well known recovery mecanism Column Store comes in addition to it > Is populated in background > Is maintained asynchronously > But improves performance when populated > Can fallback to buffer cache if not available/up-to-date > Is a shared nothing architecture when in RAC > Is not persistent: has to be populated when instance starts Dual format architecture 05.10.2015 Row store and Column store
- 16. Oracle In-Memory Column Store You can use In-Memory On your BI database > Real-time replication > Golden Gate > Dbvisit replicate > Add In-Memory to the BI database > Because bitmap indexes not possible with DML You can use In-Memory On your OLTP database > Add In-Memory directly to OLTP > It is a second store > No need to > Replicate > Denormalize > Index differently Dual format architecture 05.10.2015 Row store and Column store replication OLTP OLTPBI
- 17. Oracle In-Memory Column Store 1. Analytic queries 2. Row store and Column store 3. In-Memory population 4. Performance 5. How to use it 05.10.2015 Agenda Page 22
- 18. Oracle In-Memory Column Store In-Memory Column Store is a new static pool in SGA > In addition to the buffer cache (not a substitute) > Populated and maintained asynchronously It is not a cache > Once populated it stays there (no LRU algorithms) Columnar storage – Dedicated pool 05.10.2015 Page 23 In-Memory Column Store in the SGA System Global Area (SGA) Shared Pool Buffer Cache In-Memory Library cache block block block block SMU pool IMCU pool block block … CU CU CU CU CU row store column store
- 19. Oracle In-Memory Column Store Population is done in background Population on demand (at first access): > INMEMORY PRIORITY NONE Population on startup (starting with most critical) > INMEMORY PRIORITY LOW/MEDIUM/HIGH/CRITICAL On demand or on startup 05.10.2015 Page 25 In-Memory Column Store Population PID USER VIRT RES S %CPU %MEM TIME+ COMMAND 2933 oracle 1419732 92376 R 97.0 9.1 0:09.72 ora_w001_DEMO 7349 oracle 1413504 171016 S 1.0 16.8 0:11.31 oracleDEMO (LOCAL=NO) 2143 oracle 1395876 9644 S 0.0 0.9 0:02.45 ora_pmon_DEMO 2165 oracle 1403104 257476 S 0.0 25.3 0:13.00 ora_dbw0_DEMO 2167 oracle 1396520 9720 S 0.0 1.0 3:25.30 ora_lgwr_DEMO 2171 oracle 1397784 57776 S 0.0 5.7 0:01.15 ora_smon_DEMO 2203 oracle 1409236 90868 S 0.0 8.9 0:05.72 ora_w000_DEMO 2207 oracle 1402960 133628 S 0.0 13.1 0:11.61 ora_imco_DEMO
- 20. Oracle In-Memory Column Store By default, rows are DISTRIBUTE to all instances > By partition (supposes equal distribution), by rowid On Oracle engineered systems, can be DUPLICATE > Ensures high availiability of IMCS in case of node failure > Exadata or ODA X-5 (infiniband) > No cache fusion: sends only messages to trigger repopulation In-Memory in RAC > Makes sense only with Parallel Query and Auto DOP > It is supposed to be service aware > Currently only for on demand population > on startup population probably in 12.2 Population in RAC 05.10.2015 Page 26 In-Memory Column Store Population 147 369 258 13467 235689 124578 13579 2468
- 21. Oracle In-Memory Column Store IM population needs CPU > Reads blocks from disk or buffer cache > Need to read rows in order -> follow chained rows DML makes IM become stale, and IM scan is slower > Trickle repopulates slowly and constantly > Full repopulation is done when IMCU staleness reaches threshold > Default probably too high. Do you want to: > give all resources to population in order to have IMCS quickly or > let it populate slowly in the background keeping CPU for non-IM activity? Population 05.10.2015 Page 27 In-Memory Column Store Population NAME VALUE ------------------------------------------- ----- inmemory_max_populate_servers 2 default=CPU_COUNT/2 inmemory_trickle_repopulate_servers_percent 1
- 22. Oracle In-Memory Column Store DML must update the column store > In addition to the row store (buffer cache + redo logging) > IMCS is updated asyncronously > Updates are going to transaction log Isn’t it an overhead for OLTP ? And the DML ? 05.10.2015 Page 28 In-Memory Column Store Population No, OLTP is faster with In-Memory: > You can drop some indexes used only for analytics > And maintaining IMCS is faster than maintaining 2 or 3 indexes > But keep indexes on PK, FK, and those that you need before IMCS population is completed > and have enough CPU for re-population
- 23. Oracle In-Memory Column Store 1. Analytic queries 2. Row store and Column store 3. In-Memory population 4. Performance 5. How to use it 05.10.2015 Agenda Page 29
- 24. Oracle In-Memory Column Store Run-Length Encoding (RLE) > Columnar storage is the best suited for compression of repeated values (like HCC) > CPU can work in compressed values > Compression Unit are indexed (like Exadata Storage Index) Columnar Storage offers efficient ways to lower the RAM to CPU transfer which is now the new bottleneck (as we have no disk I/O) Columnar Compression 05.10.2015 In-Memory performance JOB CLERK SALESMAN SALESMAN SALESMAN SALESMAN MANAGER MANAGER ANALYST PRESIDENT SALESMAN CLERK CLERK ANALYST ANALYST JOB CLERK SALESMAN ‘ ‘ ‘ MANAGER ‘ ANALYST PRESIDENT SALESMAN CLERK ‘ ANALYST ‘ RLE Page 30
- 25. Oracle In-Memory Column Store Compression Unit are compressed by default > Filtering is done on compressed values. > Decompression occurs only for returned values > Compression occurs at population Levels of compression > NO MEMCOMPRESS – Compression disabled > MEMCOMPRESS FOR DML > MEMCOMPRESS FOR QUERY LOW > MEMCOMPRESS FOR QUERY HIGH > MEMCOMPRESS FOR CAPACITY LOW > MEMCOMPRESS FOR CAPACITY HIGH – Save more space Columnar Compression 05.10.2015 Page 31 In-Memory performance
- 26. Oracle In-Memory Column Store Storage indexes allows IMCU pruning > Check predicate against min/max > Skip IMCU that doesn’t match Many predicate can be filtered > Equality, range, in-list for example Reduce again volume of data to process > Eliminate unnecessary IMCUs Example: Find employees with salary between 5000 and 8000 Storage Index 05.10.2015 Page 32 In-Memory performance SALARY … … … … SALARY … … … … SALARY … … … … min: 4000 max: 6000 min: 7000 max: 9000 min: 14000 max: 20000 Has rows Has rows No rows here
- 27. Oracle In-Memory Column Store Storage indexes can skip filtering > Skipped IMCUs are not evaluated > No need to evaluate filter when all rows match Not only min/max > Also hash of distinct values Example: Find employees with salary between 5000 and 9000 Storage Index 05.10.2015 Page 33 In-Memory performance SALARY … … … … SALARY … … … … SALARY … … … … Has rows Skip filtering Has rows Needs filtering No rows here min: 4000 max: 6000 min: 7000 max: 9000 min: 14000 max: 20000
- 28. Oracle In-Memory Column Store 1. HASH JOIN reads the first input to build the hash table 2. Bloom Filter :BF0000 is created (filters but allows false positive) 3. Bloom filter used to filter while reading second input In-Memory enables Bloom Filter on serial statements > Join optimization introduced in Oracle 10g for Parallel Query Joins ? 05.10.2015 Page 34 In-Memory performance 11 - inmemory(SYS_OP_BLOOM_FILTER(:BF0000 filter(SYS_OP_BLOOM_FILTER(:BF0000
- 29. Oracle In-Memory Column Store Columnar format allows sending vector to CPU > SIMD vector used to compare multiple values in one CPU cycle Vector processing 05.10.2015 Page 35 In-Memory performance JOB CLERK SALESMAN SALESMAN SALESMAN SALESMAN MANAGER MANAGER ANALYST PRESIDENT SALESMAN CLERK CLERK vectorregister JOB CLERK SALESMAN SALESMAN SALESMAN SALESMAN MANAGER MANAGER ANALYST PRESIDENT SALESMAN CLERK CLERK JOB=MANAGER vectorregister
- 30. Oracle In-Memory Column Store 1. Analytic queries 2. Row store and Column store 3. In-Memory population 4. Performance 5. How to use it 05.10.2015 Agenda Page 37
- 31. Oracle In-Memory Column Store One parameter to enable In-Memory > Pool is static, database restart is required > Don’t forget to increase SGA_TARGET How to put table in memory? > Exclude some columns: > Choose compression > Choose population Basics 05.10.2015 Page 38 How to use In-Memory alter table SH.SALES INMEMORY; alter system set INMEMORY_SIZE=7G scope=spfile; shutdown immediate startup alter table SH.SALES INMEMORY NO INMEMORY(tax_region); alter table SH.SALES INMEMORY MEMCOMPRESS FOR CAPACITY LOW; alter table SH.SALES INMEMORY PRIORITY CRITICAL;
- 32. Oracle In-Memory Column Store Check In-Memory usage Check segment stored in Colum Store > When not enough memory Oracle just stop loading objects In-Memory advisor > Need Diagnostic Pack + Tuning Pack (SQL Performance Analyzer) Monitor In-Memory 05.10.2015 Page 41 How to use In-Memory SQL> select * from V$INMEMORY_AREA; POOL ALLOC_BYTES USED_BYTES POPULATE_STATUS CON_ID --------- ----------- ---------- --------------- ---------- 1MB POOL 5149556736 11534336 DONE 0 64KB POOL 1275068416 0 DONE 0 SQL> select owner, segment_name name, inmemory_size, populate_status pop_status, bytes, bytes_not_populated from V$IM_SEGMENTS OWNER NAME INMEMORY_SIZE POP_STATUS BYTES BYTES_NOT_POPULATED ------ ----- ------------- ---------- ------------ ------------------- SH SALES 5858787328 COMPLETED 18990759936 2151686144
- 33. Oracle In-Memory Column Store Without In-Memory > Bitmap indexes ALTER TABLE INMEMORY > New execution plan: TABLE ACCESS INMEMORY FULL > Population triggered at first query > Not optimal until fully populated Once populated > Very fast access for all queries ALTER TABLE NO INMEMORY > Only row store available > Take care of partially populated table: full table scan through buffer cache Demo: Population and Query 05.10.2015 Page 42 How to use In-Memory
- 34. Oracle In-Memory Column Store RDBMS In-Memory Technologies competitors 05.10.2015 Page 43 Oracle In-Memory Column Store Analytics (columnar) In-Memory option xVelocity HANA BLU Transactional (row storage) TimesTen Hekaton HANA SolidDB SAP HANA
- 35. Oracle In-Memory Column Store In-Memory goal > Execute analytic query in operational database or in BI database Oracle implementation strengths > Doesn’t require any change > Works on any 12.1.0.2 supported architecture > Is very efficient to filter data without any additional index Oracle implementation caveats > Performance depends on Column Store population (CPU intensive) > Do you want to drop your indexes? It's an option on Enterprise Edition (+50% on public price list) Core Message 05.10.2015 Oracle In-Memory Column Store Page 44
- 36. Oracle In-Memory Column Store 05.10.2015 Any questions? Please do ask. Infrastructure at your Service. We look forward to working with you! Page 45 Franck Pachot Senior consultant Oracle Technology Leader Mobile +41 79 963 27 22 [email protected] www.dbi-services.com