SlideShare a Scribd company logo

Same plan different performance

•
Mauro Pagano, profile picture
Mauro Pagano

The document discusses performance discrepancies in SQL execution across identical systems, emphasizing the importance of examining execution plans and SQL execution metrics. It presents various scenarios that highlight how environmental factors, wait events, and buffer cache status can affect performance, even when the execution plan remains the same. Detailed comparisons are provided between different setups and SQL executions to uncover underlying causes of performance variations.

Software
1 of 53
Downloaded 90 times
1
2
Most read
3
4
5
6
7
8
9
10
11
12
13
Most read
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
Most read
Same Plan Different Performance Mauro Pagano
SQL is slower…. • Same SQL experiences different performance in systems that are identical (or supposed to) • First check the execution plan • Most of the time plan is different, address it • But what if the plan is the same? 2
CBO is innocent (this time, maybe) • Exec plan is where CBO’s job end (kind of) • Same plan means CBO “worked” the same • Doesn’t mean everything else IS the same • Shift focus on next step, SQL execution 3
Apples vs oranges? • Make sure the comparison is fair (data) • All external factors should be similar – CPU should be similar – IO should be similar – Memory should be similar 4
“Everything is the same!” • Plan, data and hardware match, now what? • Dig into how the SQL is executed • Wait events and session statistics • Factors – configuration, storage layout, load 5
Old friends get-together • Wait events – Do they match? – Are they close in cardinality? – Do we spend the same time on them? • Session statistics – Do they match? – Are they close in values? 6
Back to the plan for a second • Exec plan is made of lots of small steps • Each one produces/handles/consumes rows • Same behaviors in short and long plans • Keep it simple, focus on the step • Remove the noise if possible (reduce TC) 7
Each scenario is a quiz • SQL is provided • Changes to the initial setup are disclosed • Each run in one environment • Identify what’s different and why 8
Setup • Linux x86-64, 11.2.0.3 • 1 table, 1M rows, 3 columns, no index – N1 unique – N2 100 NDV – C1 100chars long padded string • Identical hardware, same DDL to create table • Controlled environments to isolate behavior • Simplest SQL to reproduce desired behavior 9
Scenario #1 • SQL – select /*+ INDEX(TEST1M) */ count(*) from test1m where n1 between 1 and 1000000 • Environment – Added index on N1 10
Scenario #1 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.31 0.45 2228 2228 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.32 0.45 2228 2228 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=2228 pr=2228 pw=0 time=451619 us) 1000000 1000000 1000000 INDEX RANGE SCAN TEST1M_IDX (cr=2228 pr=2228 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 2228 0.00 0.15 11
Scenario #1 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.07 0.08 0 2228 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.08 0.08 0 2228 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=2228 pr=0 pw=0 time=80038 us) 1000000 1000000 1000000 INDEX RANGE SCAN TEST1M_IDX (cr=2228 pr=0 pw=0 12
Scenario #1 Solution • Buffer Cache cold/warm • (Part of) the data already in memory • Reduced number of physical reads (pr) • Faster performance because less reads • Number of (same) wait events is lower • Isolated environment likely to read more 13
Scenario #2 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Environment – No changes from original setup 14
Scenario #2 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.01 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.57 1.51 28574 28584 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.57 1.52 28574 28584 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28584 pr=28574 pw=0 time=1513999 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28584 pr=28574 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 1 0.00 0.00 db file scattered read 240 0.02 1.07 15
Scenario #2 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 1.04 2.42 14286 28583 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 1.04 2.42 14286 28583 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28583 pr=14286 pw=0 time=2424726 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28583 pr=14286 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 5732 0.01 0.89 db file scattered read 4277 0.00 0.75 16
Scenario #2 – Run (A) - Waits WAIT #140245916217600: nam='db file scattered read' ela= 4834 file#=26 block#=16002 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 4020 file#=26 block#=16130 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2452 file#=26 block#=16258 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 8712 file#=26 block#=16386 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 6417 file#=26 block#=16514 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2267 file#=26 block#=16642 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2637 file#=26 block#=16770 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2304 file#=26 block#=16898 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 1809 file#=26 block#=17026 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2661 file#=26 block#=17154 blocks=128 17
Scenario #2 – Run (B) - Waits WAIT #140245916165224: nam='db file sequential read' ela= 124 file#=26 block#=16002 blocks=1 WAIT #140245916165224: nam='db file scattered read' ela= 139 file#=26 block#=16004 blocks=2 WAIT #140245916165224: nam='db file sequential read' ela= 117 file#=26 block#=16007 blocks=1 ….<<another 38 waits here>> WAIT #140245916165224: nam='db file sequential read' ela= 132 file#=26 block#=16113 blocks=1 WAIT #140245916165224: nam='db file sequential read' ela= 123 file#=26 block#=16116 blocks=1 WAIT #140245916165224: nam='db file scattered read' ela= 142 file#=26 block#=16118 blocks=2 WAIT #140245916165224: nam='db file scattered read' ela= 141 file#=26 block#=16121 blocks=2 WAIT #140245916165224: nam='db file scattered read' ela= 135 file#=26 block#=16124 blocks=2 WAIT #140245916165224: nam='db file sequential read' ela= 119 file#=26 block#=16127 blocks=1 18
Scenario #2 Solution • Buffer cache status (cold/warm) • (Part of) the data already in memory • Reduced number of physical reads (pr) • Number of (same) wait events is higher • Wait events details help track it down – Non-contiguous blocks read • Slower performance because smaller reads 19
Scenario #3 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Environment – No changes – BC warm 20
Scenario #3 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.92 2.96 14286 28583 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.92 2.96 14286 28583 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28583 pr=14286 pw=0 time=2967930 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28583 pr=14286 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 5732 0.10 1.17 db file scattered read 4277 0.28 1.13 21
Scenario #3 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.11 1.01 28573 28575 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.11 1.02 28573 28575 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28575 pr=28573 pw=0 time=1019952 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28575 pr=28573 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ enq: KO - fast object checkpoint 2 0.00 0.00 direct path read 179 0.03 0.90 22
Scenario #3 – Solution • Buffered vs Direct Path reads (different waits too) • (Part of) the data already in memory • Direct Path – skips Buffer Cache and reads whole table every time – consistent performance – number of wait events is consistent • Buffered vs Direct Path decision is made AFTER plan selection (several criteria) 23
Scenario #4 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Environment – No changes – BC cold 24
Scenario #4 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.01 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.57 3.08 15872 15884 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.57 3.10 15872 15884 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=15884 pr=15872 pw=0 time=3086869 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=15884 pr=15872 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 2005 0.05 2.53 25
Scenario #4 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.32 1.66 15872 15881 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.32 1.66 15872 15881 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=15881 pr=15872 pw=0 time=1660864 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=15881 pr=15872 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 141 0.05 1.41 26
Scenario #4 – Run (A) - Waits WAIT #139702845969088: nam='db file scattered read' ela= 265 file#=25 block#=306 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 257 file#=25 block#=314 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 259 file#=25 block#=322 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 254 file#=25 block#=330 blocks=8 ….. WAIT #139702845969088: nam='db file scattered read' ela= 217 file#=25 block#=378 blocks=6 WAIT #139702845969088: nam='db file scattered read' ela= 270 file#=25 block#=386 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 283 file#=25 block#=394 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 263 file#=25 block#=402 blocks=8 27
Scenario #4 – Run (B) - Waits WAIT #139702846026760: nam='db file scattered read' ela= 13508 file#=25 block#=258 blocks=128 WAIT #139702846026760: nam='db file scattered read' ela= 9016 file#=25 block#=386 blocks=128 28
Scenario #4 – Solution 1 • Different db_file_multiblock_read_count value • Same number of blocks read from disk • Number of (same) wait events is higher • Wait events details help track it down – Contiguous blocks read • Slower performance because smaller reads 29
Scenario #4 – Solution 2 • Different extent size (64k vs 1M) • Same number of blocks read from disk • Number of (same) wait events is higher • Wait events details help track it down – Contiguous blocks read • Same params/stats but different storage org • Slower performance because smaller reads 30
Scenario #5 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Env changes – No changes – BC cold, MBRC and extent are identical 31
Scenario #5 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.26 0.72 14285 14297 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.27 0.72 14285 14297 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=14297 pr=14285 pw=0 time=723883 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=14297 pr=14285 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 128 0.04 0.51 32
Scenario #5 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.44 1.29 28574 28586 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.44 1.29 28574 28586 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28586 pr=28574 pw=0 time=1291333 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28586 pr=28574 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 240 0.04 0.95 33
Scenario #5 – Solution 1 • Different PCTFREE (0 vs 50) • Higher number of blocks read for same data • Reads are of the same size hence more reads • Data is more spread out, room for changes • Slower performance because more reads 34
Scenario #5 – Solution 2 • Empty blocks below HWM • Higher number of blocks read for same data • Reads are of the same size hence more reads • Data has been deleted, FTS reads everything • Slower performance because more reads 35
Scenario #6 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Env changes – No changes – BC cold, MBRC, PCTFREE and extent are identical 36
Scenario #6 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.44 1.29 28574 28586 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.44 1.29 28574 28586 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28586 pr=28574 pw=0 time=1291333 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28586 pr=28574 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 240 0.04 0.95 37
Scenario #6 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.73 2.49 28803 58584 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.74 2.49 28803 58584 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=58584 pr=28803 pw=0 time=2492596 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=58584 pr=28803 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 240 0.23 1.73 cell single block physical read 230 0.01 0.06 38
Scenario #6 – Waits and SesStats • Wait events show – single block reads from UNDO tbs for obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 15412 file#=26 block#=15618 blocks=128 obj#=74828 WAIT #140029131327704: nam='cell single block physical read' ela= 220 … bytes=8192 obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 11786 file#=26 block#=15746 blocks=128 obj#=74828 WAIT #140029131327704: nam='cell single block physical read' ela= 233 … bytes=8192 obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 5938 file#=26 block#=15874 blocks=128 obj#=74828 WAIT #140029131327704: nam='cell single block physical read' ela= 224 … bytes=8192 obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 12162 file#=26 block#=16002 blocks=128 obj#=74828 • v$sesstat shows high – data blocks consistent reads - undo records applied 39
Scenario #6 - Solution • Different concurrency/workload • Higher number of blocks read for same data • Waits -> reads from UNDO tbs • SesStats -> UNDO records applied • Slower performance because more reads + more work to recreate the correct image 40
Scenario #7 • SQL – select /* 1st run */ n1,c1 from test1m where n1 in (1,1000,5000) • Env changes – Index on TEST1M(N1) – BC cold, MBRC, PCTFREE and extent are identical – No concurrency at the time SQL is executed 41
Scenario #7 – Why so many cr/pr? call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.00 0.75 11 18 0 3 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.00 0.75 11 18 0 3 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 3 3 3 INLIST ITERATOR (cr=18 pr=11 pw=0 time=235681 us) 3 3 3 TABLE ACCESS BY INDEX ROWID TEST1M (cr=18 pr=11 3 3 3 INDEX RANGE SCAN TEST1M_IDX (cr=9 pr=5 pw=0 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 11 0.18 0.52 42
Scenario #7 – Waits and SesStats • Wait events show – single block reads from data tbs, same obj# WAIT #140…: nam='db file sequential read' ela= 7414 file#=26 block#=2356 blocks=1 obj#=75022 WAIT #140…: nam='db file sequential read' ela= 41395 file#=26 block#=131 blocks=1 obj#=74828 WAIT #140…: nam='db file sequential read' ela= 181594 file#=26 block#=78403 blocks=1 obj#=74828 • v$sesstat shows high – table fetch continued row 43
Scenario #7 - Solution • Row migration, index points to original rowid • Higher number of blocks read for same data • Waits -> reads are from data tbs • SesStats -> table fetch continued row • Slower performance because more reads + more work to find all the row pieces • Similar behavior happens with chained rows 44
Scenario #8 • SQL – select /* 2nd run */ n1,c1, ora_rowscn from test1m where rownum <= 5000 • Env changes – Index on TEST1M(N1) – BC cold, MBRC, PCTFREE and extent are identical – No concurrency at the time SQL is executed 45
Scenario #8 - Why so many seq read? call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 6 0.03 0.22 393 5378 0 5000 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 8 0.03 0.22 393 5378 0 5000 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- ------------------------------------------- 5000 5000 5000 COUNT STOPKEY (cr=5378 pr=393 pw=0 time=91193 5000 5000 5000 TABLE ACCESS FULL TEST1M (cr=5378 pr=393 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 381 0.00 0.19 db file scattered read 2 0.00 0.00 46
Scenario #8 – Waits and SesStats • Wait events show – single block reads from data tbs, same obj# WAIT #1405…: nam='db file scattered read' ela= 6434 file#=26 block#=132 blocks=4 obj#=74828 WAIT #1405…: nam='db file sequential read' ela= 193 file#=26 block#=78670 blocks=1 obj#=74828 WAIT #1405…: nam='db file sequential read' ela= 182 file#=26 block#=78686 blocks=1 obj#=74828 WAIT #1405…: nam='db file sequential read' ela= 3445 file#=26 block#=7890 blocks=1 obj#=74828 • v$sesstat shows high – table fetch continued row 47
Scenario #8 - Solution • Row migration, pseudo col needs row header • Higher number of blocks read for same data • Waits -> reads are from data tbs • SesStats -> table fetch continued row • Slower performance because more reads + more work to find all the row pieces • Similar behavior happens with chained rows 48
Other things to consider • Same PHV with small differences – Predicate ordering – Column projection • Exadata Optimizations – Exadata Smart Flash Cache – Storage indexes • External to the database – File system / SAN / Disk caching – Read-ahead optimizations 49
Conclusions • Same plan can still run differently • Storage organization and concurrency impact • Fix one scenario can introduce another, ie. – low PCTFREE higher chance of row migration – high caching slows down buffered mreads • Find a balance to achieve optimal performance 50
51
References • 'DB_FILE_MULTIBLOCK_READ_COUNT' AND EXTENTS MANAGEMENT (Doc ID 181272.1) • Higher 'direct path read' Waits in 11g when Compared to 10g (Doc ID 793845.1) • Why Is My Query Sometimes Slower Than Other Times with Higher Consistent Gets Although No Change in Execution Plan? (Doc ID 1558349.1) • Row Chaining and Row Migration (Doc ID 122020.1) 52
Contact Information • http://mauro-pagano.com – Email • mauro.pagano@gmail.com – Download • SQLd360 vYYMM (date) – Pages • SQLd360 53

More Related Content

PDF
Oracle statistics by example
Mauro Pagano
 
PDF
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Tanel Poder
 
PDF
SQLd360
Mauro Pagano
 
PDF
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
John Beresniewicz
 
PDF
Oracle Performance Tuning Fundamentals
Enkitec
 
PDF
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
 
PDF
Understanding oracle rac internals part 1 - slides
Mohamed Farouk
 
PPTX
Part1 of SQL Tuning Workshop - Understanding the Optimizer
Maria Colgan
 
Oracle statistics by example
Mauro Pagano
 
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Tanel Poder
 
SQLd360
Mauro Pagano
 
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
John Beresniewicz
 
Oracle Performance Tuning Fundamentals
Enkitec
 
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
 
Understanding oracle rac internals part 1 - slides
Mohamed Farouk
 
Part1 of SQL Tuning Workshop - Understanding the Optimizer
Maria Colgan
 

What's hot (20)

PPTX
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
 
PDF
SQL Monitoring in Oracle Database 12c
Tanel Poder
 
PDF
Tanel Poder - Scripts and Tools short
Tanel Poder
 
PDF
New Generation Oracle RAC Performance
Anil Nair
 
PPTX
Oracle RAC features on Exadata
Anil Nair
 
PPT
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
PDF
Maximum Availability Architecture - Best Practices for Oracle Database 19c
Glen Hawkins
 
PDF
Oracle db performance tuning
Simon Huang
 
PDF
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Aaron Shilo
 
PDF
Chasing the optimizer
Mauro Pagano
 
PDF
Understanding oracle rac internals part 2 - slides
Mohamed Farouk
 
PPTX
Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder
 
PPTX
SQL Plan Directives explained
Mauro Pagano
 
PDF
Oracle Database performance tuning using oratop
Sandesh Rao
 
PDF
Exadata master series_asm_2020
Anil Nair
 
PDF
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
PPTX
Oracle GoldenGate Performance Tuning
Bobby Curtis
 
PDF
Smart monitoring how does oracle rac manage resource, state ukoug19
Anil Nair
 
PDF
My First 100 days with an Exadata (PPT)
Gustavo Rene Antunez
 
PDF
Oracle RAC 19c: Best Practices and Secret Internals
Anil Nair
 
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
 
SQL Monitoring in Oracle Database 12c
Tanel Poder
 
Tanel Poder - Scripts and Tools short
Tanel Poder
 
New Generation Oracle RAC Performance
Anil Nair
 
Oracle RAC features on Exadata
Anil Nair
 
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
Maximum Availability Architecture - Best Practices for Oracle Database 19c
Glen Hawkins
 
Oracle db performance tuning
Simon Huang
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Aaron Shilo
 
Chasing the optimizer
Mauro Pagano
 
Understanding oracle rac internals part 2 - slides
Mohamed Farouk
 
Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder
 
SQL Plan Directives explained
Mauro Pagano
 
Oracle Database performance tuning using oratop
Sandesh Rao
 
Exadata master series_asm_2020
Anil Nair
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
Oracle GoldenGate Performance Tuning
Bobby Curtis
 
Smart monitoring how does oracle rac manage resource, state ukoug19
Anil Nair
 
My First 100 days with an Exadata (PPT)
Gustavo Rene Antunez
 
Oracle RAC 19c: Best Practices and Secret Internals
Anil Nair
 
Ad

Similar to Same plan different performance (20)

PPTX
SQL Tuning, takes 3 to tango
Mauro Pagano
 
PDF
PostgreSQL Portland Performance Practice Project - Database Test 2 Tuning
Mark Wong
 
PPT
Metadata Matters
afa reg
 
DOC
SAV
Akash Indani
 
PPTX
Oracle Database Performance Tuning Basics
nitin anjankar
 
PDF
AWR Ambiguity: Performance reasoning when the numbers don't add up
John Beresniewicz
 
PPT
Compression ow2009 r2
carldudley
 
PPTX
Analyze database system using a 3 d method
Ajith Narayanan
 
PPTX
Analyzing SQL Traces generated by EVENT 10046.pptx
ssuserbad8d3
 
PPT
zen and the art of SQL optimization
Karen Morton
 
PPT
Thomas+Niewel+ +Oracletuning
afa reg
 
PPT
Rmoug ashmaster
Kyle Hailey
 
PDF
Contiguous
oracle documents
 
PDF
Autoextend
oracle documents
 
PPT
Oracle Performance Tuning DE(v1.2)-part2.ppt
VenugopalChattu1
 
PDF
Database & Technology 1 _ Tom Kyte _ SQL Techniques.pdf
InSync2011
 
DOC
SQLQueries
karunakar81987
 
PPT
Oracle Open World Thursday 230 ashmasters
Kyle Hailey
 
PDF
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
 
PDF
Hailey_Database_Performance_Made_Easy_through_Graphics.pdf
cookie1969
 
SQL Tuning, takes 3 to tango
Mauro Pagano
 
PostgreSQL Portland Performance Practice Project - Database Test 2 Tuning
Mark Wong
 
Metadata Matters
afa reg
 
SAV
Akash Indani
 
Oracle Database Performance Tuning Basics
nitin anjankar
 
AWR Ambiguity: Performance reasoning when the numbers don't add up
John Beresniewicz
 
Compression ow2009 r2
carldudley
 
Analyze database system using a 3 d method
Ajith Narayanan
 
Analyzing SQL Traces generated by EVENT 10046.pptx
ssuserbad8d3
 
zen and the art of SQL optimization
Karen Morton
 
Thomas+Niewel+ +Oracletuning
afa reg
 
Rmoug ashmaster
Kyle Hailey
 
Contiguous
oracle documents
 
Autoextend
oracle documents
 
Oracle Performance Tuning DE(v1.2)-part2.ppt
VenugopalChattu1
 
Database & Technology 1 _ Tom Kyte _ SQL Techniques.pdf
InSync2011
 
SQLQueries
karunakar81987
 
Oracle Open World Thursday 230 ashmasters
Kyle Hailey
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
 
Hailey_Database_Performance_Made_Easy_through_Graphics.pdf
cookie1969
 
Ad

Recently uploaded (20)

PDF
New Download MiniTool Partition Wizard Crack Latest Version 2025
imang66g
 
PPTX
Maximizing Revenue with Marketo Measure: A Deep Dive into Multi-Touch Attribu...
bbedford2
 
PPTX
Web Testing.pptx528278vshbuqffqhhqiwnwuq
studylike474
 
PDF
advancepresentationskillshdhdhhdhdhdhhfhf
jasmenrojas249
 
PDF
Enhancing Healthcare RPM Platforms with Contextual AI Integration
Cadabra Studio
 
PPTX
slidesgo-unlocking-the-code-the-dynamic-dance-of-variables-and-constants-2024...
kr2589474
 
PDF
lesson-2-rules-of-netiquette.pdf.bshhsjdj
jasmenrojas249
 
PPTX
classification of computer and basic part of digital computer
ravisinghrajpurohit3
 
PDF
Exploring AI Agents in Process Industries
amoreira6
 
PDF
WatchTraderHub - Watch Dealer software with inventory management and multi-ch...
WatchDealer Pavel
 
PDF
Balancing Resource Capacity and Workloads with OnePlan – Avoid Overloading Te...
OnePlan Solutions
 
PPTX
Role Of Python In Programing Language.pptx
jaykoshti048
 
PDF
On Software Engineers' Productivity - Beyond Misleading Metrics
Romén Rodríguez-Gil
 
PDF
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
PDF
Protecting the Digital World Cyber Securit
dnthakkar16
 
PDF
MiniTool Power Data Recovery Crack New Pre Activated Version Latest 2025
imang66g
 
PPTX
Presentation about variables and constant.pptx
kr2589474
 
DOCX
Can You Build Dashboards Using Open Source Visualization Tool.docx
Varsha Nayak
 
PDF
Immersive experiences: what Pharo users do!
ESUG
 
PPTX
Presentation about Database and Database Administrator
abhishekchauhan86963
 
New Download MiniTool Partition Wizard Crack Latest Version 2025
imang66g
 
Maximizing Revenue with Marketo Measure: A Deep Dive into Multi-Touch Attribu...
bbedford2
 
Web Testing.pptx528278vshbuqffqhhqiwnwuq
studylike474
 
advancepresentationskillshdhdhhdhdhdhhfhf
jasmenrojas249
 
Enhancing Healthcare RPM Platforms with Contextual AI Integration
Cadabra Studio
 
slidesgo-unlocking-the-code-the-dynamic-dance-of-variables-and-constants-2024...
kr2589474
 
lesson-2-rules-of-netiquette.pdf.bshhsjdj
jasmenrojas249
 
classification of computer and basic part of digital computer
ravisinghrajpurohit3
 
Exploring AI Agents in Process Industries
amoreira6
 
WatchTraderHub - Watch Dealer software with inventory management and multi-ch...
WatchDealer Pavel
 
Balancing Resource Capacity and Workloads with OnePlan – Avoid Overloading Te...
OnePlan Solutions
 
Role Of Python In Programing Language.pptx
jaykoshti048
 
On Software Engineers' Productivity - Beyond Misleading Metrics
Romén Rodríguez-Gil
 
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
Protecting the Digital World Cyber Securit
dnthakkar16
 
MiniTool Power Data Recovery Crack New Pre Activated Version Latest 2025
imang66g
 
Presentation about variables and constant.pptx
kr2589474
 
Can You Build Dashboards Using Open Source Visualization Tool.docx
Varsha Nayak
 
Immersive experiences: what Pharo users do!
ESUG
 
Presentation about Database and Database Administrator
abhishekchauhan86963
 

Same plan different performance

  • 4. Apples vs oranges? • Make sure the comparison is fair (data) • All external factors should be similar – CPU should be similar – IO should be similar – Memory should be similar 4
  • 5. “Everything is the same!” • Plan, data and hardware match, now what? • Dig into how the SQL is executed • Wait events and session statistics • Factors – configuration, storage layout, load 5
  • 6. Old friends get-together • Wait events – Do they match? – Are they close in cardinality? – Do we spend the same time on them? • Session statistics – Do they match? – Are they close in values? 6
  • 7. Back to the plan for a second • Exec plan is made of lots of small steps • Each one produces/handles/consumes rows • Same behaviors in short and long plans • Keep it simple, focus on the step • Remove the noise if possible (reduce TC) 7
  • 8. Each scenario is a quiz • SQL is provided • Changes to the initial setup are disclosed • Each run in one environment • Identify what’s different and why 8
  • 9. Setup • Linux x86-64, 11.2.0.3 • 1 table, 1M rows, 3 columns, no index – N1 unique – N2 100 NDV – C1 100chars long padded string • Identical hardware, same DDL to create table • Controlled environments to isolate behavior • Simplest SQL to reproduce desired behavior 9
  • 11. Scenario #1 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.31 0.45 2228 2228 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.32 0.45 2228 2228 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=2228 pr=2228 pw=0 time=451619 us) 1000000 1000000 1000000 INDEX RANGE SCAN TEST1M_IDX (cr=2228 pr=2228 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 2228 0.00 0.15 11
  • 12. Scenario #1 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.07 0.08 0 2228 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.08 0.08 0 2228 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=2228 pr=0 pw=0 time=80038 us) 1000000 1000000 1000000 INDEX RANGE SCAN TEST1M_IDX (cr=2228 pr=0 pw=0 12
  • 13. Scenario #1 Solution • Buffer Cache cold/warm • (Part of) the data already in memory • Reduced number of physical reads (pr) • Faster performance because less reads • Number of (same) wait events is lower • Isolated environment likely to read more 13
  • 15. Scenario #2 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.01 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.57 1.51 28574 28584 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.57 1.52 28574 28584 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28584 pr=28574 pw=0 time=1513999 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28584 pr=28574 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 1 0.00 0.00 db file scattered read 240 0.02 1.07 15
  • 16. Scenario #2 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 1.04 2.42 14286 28583 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 1.04 2.42 14286 28583 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28583 pr=14286 pw=0 time=2424726 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28583 pr=14286 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 5732 0.01 0.89 db file scattered read 4277 0.00 0.75 16
  • 17. Scenario #2 – Run (A) - Waits WAIT #140245916217600: nam='db file scattered read' ela= 4834 file#=26 block#=16002 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 4020 file#=26 block#=16130 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2452 file#=26 block#=16258 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 8712 file#=26 block#=16386 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 6417 file#=26 block#=16514 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2267 file#=26 block#=16642 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2637 file#=26 block#=16770 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2304 file#=26 block#=16898 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 1809 file#=26 block#=17026 blocks=128 WAIT #140245916217600: nam='db file scattered read' ela= 2661 file#=26 block#=17154 blocks=128 17
  • 18. Scenario #2 – Run (B) - Waits WAIT #140245916165224: nam='db file sequential read' ela= 124 file#=26 block#=16002 blocks=1 WAIT #140245916165224: nam='db file scattered read' ela= 139 file#=26 block#=16004 blocks=2 WAIT #140245916165224: nam='db file sequential read' ela= 117 file#=26 block#=16007 blocks=1 ….<<another 38 waits here>> WAIT #140245916165224: nam='db file sequential read' ela= 132 file#=26 block#=16113 blocks=1 WAIT #140245916165224: nam='db file sequential read' ela= 123 file#=26 block#=16116 blocks=1 WAIT #140245916165224: nam='db file scattered read' ela= 142 file#=26 block#=16118 blocks=2 WAIT #140245916165224: nam='db file scattered read' ela= 141 file#=26 block#=16121 blocks=2 WAIT #140245916165224: nam='db file scattered read' ela= 135 file#=26 block#=16124 blocks=2 WAIT #140245916165224: nam='db file sequential read' ela= 119 file#=26 block#=16127 blocks=1 18
  • 19. Scenario #2 Solution • Buffer cache status (cold/warm) • (Part of) the data already in memory • Reduced number of physical reads (pr) • Number of (same) wait events is higher • Wait events details help track it down – Non-contiguous blocks read • Slower performance because smaller reads 19
  • 21. Scenario #3 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.92 2.96 14286 28583 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.92 2.96 14286 28583 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28583 pr=14286 pw=0 time=2967930 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28583 pr=14286 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 5732 0.10 1.17 db file scattered read 4277 0.28 1.13 21
  • 22. Scenario #3 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.11 1.01 28573 28575 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.11 1.02 28573 28575 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28575 pr=28573 pw=0 time=1019952 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28575 pr=28573 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ enq: KO - fast object checkpoint 2 0.00 0.00 direct path read 179 0.03 0.90 22
  • 23. Scenario #3 – Solution • Buffered vs Direct Path reads (different waits too) • (Part of) the data already in memory • Direct Path – skips Buffer Cache and reads whole table every time – consistent performance – number of wait events is consistent • Buffered vs Direct Path decision is made AFTER plan selection (several criteria) 23
  • 25. Scenario #4 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.01 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.57 3.08 15872 15884 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.57 3.10 15872 15884 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=15884 pr=15872 pw=0 time=3086869 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=15884 pr=15872 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 2005 0.05 2.53 25
  • 26. Scenario #4 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.32 1.66 15872 15881 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.32 1.66 15872 15881 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=15881 pr=15872 pw=0 time=1660864 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=15881 pr=15872 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 141 0.05 1.41 26
  • 27. Scenario #4 – Run (A) - Waits WAIT #139702845969088: nam='db file scattered read' ela= 265 file#=25 block#=306 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 257 file#=25 block#=314 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 259 file#=25 block#=322 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 254 file#=25 block#=330 blocks=8 ….. WAIT #139702845969088: nam='db file scattered read' ela= 217 file#=25 block#=378 blocks=6 WAIT #139702845969088: nam='db file scattered read' ela= 270 file#=25 block#=386 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 283 file#=25 block#=394 blocks=8 WAIT #139702845969088: nam='db file scattered read' ela= 263 file#=25 block#=402 blocks=8 27
  • 28. Scenario #4 – Run (B) - Waits WAIT #139702846026760: nam='db file scattered read' ela= 13508 file#=25 block#=258 blocks=128 WAIT #139702846026760: nam='db file scattered read' ela= 9016 file#=25 block#=386 blocks=128 28
  • 29. Scenario #4 – Solution 1 • Different db_file_multiblock_read_count value • Same number of blocks read from disk • Number of (same) wait events is higher • Wait events details help track it down – Contiguous blocks read • Slower performance because smaller reads 29
  • 30. Scenario #4 – Solution 2 • Different extent size (64k vs 1M) • Same number of blocks read from disk • Number of (same) wait events is higher • Wait events details help track it down – Contiguous blocks read • Same params/stats but different storage org • Slower performance because smaller reads 30
  • 31. Scenario #5 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Env changes – No changes – BC cold, MBRC and extent are identical 31
  • 32. Scenario #5 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.26 0.72 14285 14297 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.27 0.72 14285 14297 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=14297 pr=14285 pw=0 time=723883 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=14297 pr=14285 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 128 0.04 0.51 32
  • 33. Scenario #5 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.44 1.29 28574 28586 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.44 1.29 28574 28586 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28586 pr=28574 pw=0 time=1291333 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28586 pr=28574 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 240 0.04 0.95 33
  • 34. Scenario #5 – Solution 1 • Different PCTFREE (0 vs 50) • Higher number of blocks read for same data • Reads are of the same size hence more reads • Data is more spread out, room for changes • Slower performance because more reads 34
  • 35. Scenario #5 – Solution 2 • Empty blocks below HWM • Higher number of blocks read for same data • Reads are of the same size hence more reads • Data has been deleted, FTS reads everything • Slower performance because more reads 35
  • 36. Scenario #6 • SQL – select /*+ FULL(TEST1M) */ count(*) from test1m • Env changes – No changes – BC cold, MBRC, PCTFREE and extent are identical 36
  • 37. Scenario #6 – Run (A) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.44 1.29 28574 28586 1 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.44 1.29 28574 28586 1 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=28586 pr=28574 pw=0 time=1291333 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=28586 pr=28574 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 240 0.04 0.95 37
  • 38. Scenario #6 – Run (B) call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.73 2.49 28803 58584 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.74 2.49 28803 58584 0 1 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 1 1 1 SORT AGGREGATE (cr=58584 pr=28803 pw=0 time=2492596 us) 1000000 1000000 1000000 TABLE ACCESS FULL TEST1M (cr=58584 pr=28803 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file scattered read 240 0.23 1.73 cell single block physical read 230 0.01 0.06 38
  • 39. Scenario #6 – Waits and SesStats • Wait events show – single block reads from UNDO tbs for obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 15412 file#=26 block#=15618 blocks=128 obj#=74828 WAIT #140029131327704: nam='cell single block physical read' ela= 220 … bytes=8192 obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 11786 file#=26 block#=15746 blocks=128 obj#=74828 WAIT #140029131327704: nam='cell single block physical read' ela= 233 … bytes=8192 obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 5938 file#=26 block#=15874 blocks=128 obj#=74828 WAIT #140029131327704: nam='cell single block physical read' ela= 224 … bytes=8192 obj#=0 WAIT #140029131327704: nam='db file scattered read' ela= 12162 file#=26 block#=16002 blocks=128 obj#=74828 • v$sesstat shows high – data blocks consistent reads - undo records applied 39
  • 40. Scenario #6 - Solution • Different concurrency/workload • Higher number of blocks read for same data • Waits -> reads from UNDO tbs • SesStats -> UNDO records applied • Slower performance because more reads + more work to recreate the correct image 40
  • 41. Scenario #7 • SQL – select /* 1st run */ n1,c1 from test1m where n1 in (1,1000,5000) • Env changes – Index on TEST1M(N1) – BC cold, MBRC, PCTFREE and extent are identical – No concurrency at the time SQL is executed 41
  • 42. Scenario #7 – Why so many cr/pr? call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 2 0.00 0.75 11 18 0 3 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 4 0.00 0.75 11 18 0 3 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 3 3 3 INLIST ITERATOR (cr=18 pr=11 pw=0 time=235681 us) 3 3 3 TABLE ACCESS BY INDEX ROWID TEST1M (cr=18 pr=11 3 3 3 INDEX RANGE SCAN TEST1M_IDX (cr=9 pr=5 pw=0 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 11 0.18 0.52 42
  • 43. Scenario #7 – Waits and SesStats • Wait events show – single block reads from data tbs, same obj# WAIT #140…: nam='db file sequential read' ela= 7414 file#=26 block#=2356 blocks=1 obj#=75022 WAIT #140…: nam='db file sequential read' ela= 41395 file#=26 block#=131 blocks=1 obj#=74828 WAIT #140…: nam='db file sequential read' ela= 181594 file#=26 block#=78403 blocks=1 obj#=74828 • v$sesstat shows high – table fetch continued row 43
  • 44. Scenario #7 - Solution • Row migration, index points to original rowid • Higher number of blocks read for same data • Waits -> reads are from data tbs • SesStats -> table fetch continued row • Slower performance because more reads + more work to find all the row pieces • Similar behavior happens with chained rows 44
  • 45. Scenario #8 • SQL – select /* 2nd run */ n1,c1, ora_rowscn from test1m where rownum <= 5000 • Env changes – Index on TEST1M(N1) – BC cold, MBRC, PCTFREE and extent are identical – No concurrency at the time SQL is executed 45
  • 46. Scenario #8 - Why so many seq read? call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Fetch 6 0.03 0.22 393 5378 0 5000 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 8 0.03 0.22 393 5378 0 5000 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- ------------------------------------------- 5000 5000 5000 COUNT STOPKEY (cr=5378 pr=393 pw=0 time=91193 5000 5000 5000 TABLE ACCESS FULL TEST1M (cr=5378 pr=393 Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ db file sequential read 381 0.00 0.19 db file scattered read 2 0.00 0.00 46
  • 47. Scenario #8 – Waits and SesStats • Wait events show – single block reads from data tbs, same obj# WAIT #1405…: nam='db file scattered read' ela= 6434 file#=26 block#=132 blocks=4 obj#=74828 WAIT #1405…: nam='db file sequential read' ela= 193 file#=26 block#=78670 blocks=1 obj#=74828 WAIT #1405…: nam='db file sequential read' ela= 182 file#=26 block#=78686 blocks=1 obj#=74828 WAIT #1405…: nam='db file sequential read' ela= 3445 file#=26 block#=7890 blocks=1 obj#=74828 • v$sesstat shows high – table fetch continued row 47
  • 48. Scenario #8 - Solution • Row migration, pseudo col needs row header • Higher number of blocks read for same data • Waits -> reads are from data tbs • SesStats -> table fetch continued row • Slower performance because more reads + more work to find all the row pieces • Similar behavior happens with chained rows 48
  • 49. Other things to consider • Same PHV with small differences – Predicate ordering – Column projection • Exadata Optimizations – Exadata Smart Flash Cache – Storage indexes • External to the database – File system / SAN / Disk caching – Read-ahead optimizations 49
  • 50. Conclusions • Same plan can still run differently • Storage organization and concurrency impact • Fix one scenario can introduce another, ie. – low PCTFREE higher chance of row migration – high caching slows down buffered mreads • Find a balance to achieve optimal performance 50
  • 51. 51
  • 52. References • 'DB_FILE_MULTIBLOCK_READ_COUNT' AND EXTENTS MANAGEMENT (Doc ID 181272.1) • Higher 'direct path read' Waits in 11g when Compared to 10g (Doc ID 793845.1) • Why Is My Query Sometimes Slower Than Other Times with Higher Consistent Gets Although No Change in Execution Plan? (Doc ID 1558349.1) • Row Chaining and Row Migration (Doc ID 122020.1) 52