OOUG - Oracle Performance Tuning with AAS
Download as PPT, PDF15 likes12,391 views
The document provides information about Kyle Hailey and his work in Oracle performance tuning. It includes details about his career history working with Oracle from 1990 to present, including roles in support, porting software versions, benchmarking, and performance tuning. Graphics and clear visualizations are emphasized as important tools for effectively communicating complex technical information and identifying problems. The goal is stated as simplifying database tuning information to empower database administrators.
OOUG - Oracle Performance Tuning with AAS
- 1. Oracle Performance Tuning Kyle Hailey [email protected] kylehailey.com
- 2. Who is Kyle Hailey 1990 Oracle – – – – – 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST Real World Performance 2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g Success! First successful OEM design
- 3. Who is Kyle Hailey 1990 Oracle – – – – – 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST Real World Performance 2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g 2005 Embarcadero – DB Optimizer
- 4. Who is Kyle Hailey • 1990 Oracle • • • • • 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST Real World Performance 2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g 2005 Embarcadero DB Optimizer Delphix When not being a Geek - Have a little 4 year old boy who takes up all my time
- 5. Production Instance Database File system File system QA UAT Instance Instance Instance Database Database Database File system File system File system File system File system File system File system Development
- 7. My Goal Simplify the information and empower the DBA
- 8. How do you tune a Database? Database is running slow! • first step? • questions? • tools? • repeatable method?
- 9. What statistics do you look at? Disastrously difficult
- 10. Launch: Pressure Midnight January 28, the Flight 1986 Night before Lives are on the line Thanks to Edward Tufte
- 12. 13 Pages Faxed
- 13. 13 Pages Faxed 3 different types of names
- 14. 13 Pages Faxed Damage (in overwhelming detail) but No Temperatures
- 15. 13 Pages Faxed Temperatures but limited Damage Test engines, not launches, fired horizontally Missing temperatures for 5 erosion damage flights
- 16. 13 Pages Faxed “blow by”, not more important “erosion”, (at hottest and coldest launches)
- 18. Original Engineering data damages at the hottest and coldest temperature - management 3 2 1 55 60 65 70 Would you launch? 75 80
- 20. Congressional Hearings Evidence No Damage Legend Damage hard to read
- 23. Clearer 12 8 4 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage)
- 24. Clearer 12 8 4 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage) 2. Include successes
- 25. Clearer 12 8 4 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences
- 26. Clearer 12 8 4 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp
- 27. Clearer Damage on every flight below 65 12 8 4 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp No damage on every flight above 75
- 28. Clearer 12 8 4 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp
- 29. Clearer 12 12 8 8 4 X 4 30 35 40 45 50 55 60 65 70 75 80 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp 5. Scale known vs unknown
- 30. Difficult NASA Engineers Fail Congressional Investigators Fail Data Visualization is Difficult But … Lack of Clarity can be devastating
- 31. Solutions • Clear Identification • Know how to identify problems and issues • Access to details • Provide solutions and/or information to address the issues • Graphics • Easy understanding, effective communication and discussion
- 32. First Step: Graphics “The humans … are exceptionally good at parsing visual information,motion especially when that information is coded by color and/or .” Knowledge representation in cognitive science. Westbury, C. & Wilensky, U. (1998)
- 33. Why Use Graphics You can't imagine how many times I was told that nobody wanted or would use graphics … -- Jef Raskin, the father of the Macintosh Infocus – (overhead projectors) sited a study that humans can parse graphical information 400,000 times faster than textual data
- 34. Counties in US 3101 Counties 50 pages
- 35. I y x Average Standard Deviation Linear Regression x 10 8 13 9 11 14 6 4 12 7 5 9 8.04 6.95 7.58 8.81 8.33 9.96 7.24 4.26 10.84 4.82 5.68 7.5 3.31 1.33 2.03 Anscombe's Quartet II III y x y x 10 9.14 10 7.46 8 8.14 8 6.77 13 8.74 13 12.74 9 8.77 9 7.11 11 9.26 11 7.81 14 8.1 14 8.84 6 6.13 6 6.08 4 3.1 4 5.39 12 9.13 12 8.15 7 7.26 7 6.42 5 4.74 5 5.73 9 7.5 9 7.5 3.31 1.33 2.03 3.31 1.33 2.03 IV y 8 8 8 8 8 8 8 19 8 8 8 9 3.31 1.33 6.58 5.76 7.71 8.84 8.47 7.04 5.25 12.5 5.56 7.91 6.89 7.5 2.03 “If I can't picture it, I can't understand it” - Albert Einstein
- 36. Graphics for Anscombe’s Quartet
- 37. Do You Want? Engineering Data?
- 38. Do You Want? Pretty Pictures
- 39. Do You Want? Clean and Clear ? ? ? ? ? ?
- 40. Tuning the Database Complex What is a Averages life look day in the like for a DBA who has performance issues? I y x Average Standard Deviation Linear Regression 9 3.31 1.33 x 7.5 2.03 Anscombe's Quartet II III y x y x 9 7.5 9 7.5 3.31 2.03 3.31 2.03 1.33 1.33 IV y 9 3.31 1.33 7.5 2.03
- 41. What statistics do you look at? AAS
- 42. Imagine Trying to Drive your Car Would you want your dashboard to look like : And is updated once and hour Or would you like it to look …
- 43. How Can We Open the Black Box? LOAD Max CPU (yard stick) Top Activity SQL Sessions
- 44. How Can We Open the Black Box? OEM ASHMON/SASH DB Optimizer • Powerful - Identifies issues quickly and indepth • Interactive - Allows exploring the data • Easy - Understandable by everyone, DBA, Dev and Managers !
- 46. Average Active Sessions (AAS) The Most Important Metric Kyle Hailey [email protected] http://kylehailey.com
- 47. In this Session 1. AAS – – Single Metric DB Performance 1. Yardstick – – CPU Count Compare to AAS 1. AAS components – – – 10/18/13 CPU Waits Time series 47
- 48. Database Performance How quick can you find Bottleneck in DB If DB is idle Current DB Load what is DB Load ? What do you use? Statspack/AWR V$active_session_history Alerts what the *!####! *!*? is the database doing ?! what do you alert on ? 10/18/13 48
- 50. Statspack: Cheat Sheet • Install – Connect as SYSDBA – @?/rdbms/admin/spcreate.sql • Run – Exec statspack.snap; • Generate Reports – @?/rdbms/admin/spreport.sql – (AWR : @?/rdbms/admin/awrrpt.sql ) Statspack = always available Copyright 2006 Kyle Hailey
- 51. Statspack Sections (10g) Instance description Host hardware Snapshot headline Cache information Load profile Instance efficiency Shared pool stats Top timed events Host CPU load Host / Instance CPU warning VM activity Memory usage Time model stats RAC statistics All wait events Background wait events Event histogram SQL x 9 Instance activity Log switches Session details x 3 Tablespace I/O File I/O File I/O histogram Buffer pool Instance Recovery Buffer pool advisory Buffer busy waits Various PGA summaries PGA histogram PGA advisory PGA allocation summary PGA allocation top N Enqueue (lock) activity Undo stats x 2 Latch activity Latch miss details Latch parent and child Mutex Segment stats x 8 Dictionary cache Library cache RAC (GES) RAC (CR and CUR served) RAC (cache xfer x 2) RAC (Remastering) Streams x 7 Shared pool advisor Java pool advisor SGA resizing SGA target advisor SGA summary SGA detail SQL memory summary Resource limits Parameters
- 52. Statspack Method Who/When Summary Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % for Load profile goodTotal Event Waits having a (s) Call Time Time feel for the ---------------------- ------------ application and ----------- --------buffer busy waits 2,748 250 78.72 comparing two 10.16 CPU time 32 periods for changes free buffer waits 1,588 15 4.63 Efficiency 8 ratios 2.51 write complete waits 10 log buffer space 306 misleading carry 1.51 5 ---------------------------------------- 6 over from version days Waits Big Picture
- 53. Statspack Method v$system_event - wait events v$sysstat – cpu stats Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time ---------------------- ------------ ----------- --------buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ---------------------------------------- Old method: compare wait time to CPU time
- 54. Current Method • • • 1000 lines of data / 30 pages What do you look at ? Top 5 Timed Events? 188 minutes CPU Elapsed 15 minutes 48 Processors or 74% idle Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time ------------------------ ------------ ----------- -------CPU time 11,330 81.95 log file sync 163,725 1,012 7.32 enqueue 4,486 615 4.45 latch free 28,786 236 1.71 db file sequential read 1,420,355 149 1.08 ----------------------------------------Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html Analysis requires •elapsed time •CPU core count
- 55. Average Active Sessions (AAS) DB Time = wait time + CPU time AAS = DB Time/Elapsed Time AAS like run queue on UNIX
- 57. AWR Report • AAS = DB Time/Elapsed Time • 23.56/59.66 = 0.39 AAS= 0.39 10/18/13 57
- 58. ASHRPT ASH Report For TESTDB/testdb ASH Report For TESTDB/testdb DB Name DB Id Instance DB Name DB Id Instance TESTDB TESTDB CPUs CPUs Inst Num Release Inst Num Release 2371570538 testdb 2371570538 testdb SGA Size Buffer Cache SGA Size Buffer Cache RAC Host RAC Host 11 10.2.0.1.0 NO sdbe604a 10.2.0.1.0 NO sdbe604a Shared Pool ASH Buffer Size Shared Pool ASH Buffer Size 22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: 12.39 Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class %% Activity Event Event Class Activity Sessions Sessions CPU ++ Wait for CPU CPU Wait for CPU enq: TX -- row lock contention enq: TX row lock contention buffer busy waits buffer busy waits latch: cache buffers chains latch: cache buffers chains CPU CPU Application Application Concurrency Concurrency Concurrency Concurrency 67.98 67.98 23.98 23.98 4.66 4.66 2.26 2.26 8.42 8.42 2.97 2.97 0.58 0.58 0.28 0.28
- 59. Statspack AAS Look for Elapsed Time Top 5 Timed Events Start at line 52 of about 1300 10/18/13 59
- 60. Statspack AAS • Elapsed Time STATSPACK report for STATSPACK report for LABSF03 1420044432 1 10.1.0.2.0 NO labsfr LABSF03 1420044432 labsf03 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess Snap Id Snap Time Sessions Curs/Sess Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins) Elapsed: 1.00 (mins) • Look at Top 5 Timed Events Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ Event Event buffer busy waits buffer busy waits CPU time CPU time free buffer waits free buffer waits write complete waits write complete waits log buffer space log buffer space 10/18/13 Waits Waits Time (s) Time (s) 2,748 250 2,748 250 32 32 1,588 15 1,588 15 10 8 10 8 306 5 306 5 % Total % Total Call Time Call Time 78.72 78.72 10.16 10.16 4.63 4.63 2.51 2.51 1.51 1.51 60
- 61. Statspack AAS • DBTIME= CPU + WAITS – CPU = 32 – WAITS = 250+15+8+5 = 278 secs DBTIME=320 • Elapsed Time = 60 secs • 320 secs / 60 secs Top 5 Timed Events Top 5 Timed Events Event Time (s) Event Time (s) --------------------------------- --------buffer busy waits 250 buffer busy waits 250 CPU time 32 CPU time 32 free buffer waits 15 free buffer waits 15 write complete waits 8 write complete waits 8 log buffer space 5 log buffer space 5 AAS = 5.1 CPU core count = 2 61
- 62. AAS Formulas Use CPU count as yardstick: AAS < 1 Database is not blocked AAS ~= 0 Database basically idle Problems are in the APP not DB Ideal world – one database solution track CPU at OS AAS < # of CPUs CPU available Are any single sessions 100% active? • AAS > # of CPUs Could have performance problems AAS >> # of CPUS There is a bottleneck AAS > 1 still want to know if a single user is 100% active
- 64. What’s the DB Doing?! It’s 2am … your manager calls Whip out the stethoscope: AAS what the *!####!*! *? is the database doing ?! 10/18/13 Copyright 2006 Kyle Hailey 64
- 65. AAS Components Performance Page Real CPU available: Max CPU - non instance CPU Available CPU AAS: CPU + WAIT
- 67. What if AAS indicates a problem?
- 68. Latch Free Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------latch free 9,652 760 66.10 CPU time 248 21.62 PL/SQL lock timer 41 123 10.72 SQL*Net message from dblink 681 14 1.22 log file parallel write 128 1 .13 ------------------------------------------------------------- What Latch? There are 100s Latch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4 -> ordered by misses desc Latch Name Requests Misses Sleeps -------------------------- -------------- ----------- ----------cache buffers chains 8,448,787 649,484 6,930 library cache pin 8,405,896 82,915 1,427 library cache shared pool 8,435,488 55,645 58,626 7 Sleeps 1->4 -----------0/0/0/0/0 81537/1330/4 7/1/0 1,294 54375/1247/2 2/1/0 1 6/1/0/0/0
- 69. Row Locks 10g+ op 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ----------------------------------------- ------------ ----------- ------ -----enq: TX - row lock contention 59 160 2714 41.8 PL/SQL lock timer 4 117 29291 30.6 CPU time 28 7.2 buffer busy waits 1,217 18 15 4.7 log file parallel write 422 11 27 3.0 Who is waiting Who is blocking What is the SQL What is the row?
- 70. Buffer Busy Wait Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------ Buffer Busy Wait – trying to modify a block Who blocks? What object? what is the SQL? Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3 Class Waits Wait Time (s) Avg Time (ms) ------------------ ----- ------------- ------------file header block 264 203 769 data block 6,070 162 27 undo header 355 0 1 Statspack fails for analysis
- 71. CPU in Problem
- 72. OEM Per formance Page: Buf fer Busy Waits
- 73. Dilema: AAS indicates problem but not solutions Problem in •Statspack •AWR •OEM performance page Can’t analyze •Locks •Buffer Busy Waits •Latches Also can’t see when demand for CPU is greater than available CPU
- 75. Example • Look at Statspack: Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time --------------------------- ----------- --------buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 -------------------------------
- 76. What do we do? • • How do we solve “buffer busy wait” Need: • • • • SQL Sessions Objects Wait details – Type of buffer busy wait – File and block numbers • How do we get that information? – Not from Statspack or AWR – But from v$session • v$session_wait prior to 10g
- 77. V$session • Shows who is waiting – For example lists who is waiting on “buffer busy waits” – Data only exists while waits are happening – Data includes • Type of buffer busy wait • File and block involved in buffer busy wait • • Problem: Once waits are over, data is gone Solution: Sample data all the time
- 78. Sample Query select nvl(s.username,s.program) username, s.sid sid, s.serial# serial, s.sql_hash_value sql_hash_value, SQL identifier substr(decode(w.wait_time, Waiting or on CPU? 0, w.event, 'ON CPU'),1,15) event , w.p1 p1, w.p2 p2, w.p3 p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER'; This query works since v7
- 79. Sampling Output USERNAME ---------SYS SYS STARGUS (CJQ0) SID ----64 58 71 9 SERIAL -------8717 19467 6251 1 SQL_HASH_V ---------4116021597 961168820 1311875676 0 EVENT -------------------PL/SQL lock timer ON CPU direct path write rdbms ipc message Run sample query every second and save into a table “v$ash” Create table v$ash as select … ; Insert into v$ash select … ; P1 -------300 16508152 201 500 P2 -------0 1 2155902 0 P3 ---0 0 127 0
- 80. Buffer busy wait type SQL> Select count(*), p3 from v$ash where event = 'buffer busy waits' group by p3; COUNT(*) P3 ---------- ---3423 1 P3 = block type
- 81. Buffer busy wait type N CLASS Block types come from select rownum n, class from v$waitstat; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 data block sort block save undo block segment header save undo header free list extent map 1st level bmb 2nd level bmb 3rd level bmb bitmap block bitmap index block file header block unused system undo header system undo block undo header
- 82. File and Block #s select count(*), p1 filen, p2 blockn from v$ash where event='buffer busy waits' group by p1, p2, hash_value; COUNT(*) FILEN BLOCKN ---------- -------- -------1 11 90644 2 11 90651 3 11 98233 1 11 104767 3 11 113291 1 11 119842 1 11 119856 3 11 121632 1 11 126334 Pick a File and Block to find object
- 83. Find Object column segment_name format a30 select COUNT(*) FILEN BLOCKN ---------- -------- -------1 11 126334 owner, segment_name, segment_type, block_id, blocks+block_id from dba_extents where file_id = 11 and 126334 between block_id AND block_id + blocks-1; OWNER SEGMENT_NAME SEGMENT_TY BLOCK_ID BLOCKS+BLOCK_ID ---------- ------------------ ---------- ---------- --------------SYSTEM TOTO1 TABLE 125201 127249 Data block ( not header block ) 126334
- 84. What SQL ? SQL> select count(*), sql_hash_value 2 from v$ash 3 where event='buffer busy waits' 4 group by hash_value; COUNT(*) ---------3423 SQL_HASH_VALUE -------------558666863 All the same SQL SQL_HASH_VALUE= 558666863 In version 10g+ use SQL_ID
- 85. SQL Statement ? select sql_text from v$sqltext where hash_value=558666863; SQL_TEXT ----------------------------------------------------INSERT into toto1 values (:b1,lpad('a',1000,'a')) • Insert statement • Problem on a data block on • table toto1 Solution: Freelists or ASSM
- 86. Sampling Summar y • Sampling v$session – finds the root of problems – otherwise impossible to diagnose • Sample Data from v$session is called – Active Session History (ASH) – v$active_session_history Diagnosis is still complex But done automatically by ADDM
- 87. OEM & ADDM Example Four Tuning Paths Database Home Page Performance Page Drilldowns SQL Session ADDM Page Solutions
- 88. Path 1 Database Home Page Performance Page Drilldowns SQL Session ADDM Page Solutions Actions
- 90. ADDM Details Tablespace Solution ASSM Object Solution 2: Partitioning Solution 3: Free Lists
- 91. Path 2 Database Home Page Performance Page Drilldowns SQL Session ADDM Page Solutions Actions
- 93. ADDM Page
- 94. ADDM Details
- 95. EM Product Layout for Performance Database Home Page Performance Page Drilldowns SQL Session Copyright 2006 Kyle Hailey ADDM Page Solutions
- 97. Database Performance Page Lots of waits ADDM Report
- 98. ADDM Page
- 99. ADDM Details
- 100. EM Product Layout for Performance Database Home Page Performance Page Drilldowns SQL Session ADDM Page Solutions
- 101. Database Performance Page “click on the big stuff”
- 102. Wait Drilldown : Concurrency
- 103. SQL Drilldown
- 104. Session Drilldown
- 105. Session : Wait History Copyright 2006 Kyle Hailey
- 106. Part II – Taking ADDM Advice • 10 users inserting concurrently into same table insert into toto1 values (t,lpad('a',1000,'a'));
- 107. Test 1 Elapsed Time 14:44
- 108. ADDM Details • Use ASSM • Use Freelists • Partition Object => Trying out Freelists
- 109. Test 2 Elapsed Time 14:59 Was 14:44 – now slower!
- 110. ADDM Details • Increase Log File Size to 2000M to hold at least 20 minutes of redo
- 111. Test 3 14:44 Elapsed Time 1:23 10x Improvement! 14:59 Elapsed 1:23
- 112. Final ADDM Details Reduce the number of commits => Commit every 50 records instead of every record
- 113. Final • • Commit every 50 records instead of every record Got elapsed down from – 1:23 – 0:47
- 114. In Review • By using – Free Lists – Large Redo Log files • Job Processing went from – Over 14 minutes to – Just over 1 minute • Thanks to ADDM and OEM !
- 116. Measuring Active Sessions Sampling Every Second ... User 1 User 2 User 3 Time Time Fast query run rarely Query run often Slow query
- 117. Active Session idle Query 1 idle Query 2 idle Time Time Query 3 idle
- 118. Active Session idle Query 1 idle Query 2 idle Time Time CPU IO Wait Query 3 idle
- 121. Primary Fields of ASH Activity : Who, What, When, How When SAMPLE_TIME Who SESSION_ID Session SESSION_STATE State EVENT Wait SQL_ID SQL Time How What
- 123. Sampling is like taking Pictures
- 124. Graphical ASH Session 1 Session 2 Session 3 Session 4 TIME
- 125. Graph of User States
- 126. One Second Graph
- 128. 15 Second Averages
- 129. Maximum CPU Line
- 130. Idle Users
- 131. The Power DB Home ASH gives AAS Performance Based on TIME AAS=db time/elapsed time v$system_event v$sysstat Top Activity AAS=count active users /samples Based on ASH Copyright 2006 Kyle Hailey
- 133. 2. OEM 10g : Top Activity •Top Activity •Based on ASH •Enables Drilldowns •Top SQL •Top Session •Drill into a session • Stats • Raw waits • Open cursors • General info •Drill into a SQL • Stats and text • Users executing • Explain plan • Tuning options
- 134. Top Activity : Based on ASH Thanks To ASH
- 135. Statistic Lag Time ASH Real Time Wait Stats Slight lag
- 136. OEM Perf Pages Top Activity Performance SQL Session Session SQL
- 137. AAS from ASH 1. ASHRPT – Based entirely on v$active_session_history – – @?/rdbms/admin/ashrpt.sql Exec ASH_REPORT_TEXT/HTML select * from table (dbms_workload_repository.ash_report_text( (select dbid from v$database), 1, sysdate – 1/24, sysdate )) ; 10/18/13 137
- 138. ASHRPT ASH Report For TESTDB/testdb ASH Report For TESTDB/testdb DB Name DB Id Instance DB Name DB Id Instance TESTDB TESTDB CPUs CPUs Inst Num Release Inst Num Release 2371570538 testdb 2371570538 testdb SGA Size Buffer Cache SGA Size Buffer Cache RAC Host RAC Host 11 10.2.0.1.0 NO sdbe604a 10.2.0.1.0 NO sdbe604a Shared Pool ASH Buffer Size Shared Pool ASH Buffer Size 22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) Elapsed Time: 5.0 (mins) Sample Count: 3,716 Sample Count: 3,716 Average Active Sessions: 12.39 Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class %% Activity Event Event Class Activity Sessions Sessions CPU ++ Wait for CPU CPU Wait for CPU enq: TX -- row lock contention enq: TX row lock contention buffer busy waits buffer busy waits latch: cache buffers chains latch: cache buffers chains CPU CPU Application Application Concurrency Concurrency Concurrency Concurrency 67.98 67.98 23.98 23.98 4.66 4.66 2.26 2.26 8.42 8.42 2.97 2.97 0.58 0.58 0.28 0.28
- 139. ASH RPT 1) General info 1) General info 2) Top User Events *** 2) Top User Events *** 3) Top Background Events 3) Top Background Events 4) Top Event P1/P2/P3 Values 4) Top Event P1/P2/P3 Values 5) Top Service/Module 5) Top Service/Module 6) Top Client IDs 6) Top Client IDs 7) Top SQL Command Types 7) Top SQL Command Types 8) Top SQL Statements *** 8) Top SQL Statements *** 9) Top SQL using literals 9) Top SQL using literals 10) Top Sessions *** 10) Top Sessions *** 11) Top Blocking Sessions 11) Top Blocking Sessions 12) Top Sessions running PQs 12) Top Sessions running PQs 13) Top DB Objects 13) Top DB Objects 14) Top DB Files 14) Top DB Files 15) Top Latches 15) Top Latches 16) Activity Over Time *** 16) Activity Over Time ***
- 140. ASHRPT over Time • Waits over Time – Not in AAS • Difficult but better than nothing Compare to …
- 141. Custom Scripts • Hate Graphics ? – Query v$active_session_history directly – Join to dba_hist_active_sess_history for week of data • act.sql – Like top 5 timed events • Aveact.sql – Charts with text AAS by hour (15 minute, minute , etc) • Aveactn.sql – Ditto, with top 2 wait events per bucket • Following Scripts Available on http://ashmasters.com 10/18/13 141
- 142. Custom Scripts @act @act Analysis Begin Time : Analysis Begin Time : 2007-07-24 11:04:48 2007-07-24 11:04:48 Analysis End Analysis End Time : Time : 2007-07-24 11:19:45 2007-07-24 11:19:45 Start time, mins ago: 15 Start time, mins ago: 15 Request Duration : 15 Request Duration : 15 Collections : 528 Collections : 528 Data Values : 3327 Data Values : 3327 Elapsed Time: 15 mins Elapsed Time: 15 mins WAIT_EVENT WAIT_EVENT latch free latch free log buffer space log buffer space buffer busy waits buffer busy waits db file scattered read db file scattered read library cache pin library cache pin log file sync log file sync ON CPU ON CPU enqueue enqueue db file sequential read db file sequential read sum sum CNT CNT 10 10 13 13 14 14 15 15 78 78 213 213 726 726 855 855 1399 1399 % Active Ave_Act_Sess % Active Ave_Act_Sess .3 .02 .3 .02 .39 .02 .39 .02 .42 .03 .42 .03 .45 .03 .45 .03 2.34 .15 2.34 .15 6.40 .40 6.40 .40 21.82 1.38 21.82 1.38 25.70 1.62 25.70 1.62 42.05 2.65 42.05 2.65 6.30 6.30
- 143. Custom Scripts @aveact @aveact TM NPTS AVEACT GRAPH CPU WAITS TM NPTS AVEACT GRAPH CPU WAITS ---------------- ------ ------- ---------------------- ---- -------------------- ------ ------- ---------------------- ---- ----06-AUG 13:00:00 270 .33 +2 29 59 06-AUG 13:00:00 270 .33 +2 29 59 06-AUG 14:00:00 1040 2.24 ++--------2--341 1984 06-AUG 14:00:00 1040 2.24 ++--------2--341 1984 06-AUG 15:00:00 623 6.67 ++++------2---------438 3718 06-AUG 15:00:00 623 6.67 ++++------2---------438 3718 06-AUG 16:00:00 1088 2.59 ++--------2---335 2486 06-AUG 16:00:00 1088 2.59 ++--------2---335 2486 06-AUG 17:00:00 1104 1.26 ++----2 349 1043 06-AUG 17:00:00 1104 1.26 ++----2 349 1043 06-AUG 18:00:00 1093 1.38 +++---2 663 06-AUG 18:00:00 1093 1.38 +++---2 663 842 842 06-AUG 19:00:00 1012 1.74 ++------- 2 373 1388 06-AUG 19:00:00 1012 1.74 ++------- 2 373 1388 06-AUG 20:00:00 1131 .99 +---2 304 06-AUG 20:00:00 1131 .99 +---2 304 820 820 06-AUG 21:00:00 1111 1.22 ++----2 344 1012 06-AUG 21:00:00 1111 1.22 ++----2 344 1012 06-AUG 22:00:00 1010 1.66 ++------ 2 414 1259 06-AUG 22:00:00 1010 1.66 ++------ 2 414 1259 06-AUG 23:00:00 1120 1.08 +---2 298 06-AUG 23:00:00 1120 1.08 +---2 298 913 913 07-AUG 00:00:00 1024 .83 +--2 273 576 07-AUG 00:00:00 1024 .83 +--2 273 576 07-AUG 01:00:00 1006 1.74 ++------- 2 319 1428 07-AUG 01:00:00 1006 1.74 ++------- 2 319 1428 07-AUG 02:00:00 1090 2.47 ++--------2---347 2345 07-AUG 02:00:00 1090 2.47 ++--------2---347 2345 07-AUG 03:00:00 687 6.59 +++-------2---------382 4142 07-AUG 03:00:00 687 6.59 +++-------2---------382 4142 07-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 07-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 659 659 07-AUG 05:00:00 1104 3.08 +++++-----2-----1170 2226 07-AUG 05:00:00 1104 3.08 +++++-----2-----1170 2226 07-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 07-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558 558
- 144. Aveact.sql
- 145. Aveact.sql “-” = WAIT “+” = CPU which waits ? -> aveactn.sql
- 146. Aveactn.sql
- 147. Before Tuning Oracle, Tune the Machine Make sure the machine is healthy before tuning Oracle CPU => use run queue, < 2 * #CPU Memory => page out VMSTAT
- 148. Summary 1.Machine - vmstat – Memory, CPU (we can see IO response in Oracle) 1.Database - AAS – Use wait interface and graphics – Identify machine, application, database or SQL 1.SQL - VST – Indexes, stats, execution path – Visual SQL Tuning
- 150. Bibliography Refactoring SQL Applications - Stephane Faroult Troubleshooting Oracle Performance - Christian Antognini SQL Tuning - Dan Tow Oracle Wait Interface - Kirtikumar Deshpande, K. Gopalakrishnan
- 151. Books The Visual Display of Design Quantitative Information – Eduard Tufte For the Space Shuttle disaster, see http://williamwolff.org/wp-content/uploads/2013/01/tufte-challenger-1997.pdf http://www.onlineethics.org/cms/12709.aspx
- 152. Using graphics for database System Tuning SQL Tuning 152
- 153. ASSM ? select FREELISTS, TABLESPACE_NAME from dba_tables where table_name='TOTO1' and owner='SCOTT'; FREELISTS ---------1 TABLESPACE_NAME -----------------------------USERS Table only has 1 freelist The table is in tablespace Users Is it an ASSM tablespace ? Copyright 2006 Kyle Hailey
- 154. Sampling Output select tablespace_name, SEGMENT_SPACE_MANAGEMENT from dba_tablespaces; TABLESPACE_NAME -----------------------------SYSTEM UNDOTBS1 TEMP USERS TOOLS SEGMEN -----MANUAL MANUAL MANUAL MANUAL AUTO USERS tablespace is not in ASSM mode Copyright 2006 Kyle Hailey
- 155. Let’s add Freelists How many freelists? select count(*), sid, serial from v$ash where event='buffer busy waits‘ group by sid, serial; COUNT(*) SID SERIAL ---------- ----- -------7 12 79 4 14 99 4 19 770 8 20 176 Max Concurrency is 4 Freelists >= 4 Copyright 2006 Kyle Hailey
Editor's Notes
- #2: {"115":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","60":"Statspack is probably the most reliable source of performance information.\nThe statspack report, ?/rdbms/admin/spreport.sql , generates over 1000 lines of information, but the first and possible only place to go in the report is “Top 5 Timed Events”.\nIn “Top 5 Timed Events” we can determine if the database has any performance issues. If it does have performance issues, then we can find out if it’s CPU or a wait. If it’s a wait we can tune that particular wait.\n","49":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","16":"The O-rings of the solid rocket boosers were not designed to erode.\nErosion was a clue that something was wrong. \nErosion was not something from which safety could be inferred\n- Richard Feynman\n","33":"What allows Oracle, all the dials and knobs, to be the fastest, most robust database on the market also make it one of the most complicated on the market. For those few users that know and love the 1000s of performance options in Oracle, access to those performance options through sql and C code is the defacto standard. For seasoned users it is often difficult to understand why anyone would want to be limited by a GUI which can often be slower, and almost always lacking the latest commands allowed through interfaces such as SQL and C. \nOn the OEM side, when a new tuning feature in the kernel is externalized in OEM (Oracle Enterprise Manager) it is often difficult to see the forest for the tree, or where and how that new feature fits into the grand scheme of things. \n","127":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","116":"If happens a lot or for long … we’ll catch it, guaranteed\n","122":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","56":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","45":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","117":"To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches\n","2":"In Oracle Support I learned more faster than I think I could have anywhere.\nPorting gave me my first appreciation for the shareable nature of Oracle code and also a bit of disbelief that it worked as well as it did.\nOracle France gave me an opportunity to concentrate on the Oracle kernel.\nAt Oracle France I had 3 amazing experiences. First was being sent to the Europecar site where I first met a couple of the people who would later become the founding members of the Oaktable, James Morle and Anjo Kolk. The Europecar site introduced me to a fellow name Roger Sanders who first showed me the wait interface before anyone knew what it was. Roger not only used it but read it directly from shared memory without using SQL.\nSoon after Europecar I began to be sent often to benchmarks at Digital Europe. These benchmarks were some of my favorite work at Oracle. The benchmarks usually consisted of installing some unknown Oracle customer application and then having a few days to make it run as fast as possible. I first started using TCL/TK and direct shared memory access (DMA) at Digital Europe and got solid hands on tuning experience testing things like striping redo and proving it was faster long before people gave up arguing that this was bad from a theoretical point of view.\nFinally in France, my boss, Jean Yves Caleca was by far the best boss I ever had, but on top of that he was wonderful at exploring the depths of Oracle and explaining it to others, teaching me much about the internals of block structure, UNDO, REDO and Freelsits.\nI came back from France wanting to do performance work and especially graphical monitoring. The kernel performance group had limited scope in that domain, so I left for a dot com where I had my first run as the sole DBA for everything, backup, recovery, performance, installation and administration. I was called away by Quest who had my favorite performance tool Spotlight. It turns out thought that scope for expanding Spotlight was limited so I jumped at the chance in 2002 to restructure Oracle OEM. The work at OEM I’m proud of but still want to do much more to make performance tuning faster, easier and more graphical.\n","118":"To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches\nActivity can thus further be broken down into the type of activity: CPU, IO or WAIT\n","74":"This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another. \n","3":"In Oracle Support I learned more faster than I think I could have anywhere.\nPorting gave me my first appreciation for the shareable nature of Oracle code and also a bit of disbelief that it worked as well as it did.\nOracle France gave me an opportunity to concentrate on the Oracle kernel.\nAt Oracle France I had 3 amazing experiences. First was being sent to the Europecar site where I first met a couple of the people who would later become the founding members of the Oaktable, James Morle and Anjo Kolk. The Europecar site introduced me to a fellow name Roger Sanders who first showed me the wait interface before anyone knew what it was. Roger not only used it but read it directly from shared memory without using SQL.\nSoon after Europecar I began to be sent often to benchmarks at Digital Europe. These benchmarks were some of my favorite work at Oracle. The benchmarks usually consisted of installing some unknown Oracle customer application and then having a few days to make it run as fast as possible. I first started using TCL/TK and direct shared memory access (DMA) at Digital Europe and got solid hands on tuning experience testing things like striping redo and proving it was faster long before people gave up arguing that this was bad from a theoretical point of view.\nFinally in France, my boss, Jean Yves Caleca was by far the best boss I ever had, but on top of that he was wonderful at exploring the depths of Oracle and explaining it to others, teaching me much about the internals of block structure, UNDO, REDO and Freelsits.\nI came back from France wanting to do performance work and especially graphical monitoring. The kernel performance group had limited scope in that domain, so I left for a dot com where I had my first run as the sole DBA for everything, backup, recovery, performance, installation and administration. I was called away by Quest who had my favorite performance tool Spotlight. It turns out thought that scope for expanding Spotlight was limited so I jumped at the chance in 2002 to restructure Oracle OEM. The work at OEM I’m proud of but still want to do much more to make performance tuning faster, easier and more graphical.\n","119":"To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches\nActivity can thus further be broken down into the type of activity: CPU, IO or WAIT\n","15":"The O-rings of the solid rocket boosers were not designed to erode.\nErosion was a clue that something was wrong. \nErosion was not something from which safety could be inferred\n- Richard Feynman\n","4":"In Oracle Support I learned more faster than I think I could have anywhere.\nPorting gave me my first appreciation for the shareable nature of Oracle code and also a bit of disbelief that it worked as well as it did.\nOracle France gave me an opportunity to concentrate on the Oracle kernel.\nAt Oracle France I had 3 amazing experiences. First was being sent to the Europecar site where I first met a couple of the people who would later become the founding members of the Oaktable, James Morle and Anjo Kolk. The Europecar site introduced me to a fellow name Roger Sanders who first showed me the wait interface before anyone knew what it was. Roger not only used it but read it directly from shared memory without using SQL.\nSoon after Europecar I began to be sent often to benchmarks at Digital Europe. These benchmarks were some of my favorite work at Oracle. The benchmarks usually consisted of installing some unknown Oracle customer application and then having a few days to make it run as fast as possible. I first started using TCL/TK and direct shared memory access (DMA) at Digital Europe and got solid hands on tuning experience testing things like striping redo and proving it was faster long before people gave up arguing that this was bad from a theoretical point of view.\nFinally in France, my boss, Jean Yves Caleca was by far the best boss I ever had, but on top of that he was wonderful at exploring the depths of Oracle and explaining it to others, teaching me much about the internals of block structure, UNDO, REDO and Freelsits.\nI came back from France wanting to do performance work and especially graphical monitoring. The kernel performance group had limited scope in that domain, so I left for a dot com where I had my first run as the sole DBA for everything, backup, recovery, performance, installation and administration. I was called away by Quest who had my favorite performance tool Spotlight. It turns out thought that scope for expanding Spotlight was limited so I jumped at the chance in 2002 to restructure Oracle OEM. The work at OEM I’m proud of but still want to do much more to make performance tuning faster, easier and more graphical.\n","120":"WAIT_TIME is time of last wait, > 0 , means on CPU\nTIME_WAITED is the actual wait time\n"}