15 Troubleshooting tips and Tricks for Database 21c - KSAOUG
0 likes383 views
The document discusses analyzing Oracle database logs using the Trace File Analyzer (TFA) tool. It provides examples of TFA commands to search and analyze logs for specific errors or time periods. The output includes summaries of matching errors, including the number of occurrences and server names. Investigating the Attention log and using TFA can help identify and troubleshoot database issues.
![9
Example Attention Log Curated Message – CDB Warning
Copyright © 2020, Oracle and/or its affiliates
[
IMMEDIATE Parameter SGA_MAX_SIZE specified is high
CAUSE: Memory parameter specified for this instance is high
ACTION: Check alert log or trace file for more information relating to instance
configuration, reconfigure the parameter and restart the instance
CLASS: CDB Instance / CDB ADMINISTRATOR / WARNING / AL-2000
TIME: 2020-05-01T11:09:02.223-07:00
ADDITIONAL INFO: -
WARNING: SGA_MAX_SIZE (6144 MB) is too high - it should be
less than 5634 MB (80 percent of physical memory).
]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-9-320.jpg)
![11
Example Attention Log Curated Message – CDB Error
Copyright © 2020, Oracle and/or its affiliates
[
IMMEDIATE Shutting down ORACLE instance (abort) (OS id: 8394)
CAUSE: A command to shutdown the instance was executed
ACTION: Check alert log for progress and completion of command
CLASS: CDB Instance / CDB ADMINISTRATOR / ERROR / AL-1002
TIME: 2020-05-08T17:09:33.773-07:00
ADDITIONAL INFO: -
Shutdown is initiated by sqlplus@den02tlh (TNS V1-V3).
]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-11-320.jpg)
![13
Example Attention Log Curated Message – Server Warning
Copyright © 2020, Oracle and/or its affiliates
[
SOON Heavy swapping observed on system
CAUSE: Memory usage by one more application is leading to heavy swapping
ACTION: Check alert log for more information, use tools to analyze memory
usage and take action
CLASS: CDB Instance / SERVER ADMINISTRATOR / WARNING / AL-2100
TIME: 2020-05-01T11:09:02.223-07:00
ADDITIONAL INFO: -
WARNING: Heavy swapping observed on system in last 15 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-13-320.jpg)
![Investigate logs and look for errors
tfactl analyze -since 1d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440
minutes...
...
Unique error messages for last ~1 day(s)
Occurrences percent server name error
----------- ------- -------------------- -----
1 100.0% myserver1 Errors in file
/u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_12272.trc
(incident=10151):
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [],
[], [], [], []
Incident details in:
/u01/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_10151/orcl2_ora_12272_i101
51.trc
...](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-17-320.jpg)
![Investigate logs and look for errors
tfactl analyze -search "ORA-04031" -last 1d
INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440
minutes...
...
Matching regex: ORA-04031
Case sensitive: false
Match count: 1
[Source: /u01/oracle/diag/rdbms/orcl2/orcl2/trace/alert_orcl2.log, Line: 1941]
Oct 01 12:09:05 2020
Errors in file /u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_6982.trc
(incident=7665):
ORA-04031: unable to allocate bytes of shared memory ("","","","")
Incident details in:
/u01/app/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_7665/orcl2_ora_6982_i76
65.trc
...](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-18-320.jpg)
![Investigate logs and look for errors
Unique generic messages for last ~7 day(s)
Occurrences percent server name generic
----------- ------- -------------------- -----
1,504 47.9% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid
Infrastructure Managem...
487 15.5% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid
Infrastructure Managem...
336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdv, SMART
Failure: FAILURE...
336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdag, SMART
Failure: FAILURE ...
103 3.3% myhost1 myhost1 last message repeated 9 times
103 3.3% myhost1 myhost1 kernel: oracle: sending ioctl 2285 to a
partition!
...snipping for brevity...](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-21-320.jpg)
![Pattern match search output
tfactl analyze -search "ORA-" -since 7d
...
[Source: /u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/alert_RATODA1.log, Line:
9494]
Feb 25 22:00:02 2014
Errors in file
/u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/RATODA1_j003_10948.trc:
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
End automatic SQL Tuning Advisor run for special tuning task
"SYS_AUTO_SQL_TUNING_TASK”
...](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-22-320.jpg)
![tfactl diagcollect –srdc <srdc_type>
• Scans system to identify recent events
• Once the relevant event is chosen, proceeds with diagnostic collection
One command SRDC
tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] :
Enter the Database Name [<RETURN>=ALL] :
1. Oct/01/2020 05:29:58 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
2. Oct/01/2020 06:55:08 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
Selected value is : 1 (Oct/01/2020 05:29:58 )](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-51-320.jpg)
![tfactl changes
Output from host : myserver69
------------------------------
[Oct/01/2020 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024
[Oct/01/2020 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744
131259
[Oct/01/2020 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 =>
158
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-
40a3-82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Oct/01/2020 05:46:15.397]: Parameter: fs.aio-nr: Value: 119680 => 122880
[Oct/01/2020 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 =>
1562320 768555
[Oct/01/2020 05:46:15.397]: Parameter: kernel.pty.nr: Value: 19 => 18
[Oct/01/2020 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-
459e-8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
...
Has anything changed recently?](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-60-320.jpg)
![...
Output from host : myserver70
------------------------------
[Oct/01/2020 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024
[Oct/01/2020 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744
131259
[Oct/01/2020 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 =>
158
[Oct/01/2020 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9-
40a3-82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab
[Oct/01/2020 05:46:15.397]: Parameter: fs.aio-nr: Value: 119680 => 122880
[Oct/01/2020 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 =>
1562320 768555
[Oct/01/2020 05:46:15.397]: Parameter: kernel.pty.nr: Value: 19 => 18
[Oct/01/2020 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31-
459e-8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e
[Oct/01/2020 16:56:15.398]: Parameter: fs.aio-nr: Value: 97024 => 98560
Has anything changed recently?](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-61-320.jpg)
![Other Server Technology
Enterprise Manager
Data Guard
GoldenGate
Exalogic
Database areas
Errors / Corruption
Performance
Install / patching / upgrade
RAC / Grid Infrastructure
Import / Export
RMAN
Transparent Data Encryption
Storage / partitioning
Undo / auditing
Listener / naming services
Spatial / XDB
Some problem areas covered in SRDCs
Full list in documentation
Around 100 problem types covered
tfactl diagcollect –srdc <srdc_type>
[-sr <sr_number>]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-63-320.jpg)
![TFA SRDC
Manual method
Manual collection vs TFA SRDC for database performance
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR reviewing
Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
Document 1903145.1 (multiple steps)
5. Collect OSWatcher data reviewing Document 301137.1 (multiple steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic output using
Document 1064487.1 (multiple steps)
9. Check alert.log if there are any errors during the “problem” period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
1. Run
tfactl diagcollect –srdc dbperf
[-sr <sr_number>]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-64-320.jpg)
![tfactl diagcollect –srdc <srdc_type>
• Scans system to identify recent events
• Once the relevant event is chosen, proceeds with diagnostic collection
One command SRDC
tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] :
Enter the Database Name [<RETURN>=ALL] :
1. Oct/01/2020 05:29:58 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
2. Oct/01/2020 06:55:08 : [orcl2] ORA-00600: internal error code,
arguments: [600], [], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
Selected value is : 1 (Oct/01/2020 05:29:58 )](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-65-320.jpg)
![TFA can manage ADR log and trace files
tfactl managelogs <options>
–show usage #Show disk space usage per diagnostic directory for both
GI and database logs
-show variation –older <n><m|h|d> #Show disk space growth for specified period
-purge –older <n><m|h|d> #Remove ADR files older than the time specified
–gi #Restrict command to only files under the GI_BASE
–database [all | dbname] #Restrict command to only files under the database
directory
-dryrun #Use with –purge to estimate how many files will be affected and how
much disk space will be freed by a potential purge command
Manual Database Log Purge](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-69-320.jpg)
![Run a database log purge dryrun
tfactl managelogs -purge -older 30d -dryrun
Output from host : myserver74
------------------------------
Estimating files older than 30 days
Estimating purge for diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files ~ 2 files deleted , 22.58 KB freed ]
Estimating purge for diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files ~ 2 files deleted , 11.72 KB freed ]
Estimating purge for diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files ~ 2 files deleted , 21.36 KB freed ]
Estimating purge for diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files ~ 3 files deleted , 23.22 KB freed ]
Estimating purge for diagnostic destination "diag/tnslsnr/myserver74/listener" for files ~ 23 files deleted , 225.33 MB freed ]
Estimating purge for diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files ~ 73 files deleted , 517.69 KB freed ]
Estimating purge for diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files ~ 38 files deleted , 17.15 KB freed ]
Estimating purge for diagnostic destination "diag/asm/+asm/+ASM" for files ~ 0 files deleted , 0 bytes freed ]
Estimating purge for diagnostic destination "diag/asm/user_root/host_309243680_96" for files ~ 1 files deleted , 19.52 KB freed ]
Estimating purge for diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files ~ 1 files deleted , 20.25 KB freed ]
Estimating purge for diagnostic destination "diag/crs/myserver74/crs" for files ~ 40 files deleted , 219.39 MB freed ]
Estimation for Grid Infrastructure [ Files to delete : ~ 185 files | Space to be freed : ~ 445.36 MB ]
Estimating purge for diagnostic destination "diag/rdbms/cdb674/CDB674" for files ~ 27760 files deleted , 66.57 MB freed ]
Estimation for Database Home [ Files to delete : ~ 27760 files | Space to be freed : ~ 66.57 MB ]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-74-320.jpg)
![Run a database log purge
...
Grid Infrastructure [ Files deleted : 18 files | Space Freed : 253.75 KB ]
.-----------------------------------------------------------------------------------------------.
| File System Variation : /u01/app/crsusr/12.2.0/grid2 |
+--------+-----------------------------------+----------+----------+---------+----------+-------+
| State | Name | Size | Used | Free | Capacity | Mount |
+--------+-----------------------------------+----------+----------+---------+----------+-------+
| Before | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / |
| After | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / |
'--------+-----------------------------------+----------+----------+---------+----------+-------'](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-76-320.jpg)
![tail files
tfactl tail alert
Output from host : myserver69
------------------------------
/scratch/app/11.2.0.4/grid/log/myserver69/alertmyserver69.log
2020-10-01 23:28:22.532:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with
the mean cluster time. No action has been taken as the Cluster Time
Synchronization Service is running in observer mode.
2020-10-01 23:58:22.964:
[ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with
the mean cluster time. No action has been taken as the Cluster Time
Synchronization Service is running in observer mode.
...](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-78-320.jpg)
![tail files
...
/scratch/app/oragrid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log <==
Thu Oct 01 04:42:22 2020
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 2323] opening OCR file
Thu Oct 01 01:05:39 2020
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 16591] opening OCR file
Thu Oct 01 01:05:41 2020
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 16603] opening OCR file
Thu Oct 01 01:21:12 2020
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 1803] opening OCR file
Thu Oct 01 01:21:12 2020
NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 1816] opening OCR file
...](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-81-320.jpg)
![Enable CHA monitoring on RAC database with optional model
Enable CHA monitoring on RAC database with optional verbose
Command line operations
chactl monitor database –db oltpacdb [-model model_name]
chactl status –verbose
monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER
monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using
model DEFAULT_DB](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-94-320.jpg)
![Check for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS
Command line operations
chactl query diagnosis -db oltpacdb -start "2020-10-01 01:42:50" -end "2020-10-01 03:19:15"
2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected]
2020-10-01 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
2020-10-01 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]
Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were
slow because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR)
performance.
Action: Separate the control files from other database files and move them to faster disks or Solid
State Devices.
Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected
for log switch completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-95-320.jpg)
![Diagnose cluster health
chactl query diagnosis -db oltpacdb -start ”2020-10-01 01:42:50.0" -end " 2020-10-01 03:19:15.0"
2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected]
2020-10-01 02:52:15.0 Database oltpacdb DB CPU Utilization (oltpacdb_2) [detected]
2020-10-01 02:52:50.0 Database oltpacdb DB CPU Utilization (oltpacdb_1) [detected]
2020-10-01 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
2020-10-01 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]](https://image.slidesharecdn.com/15troubleshootingtipsandtricksfordatabase21c-ksaoug-211117170040/85/15-Troubleshooting-tips-and-Tricks-for-Database-21c-KSAOUG-97-320.jpg)
15 Troubleshooting tips and Tricks for Database 21c - KSAOUG
- 1. VP AIOps for the Autonomous Database Sandesh Rao KSAOUG – Oct 2021 15 Troubleshooting tips and tricks for Database 21c @sandeshr https://www.linkedin.com/in/raosandesh/ https://www.slideshare.net/SandeshRao4
- 2. Copyright © 2020, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted 2 IT Operations It’s 2am. Do I page? Who do I page? ALERT!
- 3. Welcome to the attention log
- 4. 4 CONS • Includes both critical and non-critical events • Incudes messages not intended for DBAs • Inconsistently reports severity level • Can report unintuitive cause and action • New undocumented messages in every release PRO • Destination for Important DB Events • Single file to monitor by DBAs • Many tools available to parse • Supported by TFA for generating alarms Oracle Database Alert Log Copyright © 2020, Oracle and/or its affiliates
- 5. 5 Contains only important events requiring customer attention Includes documented set of messages and attributes All Messages include these attributes: • Type • Urgency • Scope • Target User • Cause and Action • Additional debug information The Curated Solution - New 21c Attention Log Copyright © 2020, Oracle and/or its affiliates
- 6. 6 Oracle Database Attention Log Message Flow Copyright © 2020, Oracle and/or its affiliates Diagnostic Framework alert/log.xml log/attention.log attention.amb DB Component Message Definitions Attention Log Message Attention Curated Message
- 7. Copyright © 2020, Oracle and/or its affiliates 7 SCOPE 1. Session 2. Process 3. PDB-Instance 4. CDB-Instance 5. CDB-Cluster 6. PDB-Persistent 7. CDB-Persistent URGENCY 1. Immediate 2. Soon 3. Deferable 4. Info TYPE 1. Error 2. Warning 3. Notification Attention Log Curation - Message Attributes TARGET USER 1. App-Dev 2. Sec-Admin 3. Net-Admin 4. Cluster-Admin 5. PDB-Admin 6. CDB-Admin 7. Server-Admin 8. Storage-Admin 9. DataOps-Admin
- 8. 8 Example Attention Message Definition – CDB Warning Copyright © 2020, Oracle and/or its affiliates // TYPE - 1 error, 2 warning, 3 notification // URGENCY - 1 immediate, 2 soon, 3 deferable, 4 info // SCOPE - 1 session, 2 process, 3 pdb-instance, 4 cdb-instance, 5 cdb-cluster, 6 pdb- persistent, 7 cdb-persistent // TARGETUSER - 1 app-dev, 2 sec-admin, 3 net-admin, 4 cluster-admin, 5 pdb-admin, 6 cdb- admin, 7 server-admin, 8 storage-admin, 9 dataops-admin ID::2000 TYPE::2 URGENCY::1 SCOPE::4 TARGETUSER::6 TEXT::Parameter %s specified is high CAUSE::Memory parameter specified for this instance is high ACTION::Check alert log or trace file for more information relating to instance configuration, reconfigure the parameter and restart the instance STARTVERSION::21.1
- 9. 9 Example Attention Log Curated Message – CDB Warning Copyright © 2020, Oracle and/or its affiliates [ IMMEDIATE Parameter SGA_MAX_SIZE specified is high CAUSE: Memory parameter specified for this instance is high ACTION: Check alert log or trace file for more information relating to instance configuration, reconfigure the parameter and restart the instance CLASS: CDB Instance / CDB ADMINISTRATOR / WARNING / AL-2000 TIME: 2020-05-01T11:09:02.223-07:00 ADDITIONAL INFO: - WARNING: SGA_MAX_SIZE (6144 MB) is too high - it should be less than 5634 MB (80 percent of physical memory). ]
- 10. 10 Example Attention Message Definition – CDB Error Copyright © 2020, Oracle and/or its affiliates // TYPE - 1 error, 2 warning, 3 notification // URGENCY - 1 immediate, 2 soon, 3 deferable, 4 info // SCOPE - 1 session, 2 process, 3 pdb-instance, 4 cdb-instance, 5 cdb-cluster, 6 pdb- persistent, 7 cdb-persistent // TARGETUSER - 1 app-dev, 2 sec-admin, 3 net-admin, 4 cluster-admin, 5 pdb-admin, 6 cdb- admin, 7 server-admin, 8 storage-admin, 9 dataops-admin ID::1002 TYPE::1 URGENCY::1 SCOPE::4 TARGETUSER::6 TEXT::Shutting down ORACLE instance (abort) %s CAUSE::A command to shutdown the instance was executed ACTION::Check alert log for progress and completion of command STARTVERSION::21.1
- 11. 11 Example Attention Log Curated Message – CDB Error Copyright © 2020, Oracle and/or its affiliates [ IMMEDIATE Shutting down ORACLE instance (abort) (OS id: 8394) CAUSE: A command to shutdown the instance was executed ACTION: Check alert log for progress and completion of command CLASS: CDB Instance / CDB ADMINISTRATOR / ERROR / AL-1002 TIME: 2020-05-08T17:09:33.773-07:00 ADDITIONAL INFO: - Shutdown is initiated by sqlplus@den02tlh (TNS V1-V3). ]
- 12. 12 Example Attention Message Definition – Server Warning Copyright © 2020, Oracle and/or its affiliates // TYPE - 1 error, 2 warning, 3 notification // URGENCY - 1 immediate, 2 soon, 3 deferable, 4 info // SCOPE - 1 session, 2 process, 3 pdb-instance, 4 cdb-instance, 5 cdb-cluster, 6 pdb- persistent, 7 cdb-persistent // TARGETUSER - 1 app-dev, 2 sec-admin, 3 net-admin, 4 cluster-admin, 5 pdb-admin, 6 cdb- admin, 7 server-admin, 8 storage-admin, 9 dataops-admin ID::2100 TYPE::2 URGENCY::2 SCOPE::4 TARGETUSER::7 TEXT::Heavy swapping observed on system CAUSE::Memory usage by one more application is leading to heavy swapping ACTION::Check alert log for more information, use tools to analyze memory usage and take action STARTVERSION::21.1
- 13. 13 Example Attention Log Curated Message – Server Warning Copyright © 2020, Oracle and/or its affiliates [ SOON Heavy swapping observed on system CAUSE: Memory usage by one more application is leading to heavy swapping ACTION: Check alert log for more information, use tools to analyze memory usage and take action CLASS: CDB Instance / SERVER ADMINISTRATOR / WARNING / AL-2100 TIME: 2020-05-01T11:09:02.223-07:00 ADDITIONAL INFO: - WARNING: Heavy swapping observed on system in last 15 mins. Heavy swapping can lead to timeouts, poor performance, and instance eviction. ]
- 14. 14 Attention Log Use Cases – Autonomous Health Framework Copyright © 2020, Oracle and/or its affiliates Autonomous Health Framework Trace File Analyzer … … CDB DBA Sys Admin PDB DBA Attention Log Repository
- 15. 15 Attention Log Use Cases – 3rd Party Monitoring Copyright © 2020, Oracle and/or its affiliates 3rd Party Monitoring, Parsing and Routing … … Attention Log Management VCN Ops Dashboard Ops Admin attention.amb Message Definitions Runbooks
- 16. Analyze logs and look for errors using AHF
- 17. Investigate logs and look for errors tfactl analyze -since 1d INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440 minutes... ... Unique error messages for last ~1 day(s) Occurrences percent server name error ----------- ------- -------------------- ----- 1 100.0% myserver1 Errors in file /u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_12272.trc (incident=10151): ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_10151/orcl2_ora_12272_i101 51.trc ...
- 18. Investigate logs and look for errors tfactl analyze -search "ORA-04031" -last 1d INFO: analyzing all (Alert and Unix System Logs) logs for the last 1440 minutes... ... Matching regex: ORA-04031 Case sensitive: false Match count: 1 [Source: /u01/oracle/diag/rdbms/orcl2/orcl2/trace/alert_orcl2.log, Line: 1941] Oct 01 12:09:05 2020 Errors in file /u01/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_6982.trc (incident=7665): ORA-04031: unable to allocate bytes of shared memory ("","","","") Incident details in: /u01/app/oracle/diag/rdbms/orcl2/orcl2/incident/incdir_7665/orcl2_ora_6982_i76 65.trc ...
- 19. Examples tfactl analyze -since 5h #Show summary of events from alert logs, system messages in last 5 hours tfactl analyze -comp os -since 1d #Show summary of events from system messages in last 1 day tfactl analyze -search "ORA-" -since 2d #Search string ORA- in alert and system logs in past 2 days tfactl analyze -search "/Starting/c" - since 2d #Search case sensitive string "Starting" in past 2 days tfactl analyze -comp os -for ”Oct/01/2020 11" -search "." #Show all system log messages at time Oct/01/2020 11 tfactl analyze -comp osw -since 6h #Show OSWatcher Top summary in last 6 hours tfactl analyze -comp oswslabinfo -from ”Oct/01/2020 05:00:01" -to ”Oct/01/2020 06:00:01" #Show OSWatcher slabinfo summary for specified time period tfactl analyze -since 1h -type generic #Analyze all generic messages in last one hour
- 20. Investigate logs and look for errors $ ./tfactl analyze -type generic -since 7d INFO: analyzing all (Alert and Unix System Logs) logs for the last 10080 minutes... ... Total message count: 54,807, from 01-Oct-2020 02:41:34 PM PST to 08-Oct-2020 02:41:34 Messages matching last ~7 day(s): 3,139, from 02-Oct-2020 02:46:23 PM PST to 08-Oct-2020 02:41:34 last ~7 day(s) generic count: 3,139, from 06-Oct-2020 02:46:23 PM PST to 08-Oct-2020 02:41:34 last ~7 day(s) unique generic count: 94 Message types for last ~7 day(s) Occurrences percent server name type ----------- ------- -------------------- ----- 3,139 100.0% myhost1 generic ...
- 21. Investigate logs and look for errors Unique generic messages for last ~7 day(s) Occurrences percent server name generic ----------- ------- -------------------- ----- 1,504 47.9% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid Infrastructure Managem... 487 15.5% myhost1 : [crflogd(13931)]CRS-9520:The storage of Grid Infrastructure Managem... 336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdv, SMART Failure: FAILURE... 336 10.7% myhost1 myhost1 smartd[13812]: Device: /dev/sdag, SMART Failure: FAILURE ... 103 3.3% myhost1 myhost1 last message repeated 9 times 103 3.3% myhost1 myhost1 kernel: oracle: sending ioctl 2285 to a partition! ...snipping for brevity...
- 22. Pattern match search output tfactl analyze -search "ORA-" -since 7d ... [Source: /u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/alert_RATODA1.log, Line: 9494] Feb 25 22:00:02 2014 Errors in file /u01/app/oracle/diag/rdbms/ratoda/RATODA1/trace/RATODA1_j003_10948.trc: ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1" ORA-29280: invalid directory path ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436 ORA-06512: at line 1 End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK” ...
- 23. OS Watcher top data tfactl analyze -comp osw -since 6h ... statistic: t first highest (time) lowest (time) average non zero 3rd last 2nd last last trend top.cpu.util.id: % 98.0 99.7 @10:35AM 72.8 @03:11PM 97.3 2,059 95.2 96.8 96.0 -2% top.cpu.util.st: % 0.1 0.1 @09:14AM 0.0 @09:14AM 0.0 889 0.0 0.0 0.0 -100% top.cpu.util.us: % 0.1 8.8 @11:31AM 0.0 @09:14AM 0.6 1,966 4.3 0.8 3.4 3300% top.cpu.util.wa: % 1.7 18.7 @03:11PM 0.1 @10:35AM 1.1 2,059 0.3 0.4 0.4 -76% top.loadavg.last01min: 1.17 3.12 @09:44AM 0.07 @12:45PM 0.93 1,823 0.31 0.26 0.22 -81% top.loadavg.last05min: 0.94 2.26 @09:44AM 0.27 @12:45PM 0.93 1,823 0.82 0.79 0.77 -18% top.loadavg.last15min: 0.79 1.60 @09:46AM 0.44 @01:18PM 0.92 1,823 0.96 0.95 0.94 18% top.mem.buffers: k 808232 808388 @09:41AM 785608 @02:57PM 796511 2,093 785744 785744 785744 -2% top.mem.free: k 1130332 1291344 @10:02AM 927576 @09:43AM 1188576 2,093 1244020 1265248 1265188 11% top.swap.used: k 47556 48088 @03:00PM 47556 @09:14AM 47828 2,097 48088 48088 48088 1% top.tasks.running: 1 4 @12:04PM 1 @09:14AM 1 1,996 1 2 2 100% top.tasks.total: 514 527 @02:57PM 509 @09:18AM 514 1,996 518 521 520 1% top.tasks.zombie: 0 5 @11:04AM 0 @09:14AM 0 62 0 0 0 n/a top.users: 5 6 @03:00PM 5 @09:14AM 5 1,823 6 6 6 20% ...
- 24. OS Watcher slabinfo data tfactl analyze -comp oswslabinfo -from ”Oct/01/2020 05:00:01" -to ”Oct/01/2020 06:00:01" ... statistic: t first highest (time) lowest (time) average non zero 3rd last 2nd last last trend slabinfo.acfs_ccb_cache.active_objs: 4 38 @05:52AM 0 @05:01AM 10 294 3 1 8 100% slabinfo.inet_peer_cache.active_objs: 23 39 @05:59AM 23 @05:00AM 23 351 23 23 39 69% slabinfo.sigqueue.active_objs: 385 768 @05:28AM 285 @05:27AM 554 351 712 621 577 49% slabinfo.skbuff_fclone_cache.active_objs: 55 133 @05:51AM 11 @05:20AM 69 351 56 77 70 27% slabinfo.names_cache.active_objs: 126 180 @05:00AM 110 @05:23AM 146 351 171 166 156 23% slabinfo.sgpool-8.active_objs: 135 228 @05:31AM 59 @05:11AM 152 351 180 165 157 16% slabinfo.UDP.active_objs: 568 675 @05:28AM 492 @05:17AM 597 351 630 596 626 10% slabinfo.size-8192.active_objs: 174 209 @05:36AM 160 @05:14AM 181 351 205 187 188 8% slabinfo.task_delay_info.active_objs: 1477 1856 @05:28AM 1334 @05:57AM 1574 351 1529 1411 1579 6% slabinfo.pid.active_objs: 1608 1980 @05:29AM 1452 @05:21AM 1678 351 1564 1487 1689 5% slabinfo.blkdev_requests.active_objs: 720 880 @05:04AM 651 @05:54AM 745 351 707 736 761 5% slabinfo.size-256.active_objs: 1116 1305 @05:06AM 846 @05:11AM 1091 351 1245 1143 1166 4% slabinfo.ip_dst_cache.active_objs: 1497 1800 @05:28AM 1279 @05:36AM 1517 351 1594 1466 1560 4% slabinfo.sock_inode_cache.active_objs: 2168 2329 @05:11AM 2106 @05:56AM 2225 351 2322 2278 2232 2% slabinfo.size-512.active_objs: 3036 3152 @05:38AM 3007 @05:01AM 3088 351 3136 3112 3075 1% ...
- 25. How to connect to a hung database for diagnostics
- 26. How do you connect to a database when connections are hanging? • sqlplus preliminary connection will connect to database since no session is created • You will have limited access to the SGA • This will help in capturing diagnostic information like a systemstate dump • Two ways to connect to sqlplus using a preliminary connection: or sqlplus -prelim sqlplus -prelim / as sysdba SQL> set _prelim on SQL> connect / as sysdba Prelim connection established
- 27. Analyze a hung database
- 28. Always on - Enabled by default Reliably detects database hangs and deadlocks Autonomously resolves them Logs all detections and resolutions New SQL interface to configure sensitivity (Normal/High) and trace file sizes Oracle Hang Manager Session DIA0 EVALUATE DETECT ANALYZE Hung? VERIFY Victim Policy
- 29. Monitors Session snapshots for progress Evaluates potential hangs over time with based upon Wait Graphs Analyzes hang chain of sessions to identify blocker/victim Discovers blocker is located in ASM instance Requests ASM terminate session or instance relying on Flex ASM for recovery Detection and resolution is bi-directional Database Hang Management - Infrastructure Database ASM
- 30. Full Resolution Dump Trace File and DB Alert Log Audit Reports Oracle 21c Hang Manager Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc Oracle Database 12c Enterprise Edition Release 18/19c.0.0.0 - 64bit Beta With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics and Real Application Testing options Build label: RDBMS_MAIN_LINUX.X64_151013 ORACLE_HOME: …/3775268204/oracle System name: Linux Node name: slc05kyr Release: 2.6.39-400.211.1.el6uek.x86_64 Version: #1 SMP Fri Nov 15 13:39:16 PST 2020 Machine: x86_64 VM name: Xen Version: 3.4 (PVM) Instance name: hm62 Redo thread mounted by this instance: 2 Oracle process number: 19 Unix process pid: 12656, image: oracle@slc05kyr (DIA0) *** 2020-10-01T16:47:59.541509+17:00 *** SESSION ID:(96.41299) 2020-10-01T16:47:59.541519+17:00 *** CLIENT ID:() 2020-10-01T16:47:59.541529+17:00 *** SERVICE NAME:(SYS$BACKGROUND) 2020-10-01T16:47:59.541538+17:00 *** MODULE NAME:() 2020-10-01T16:47:59.541547+17:00 *** ACTION NAME:() 2020-10-01T16:47:59.541556+17:00 *** CLIENT DRIVER:() 2020-10-01T3T16:47:59.541565+17:00
- 31. Full Resolution Dump Trace File and DB Alert Log Audit Reports Oracle 21c Hang Manager 2020-10-01T16:47:59.435039+17:00 Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353): ORA-32701: Possible hangs up to hang ID=1 detected Incident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc 2020-10-01T16:47:59.506775+17:00 DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2 due to a GLOBAL, HIGH confidence hang with ID=1. Hang Resolution Reason: Automatic hang resolution was performed to free a significant number of affected sessions. DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1. In the alert log on the instance local to the session (instance 2 in this case), we see the following: 2020-10-01T16:47:59.538673+17:00 Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753): ORA-32701: Possible hangs up to hang ID=1 detected Incident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc 2020-10-01T16:48:04.222661+17:00 DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1 requested by master DIA0 process on instance 1 Hang Resolution Reason: Automatic hang resolution was performed to free a significant number of affected sessions. by terminating session sid:40 with serial # 43179 (ospid:13031)
- 32. Refer MOS Note 1428210.1 for scripts Oracle 21c Hang Manager SQL> SELECT chain_id, num_waiters, in_wait_secs, osid, blocker_osid, substr(wait_event_text,1,30) FROM v$wait_chains; 2 CHAIN_ID NUM_WAITERS IN_WAIT_SECS OSID BLOCKER_OSID SUBSTR(WAIT_EVENT_TEXT,1,30) ---------- ----------- ------------ ------------------------- ------------------------- ----------------------------- - 1 0 10198 21045 21044 enq: TX - row lock contention 1 1 10214 21044 SQL*Net message from client
- 33. Refer MOS Note 1428210.1 Oracle 21c Hang Manager SELECT * FROM (SELECT 'Current Process: '||osid W_PROC, 'SID '||i.instance_name INSTANCE, 'INST #: '||instance INST,'Blocking Process: '||decode(blocker_osid,null,'<none>',blocker_osid)|| ' from Instance '||blocker_instance BLOCKER_PROC,'Number of waiters: '||num_waiters waiters, 'Wait Event: ' ||wait_event_text wait_event, 'P1: '||p1 p1, 'P2: '||p2 p2, 'P3: '||p3 p3, 'Seconds in Wait: '||in_wait_secs Seconds, 'Seconds Since Last Wait: '||time_since_last_wait_secs sincelw, 'Wait Chain: '||chain_id ||': '||chain_signature chain_signature,'Blocking Wait Chain: '||decode(blocker_chain_id,null, '<none>',blocker_chain_id) blocker_chain FROM v$wait_chains wc, v$instance i WHERE wc.instance = i.instance_number (+) AND ( num_waiters > 0 OR ( blocker_osid IS NOT NULL AND in_wait_secs > 10 ) ) ORDER BY chain_id, num_waiters DESC) WHERE ROWNUM < 101;
- 34. Query trace files using SQL
- 35. SQL> describe V$DIAG_TRACE_FILE Name Null? Type ----------------------------------------- -------- ---------------------------- ADR_HOME VARCHAR2(444) TRACE_FILENAME VARCHAR2(68) CHANGE_TIME TIMESTAMP(3) WITH TIME ZONE MODIFY_TIME TIMESTAMP(3) WITH TIME ZONE CON_ID NUMBER V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS
- 36. SQL> describe V$DIAG_TRACE_FILE_CONTENTS Name Null? Type ----------------------------------------- -------- ---------------------------- ADR_HOME VARCHAR2(444) TRACE_FILENAME VARCHAR2(68) RECORD_LEVEL NUMBER PARENT_LEVEL NUMBER RECORD_TYPE NUMBER TIMESTAMP TIMESTAMP(3) WITH TIME ZONE PAYLOAD VARCHAR2(4000) SECTION_ID NUMBER SECTION_NAME VARCHAR2(64) COMPONENT_NAME VARCHAR2(64) OPERATION_NAME VARCHAR2(64) FILE_NAME VARCHAR2(64) FUNCTION_NAME VARCHAR2(64) LINE_NUMBER NUMBER THREAD_ID VARCHAR2(64) SESSION_ID NUMBER SERIAL# NUMBER CON_UID NUMBER CONTAINER_NAME VARCHAR2(64) CON_ID NUMBER V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS
- 37. SQL> select trace_filename from v$diag_trace_file; TRACE_FILENAME -------------------------------------------------------------------- ORCL1_mz00_21108.trc ORCL1_gcr2_16504.trc ORCL1_gcr3_12849.trc ORCL1_gcr1_28159.trc ORCL1_gcr1_27603.trc ORCL1_gcr0_29971.trc ORCL1_mz00_26487.trc ORCL1_mz00_28329.trc ORCL1_ora_19005.trc ORCL1_gcr3_12879.trc ORCL1_gcr1_11688.trc V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS
- 38. SQL> select payload from v$diag_trace_file_contents where trace_filename ='ORCL1_ora_19005.trc'; PAYLOAD -------------------------------------------------------------------------------- Trace file /u01/app/oracle/diag/rdbms/orcl_unq/ORCL1/trace/ORCL1_ora_19005.trc Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.2.0.0.0 Build label: RDBMS_19.2.0.0.0_LINUX.X64_190121 ORACLE_HOME: /u01/app/oracle/product/19c/dbhome_1 System name: Linux Node name: myserver65 Release: 4.14.35-1844.1.3.el7uek.x86_64 Version: #2 SMP Wed Jan 2 21:18:29 PST 2019 Machine: x86_64 VM name: Xen Version: 4.1 (HVM) ... V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS
- 39. ... PAYLOAD -------------------------------------------------------------------------------- Instance name: ORCL1 Redo thread mounted by this instance: 1 Oracle process number: 12 Unix process pid: 19005, image: oracle@myserver65 (TNS V1-V3) *** 2020-10-01T01:22:10.770960+00:00 *** SESSION ID:(106.17196) 2020-10-01T01:22:10.771014+00:00 *** CLIENT ID:() 2020-10-01T01:22:10.771027+00:00 *** SERVICE NAME:(SYS$USERS) 2020-10-01T01:22:10.771039+00:00 ... V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS
- 40. SQL> describe V$DIAG_SESS_SQL_TRACE_RECORDS Name Null? Type ----------------------------------------- -------- ---------------------------- ADR_HOME VARCHAR2(444) TRACE_FILENAME VARCHAR2(68) RECORD_LEVEL NUMBER PARENT_LEVEL NUMBER RECORD_TYPE NUMBER TIMESTAMP TIMESTAMP(3) WITH TIME ZONE PAYLOAD VARCHAR2(4000) SECTION_ID NUMBER SECTION_NAME VARCHAR2(64) COMPONENT_NAME VARCHAR2(64) OPERATION_NAME VARCHAR2(64) FILE_NAME VARCHAR2(64) FUNCTION_NAME VARCHAR2(64) LINE_NUMBER NUMBER THREAD_ID VARCHAR2(64) SESSION_ID NUMBER SERIAL# NUMBER CON_UID NUMBER CONTAINER_NAME VARCHAR2(64) CON_ID NUMBER V$DIAG_SESS_SQL_TRACE_RECORDS
- 41. SQL> SELECT sid,serial# FROM v$session WHERE username = 'SYS’; SID SERIAL# ---------- ---------- 33 45888 129 6051 SQL> EXECUTE DBMS_SYSTEM.SET_SQL_TRACE_IN_SESSION(129,6051,TRUE); PL/SQL procedure successfully completed. V$DIAG_SESS_SQL_TRACE_RECORDS Enable session tracing
- 42. SQL> select unique trace_filename from V$DIAG_SESS_SQL_TRACE_RECORDS; TRACE_FILENAME -------------------------------------------------------------------- ORCL1_ora_14151.trc SQL> select payload from V$DIAG_SESS_SQL_TRACE_RECORDS where trace_filename = 'ORCL1_ora_14151.trc'; PAYLOAD -------------------------------------------------------------------------------- CLOSE #140506358472544:c=19,e=18,dep=0,type=1,tim=7769230586778 ===================== PARSING IN CURSOR #140506358494608 len=97 dep=1 uid=0 oct=3 lid=0 tim=7769230600 163 hv=791757000 ad='7fa0c290' sqlid='87gaftwrm2h68' select o.owner#,o.name,o.namespace,o.remoteowner,o.linkname,o.subname from obj$ o where o.obj#=:1 END OF STMT EXEC #140506358494608:c=65,e=65,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=107238262 4,tim=7769230600159 ... V$DIAG_SESS_SQL_TRACE_RECORDS
- 43. ... PAYLOAD -------------------------------------------------------------------------------- FETCH #140506358494608:c=38,e=37,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=10723826 24,tim=7769230600324 CLOSE #140506358494608:c=5,e=4,dep=1,type=3,tim=7769230600381 EXEC #140506358494608:c=23,e=23,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=107238262 4,tim=7769230600500 FETCH #140506358494608:c=11,e=12,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=10723826 24,tim=7769230600547 ... V$DIAG_SESS_SQL_TRACE_RECORDS SQL> EXECUTE DBMS_SYSTEM.SET_SQL_TRACE_IN_SESSION(129,6051,FALSE); PL/SQL procedure successfully completed.
- 44. Keep track of the attribute of import file pre and post patching
- 45. Start tracking using –fileattr start Automatically discovers Grid Infrastructure and Database directories and files • Prevent discovery using –excludediscovery Further configure the list of monitored directories using –includedir Track attribute changes on important files tfactl <orachk|exachk> -fileattr start -includedir "/root/myapp/config" ... List of directories(recursive) for checking file attributes: /u01/app/oradb/product/11.2.0/dbhome_11203 /u01/app/oradb/product/11.2.0/dbhome_11204 /root/myapp/config orachk has taken snapshot of file attributes for above directories at: /orahome/oradb/orachk/orachk_mysrv21_20201001_041214
- 46. Compare current attributes against first snapshot using –fileattr check When checking, use the same include/exclude arguments you started with Track attribute changes on important files tfactl <orachk|exachk> -fileattr check -includedir "/root/myapp/config” ... List of directories(recursive) for checking file attributes: /u01/app/oradb/product/11.2.0/dbhome_11203 /u01/app/oradb/product/11.2.0/dbhome_11204 /root/myapp/config Checking file attribute changes... "/root/myapp/config/myappconfig.xml" is different: Baseline : 0644 oracle root /root/myapp/config/myappconfig.xml Current : 0644 root root /root/myapp/config/myappconfig.xml ...
- 47. Automatically proceeds to run compliance checks after file attribute checks • Only run attribute checks by using -fileattronly File Attribute Changes are shown in HTML report output Track attribute changes on important files
- 48. Self analysis in MOS using TFA collections
- 51. tfactl diagcollect –srdc <srdc_type> • Scans system to identify recent events • Once the relevant event is chosen, proceeds with diagnostic collection One command SRDC tfactl diagcollect -srdc ORA-00600 Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] : Enter the Database Name [<RETURN>=ALL] : 1. Oct/01/2020 05:29:58 : [orcl2] ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [], [], [], [], [] 2. Oct/01/2020 06:55:08 : [orcl2] ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [], [], [], [], [] Please choose the event : 1-2 [1] Selected value is : 1 (Oct/01/2020 05:29:58 )
- 52. All required files are identified • Trimmed where applicable • Package in a zip ready to provide to support One command SRDC ... 2020/10/01 06:14:24 EST : Getting List of Files to Collect 2020/10/01 06:14:27 EST : Trimming file : myserver1/rdbms/orcl2/orcl2/trace/orcl2_lmhb_3542.trc with original file size : 163MB ... 2020/10/01 06:14:58 EST : Total time taken : 39s 2020/10/01 06:14:58 EST : Completed collection of zip files. ... /opt/oracle.ahf/data/repository/srdc_ora600_collection_Tue_Sep_7_06_14_17 _EST_2020_node_local/myserver1.tfa_srdc_ora600_Thu_Oct_1_06_14_17_EST_202 0.zip
- 55. Sensitive information can be hidden from diagnostics Machine learning algorithms determine sensitive data like: • Host names • IP addresses • MAC addresses • Oracle Database names • Tablespace names • Service names • Ports • Operating system user names Sanitize or mask sensitive information
- 56. Add –sanitize or –mask to any command • –sanitize replaces a sensitive value with random characters • myhost123 >>>> JnsF3km9 • –mask replaces a sensitive value with a series of ‘X’ • myhost123 >>>> XXXXXXXX Sanitize or mask sensitive information
- 57. Sanitized hostname Sanitized hostname tfactl orachk –preupgrade -sanitize
- 58. tfactl orachk -rmap qzh024703246tsa1 TFA using ORAchk : /opt/oracle.ahf/orachk/orachk ___________________________________________________________________________ | Entity Type | Substituted Entity Name | Original Entity Name | ___________________________________________________________________________ | hostname | qzh024703246tsa1 | myserver1 | ___________________________________________________________________________ Reverse map the sanitization
- 59. Find if anything has changed
- 60. tfactl changes Output from host : myserver69 ------------------------------ [Oct/01/2020 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024 [Oct/01/2020 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744 131259 [Oct/01/2020 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1 [Oct/01/2020 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158 [Oct/01/2020 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9- 40a3-82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab [Oct/01/2020 05:46:15.397]: Parameter: fs.aio-nr: Value: 119680 => 122880 [Oct/01/2020 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 => 1562320 768555 [Oct/01/2020 05:46:15.397]: Parameter: kernel.pty.nr: Value: 19 => 18 [Oct/01/2020 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31- 459e-8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e ... Has anything changed recently?
- 61. ... Output from host : myserver70 ------------------------------ [Oct/01/2020 04:54:15.397]: Parameter: fs.aio-nr: Value: 95488 => 97024 [Oct/01/2020 04:54:15.397]: Parameter: fs.inode-nr: Value: 764974 131561 => 740744 131259 [Oct/01/2020 04:54:15.397]: Parameter: kernel.pty.nr: Value: 2 => 1 [Oct/01/2020 04:54:15.397]: Parameter: kernel.random.entropy_avail: Value: 189 => 158 [Oct/01/2020 04:54:15.397]: Parameter: kernel.random.uuid: Value: 36269877-9bc9- 40a3-82e0-1619865096f2 => 7551c5e7-c59f-40fa-b55f-5bd170e8b1ab [Oct/01/2020 05:46:15.397]: Parameter: fs.aio-nr: Value: 119680 => 122880 [Oct/01/2020 05:46:15.397]: Parameter: fs.inode-nr: Value: 1580316 810036 => 1562320 768555 [Oct/01/2020 05:46:15.397]: Parameter: kernel.pty.nr: Value: 19 => 18 [Oct/01/2020 05:46:15.397]: Parameter: kernel.random.uuid: Value: 37cc31aa-ee31- 459e-8f2a-0766b34b1b64 => f5176cdc-6390-415d-882e-02c4cff2ae4e [Oct/01/2020 16:56:15.398]: Parameter: fs.aio-nr: Value: 97024 => 98560 Has anything changed recently?
- 62. Detect and collect using SRDC
- 63. Other Server Technology Enterprise Manager Data Guard GoldenGate Exalogic Database areas Errors / Corruption Performance Install / patching / upgrade RAC / Grid Infrastructure Import / Export RMAN Transparent Data Encryption Storage / partitioning Undo / auditing Listener / naming services Spatial / XDB Some problem areas covered in SRDCs Full list in documentation Around 100 problem types covered tfactl diagcollect –srdc <srdc_type> [-sr <sr_number>]
- 64. TFA SRDC Manual method Manual collection vs TFA SRDC for database performance 1. Generate ADDM reviewing Document 1680075.1 (multiple steps) 2. Identify “good” and “problem” periods and gather AWR reviewing Document 1903158.1 (multiple steps) 3. Generate AWR compare report (awrddrpt.sql) using “good” and “problem” periods 4. Generate ASH report for “good” and “problem” periods reviewing Document 1903145.1 (multiple steps) 5. Collect OSWatcher data reviewing Document 301137.1 (multiple steps) 6. Collect Hang Analyze output at Level 4 7. Generate SQL Healthcheck for problem SQL id using Document 1366133.1 (multiple steps) 8. Run support provided sql scripts – Log File sync diagnostic output using Document 1064487.1 (multiple steps) 9. Check alert.log if there are any errors during the “problem” period 10. Find any trace files generated during the “problem” period 11. Collate and upload all the above files/outputs to SR 1. Run tfactl diagcollect –srdc dbperf [-sr <sr_number>]
- 65. tfactl diagcollect –srdc <srdc_type> • Scans system to identify recent events • Once the relevant event is chosen, proceeds with diagnostic collection One command SRDC tfactl diagcollect -srdc ORA-00600 Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] : Enter the Database Name [<RETURN>=ALL] : 1. Oct/01/2020 05:29:58 : [orcl2] ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [], [], [], [], [] 2. Oct/01/2020 06:55:08 : [orcl2] ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], [], [], [], [], [] Please choose the event : 1-2 [1] Selected value is : 1 (Oct/01/2020 05:29:58 )
- 66. All required files are identified • Trimmed where applicable • Package in a zip ready to provide to support One command SRDC ... 2020/10/01 06:14:24 EST : Getting List of Files to Collect 2020/10/01 06:14:27 EST : Trimming file : myserver1/rdbms/orcl2/orcl2/trace/orcl2_lmhb_3542.trc with original file size : 163MB ... 2020/10/01 06:14:58 EST : Total time taken : 39s 2020/10/01 06:14:58 EST : Completed collection of zip files. ... /opt/oracle.ahf/data/repository/srdc_ora600_collection_Thu_Oct_1_06_14_17 _EST_2020_node_local/myserver1.tfa_srdc_ora600_Thu_Oct_1_06_14_17_EST_202 0.zip
- 67. Manage logs
- 68. TFA can automatically purge database logs Purging automatically removes logs older than 30 days • Configurable with Purging runs every 60 minutes • Configurable with: Automatic Database Log Purge tfactl set manageLogsAutoPurge=ON tfactl set manageLogsAutoPurgePolicyAge=<n><d|h> tfactl set manageLogsAutoPurgeInterval=<minutes>
- 69. TFA can manage ADR log and trace files tfactl managelogs <options> –show usage #Show disk space usage per diagnostic directory for both GI and database logs -show variation –older <n><m|h|d> #Show disk space growth for specified period -purge –older <n><m|h|d> #Remove ADR files older than the time specified –gi #Restrict command to only files under the GI_BASE –database [all | dbname] #Restrict command to only files under the database directory -dryrun #Use with –purge to estimate how many files will be affected and how much disk space will be freed by a potential purge command Manual Database Log Purge
- 70. tfactl managelogs -show usage ... .---------------------------------------------------------------------------------. | Grid Infrastructure Usage | +---------------------------------------------------------------------+-----------+ | Location | Size | +---------------------------------------------------------------------+-----------+ | /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/alert | 28.00 KB | | /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/incident | 4.00 KB | | /u01/app/crsusr/diag/afdboot/user_root/host_309243680_94/trace | 8.00 KB | ... +---------------------------------------------------------------------+-----------+ | Total | 739.06 MB | '---------------------------------------------------------------------+-----------’ ... Understand Database log disk space usage Use -gi to only show grid infrastructure
- 71. ... .---------------------------------------------------------------. | Database Homes Usage | +---------------------------------------------------+-----------+ | Location | Size | +---------------------------------------------------+-----------+ | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB | | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB | | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 146.19 MB | | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB | | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB | +---------------------------------------------------+-----------+ | Total | 147.26 MB | '---------------------------------------------------+-----------' Understand Database log disk space usage Use -database to only show database
- 72. Understand Database log disk space usage variations tfactl managelogs -show variation -older 30d Output from host : myserver74 ------------------------------ 2020-10-01 12:30:42: INFO Checking space variation for 30 days .---------------------------------------------------------------------------------------------. | Grid Infrastructure Variation | +---------------------------------------------------------------------+-----------+-----------+ | Directory | Old Size | New Size | +---------------------------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/asm/user_root/host_309243680_96/alert | 22.00 KB | 28.00 KB | +---------------------------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/clients/user_crsusr/host_309243680_96/cdump | 4.00 KB | 4.00 KB | +---------------------------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/tnslsnr/myserver74/listener/alert | 15.06 MB | 244.10 MB | +---------------------------------------------------------------------+-----------+-----------+ ...
- 73. Understand Database log disk space usage variations ... .---------------------------------------------------------------------------. | Database Homes Variation | +---------------------------------------------------+-----------+-----------+ | Directory | Old Size | New Size | +---------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/hm | 4.00 KB | 4.00 KB | +---------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/trace | 16.63 MB | 146.19 MB | +---------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/cdump | 4.00 KB | 4.00 KB | +---------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/incident | 4.00 KB | 4.00 KB | +---------------------------------------------------+-----------+-----------+ | /u01/app/crsusr/diag/rdbms/cdb674/CDB674/alert | 1.06 MB | 1.06 MB | '------------------------------------------------------------+-------------+-------------'
- 74. Run a database log purge dryrun tfactl managelogs -purge -older 30d -dryrun Output from host : myserver74 ------------------------------ Estimating files older than 30 days Estimating purge for diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files ~ 2 files deleted , 22.58 KB freed ] Estimating purge for diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files ~ 2 files deleted , 11.72 KB freed ] Estimating purge for diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files ~ 2 files deleted , 21.36 KB freed ] Estimating purge for diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files ~ 3 files deleted , 23.22 KB freed ] Estimating purge for diagnostic destination "diag/tnslsnr/myserver74/listener" for files ~ 23 files deleted , 225.33 MB freed ] Estimating purge for diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files ~ 73 files deleted , 517.69 KB freed ] Estimating purge for diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files ~ 38 files deleted , 17.15 KB freed ] Estimating purge for diagnostic destination "diag/asm/+asm/+ASM" for files ~ 0 files deleted , 0 bytes freed ] Estimating purge for diagnostic destination "diag/asm/user_root/host_309243680_96" for files ~ 1 files deleted , 19.52 KB freed ] Estimating purge for diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files ~ 1 files deleted , 20.25 KB freed ] Estimating purge for diagnostic destination "diag/crs/myserver74/crs" for files ~ 40 files deleted , 219.39 MB freed ] Estimation for Grid Infrastructure [ Files to delete : ~ 185 files | Space to be freed : ~ 445.36 MB ] Estimating purge for diagnostic destination "diag/rdbms/cdb674/CDB674" for files ~ 27760 files deleted , 66.57 MB freed ] Estimation for Database Home [ Files to delete : ~ 27760 files | Space to be freed : ~ 66.57 MB ]
- 75. Run a database log purge tfactl managelogs -purge -older 30d Output from host : myserver74 ------------------------------ Purging files older than 30 days Cleaning Grid Infrastructure destinations Purging diagnostic destination "diag/afdboot/user_root/host_309243680_94" for files - 0 files deleted , 0 bytes freed Purging diagnostic destination "diag/afdboot/user_crsusr/host_309243680_94" for files - 1 files deleted , 10.16 KB freed Purging diagnostic destination "diag/asmtool/user_root/host_309243680_96" for files - 1 files deleted , 10.16 KB freed Purging diagnostic destination "diag/asmtool/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/tnslsnr/myserver74/listener" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/diagtool/user_root/adrci_309243680_96" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/clients/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/asm/+asm/+ASM" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/asm/user_root/host_309243680_96" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/asm/user_crsusr/host_309243680_96" for files - 2 files deleted , 29.18 KB freed Purging diagnostic destination "diag/crs/myserver74/crs" for files - 2 files deleted , 29.18 KB freed ...
- 76. Run a database log purge ... Grid Infrastructure [ Files deleted : 18 files | Space Freed : 253.75 KB ] .-----------------------------------------------------------------------------------------------. | File System Variation : /u01/app/crsusr/12.2.0/grid2 | +--------+-----------------------------------+----------+----------+---------+----------+-------+ | State | Name | Size | Used | Free | Capacity | Mount | +--------+-----------------------------------+----------+----------+---------+----------+-------+ | Before | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / | | After | /dev/mapper/vg_rws1270665-lv_root | 51475068 | 46597152 | 2256476 | 96% | / | '--------+-----------------------------------+----------+----------+---------+----------+-------'
- 78. tail files tfactl tail alert Output from host : myserver69 ------------------------------ /scratch/app/11.2.0.4/grid/log/myserver69/alertmyserver69.log 2020-10-01 23:28:22.532: [ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode. 2020-10-01 23:58:22.964: [ctssd(5630)]CRS-2409:The clock on host myserver69 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode. ...
- 79. tail files ... /scratch/app/oradb/diag/rdbms/apxcmupg/apxcmupg_2/trace/alert_apxcmupg_2.log Thu Oct 01 06:00:00 2020 VKRM started with pid=82, OS id=4903 Thu Oct 01 06:00:02 2020 Begin automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK" Thu Oct 01 06:00:37 2020 End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK" Thu Oct 01 23:00:28 2020 Thread 2 advanced to log sequence 759 (LGWR switch) Current log# 3 seq# 759 mem# 0: +DATA/apxcmupg/onlinelog/group_3.289.917164707 Current log# 3 seq# 759 mem# 1: +FRA/apxcmupg/onlinelog/group_3.289.917164707 ...
- 80. tail files ... /scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/trace/alert_ogg112041.log Clearing Resource Manager plan via parameter Thu Oct 01 05:59:59 2020 Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter Thu Oct 01 05:59:59 2020 Starting background process VKRM Thu Oct 01 05:59:59 2020 VKRM started with pid=36, OS id=4901 Thu Oct 01 22:00:31 2020 Thread 1 advanced to log sequence 305 (LGWR switch) Current log# 1 seq# 305 mem# 0: +DATA/ogg11204/redo01.log ...
- 81. tail files ... /scratch/app/oragrid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log <== Thu Oct 01 04:42:22 2020 NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 2323] opening OCR file Thu Oct 01 01:05:39 2020 NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 16591] opening OCR file Thu Oct 01 01:05:41 2020 NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 16603] opening OCR file Thu Oct 01 01:21:12 2020 NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 1803] opening OCR file Thu Oct 01 01:21:12 2020 NOTE: [ocrcheck.bin@myserver69 (TNS V1-V3) 1816] opening OCR file ...
- 83. Near real-time Database monitoring • Single instance & RAC • Monitoring current database activities • Database performance • Identifying contentions and bottleneck • Process & SQL Monitoring • Real time wait events • Active Data Guard support • Multitenant Database (CDB) support oratop (Support Tools Bundle)
- 84. Monitor Database performance tfactl run oratop -database ogg19c
- 85. Section 1 DATABASE: Global database information Section 2 INSTANCE: Database instance Activity Section 3 EVENT: AWR like “Top 5 Timed Events“ Section 4 PROCESS | SQL: Processes or SQL mode information Monitor Database performance more info 1500864.1
- 87. Generates view of Cluster and Database diagnostic metrics • Always on - Enabled by default • Provides Detailed OS Resource Metrics • Assists Node eviction analysis • Locally logs all process data • User can define pinned processes • Listens to CSS and GIPC events • Categorizes processes by type • Supports plug-in collectors (ex. traceroute, netstat, ping, etc.) • New CSV output for ease of analysis Cluster Health Monitor (CHM) GIMR ologgerd (master) osysmond osysmond osysmond osysmond 12c Grid Infrastructure Management Repository OS Data OS Data OS Data OS Data
- 88. Cluster Health Monitor (CHM) Confidential – Oracle Internal/Restricted/Highly Restricted Oclumon CLI or full integration with EM Cloud Control
- 89. Always on - Enabled by default Detects node and database performance problems Provides early-warning alerts and corrective action Supports on-site calibration to improve sensitivity Integrated into EMCC Incident Manager and notifications Standalone Interactive GUI Tool Cluster Health Advisor (CHA)* OS Data GIMR ochad DB Data CHM Node Health Prognostics Engine Database Health Prognostics Engine * Requires and Included with RAC or R1N License
- 90. Choosing a Data Set for Calibration – Defining “normal” Calibrating CHA to your RAC deployment chactl query calibration –cluster –timeranges ‘start=2020-10-01 07:00:00,end=2020-10-01 13:00:00’ Cluster name : mycluster Start time : 2020-10-01 07:00:00 End time : 2020-10-01 13:00:00 Total Samples : 11524 Percentage of filtered data : 100% 1) Disk read (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.11 0.00 2.62 0.00 114.66 <25 <50 <75 <100 >=100 99.87% 0.08% 0.00% 0.02% 0.03% ...
- 91. Choosing a Data Set for Calibration – Defining “normal” Calibrating CHA to your RAC deployment ... 2) Disk write (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.01 0.00 0.15 0.00 6.77 <50 <100 <150 <200 >=200 100.00% 0.00% 0.00% 0.00% 0.00% ...
- 92. Choosing a Data Set for Calibration – Defining “normal” Calibrating CHA to your RAC deployment ... 3) Disk throughput (ASM) (IO/sec) MEAN MEDIAN STDDEV MIN MAX 2.20 0.00 31.17 0.00 1100.00 <5000 <10000 <15000 <20000 >=20000 100.00% 0.00% 0.00% 0.00% 0.00% 4) CPU utilization (total) (%) MEAN MEDIAN STDDEV MIN MAX 9.62 9.30 7.95 1.80 77.90 <20 <40 <60 <80 >=80 92.67% 6.17% 1.11% 0.05% 0.00% ...
- 93. Create and store a new model Begin using the new model Confirm the new model is working Calibrating CHA to your RAC deployment chactl query calibrate cluster –model daytime –timeranges ‘start=2020-10-01 07:00:00, end= 2020-10-01 13:00:00’ chactl monitor cluster –model daytime chactl status –verbose monitoring nodes svr01, svr02 using model daytime monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
- 94. Enable CHA monitoring on RAC database with optional model Enable CHA monitoring on RAC database with optional verbose Command line operations chactl monitor database –db oltpacdb [-model model_name] chactl status –verbose monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
- 95. Check for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS Command line operations chactl query diagnosis -db oltpacdb -start "2020-10-01 01:42:50" -end "2020-10-01 03:19:15" 2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected] 2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected] 2020-10-01 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected] 2020-10-01 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected] Problem: DB Control File IO Performance Description: CHA has detected that reads or writes to the control files are slower than expected. Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were slow because of an increase in disk IO. The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance. Action: Separate the control files from other database files and move them to faster disks or Solid State Devices. Problem: DB Log File Switch Description: CHA detected that database sessions are waiting longer than expected for log switch completions. Cause: The Cluster Health Advisor (CHA) detected high contention during log switches because the redo log files were small and the redo logs switched frequently. Action: Increase the size of the redo logs.
- 96. HTML diagnostic health output available (-html <file_name>) Command line operations
- 97. Diagnose cluster health chactl query diagnosis -db oltpacdb -start ”2020-10-01 01:42:50.0" -end " 2020-10-01 03:19:15.0" 2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected] 2020-10-01 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected] 2020-10-01 02:52:15.0 Database oltpacdb DB CPU Utilization (oltpacdb_2) [detected] 2020-10-01 02:52:50.0 Database oltpacdb DB CPU Utilization (oltpacdb_1) [detected] 2020-10-01 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected] 2020-10-01 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]
- 98. How to view Patch Details in Autonomous DB
- 99. View Patch Details View Autonomous Database maintenance event history to see details about past maintenance events (requires ADMIN user) OVERVIEW
- 100. View Patch Details HOW IT WORKS SELECT * FROM DBA_CLOUD_PATCH_INFO; SELECT * FROM DBA_CLOUD_PATCH_INFO WHERE PATCH_VERSION = 'ADBS-21.7.1.2';
- 101. Performance monitoring with Autonomous Database
- 102. • Monitor health, capacity, performance of ADB instances • Uses metrics, alarms, and notifications • Metrics accessible via OCI console or using APIs Monitor Performance with ADB Metrics
- 103. 1. CPU Utilization 2. Memory Utilization 3. Sessions 4. Failed Connections 5. Execute Count 6. Queued Statements 7. Running Statements 8. Failed Logons 9. Current Logons 10. Transaction Count 11. User Calls 12. Parse Count (Total) Available Service Metrics
- 104. Autonomous Database Details page provides top 6 view of library of service metrics. Viewing Top 6 Metrics on ADB Console
- 105. Complete library of only metrics available via the OCI Console Service Metrics page or by using the Monitoring API Viewing Full Library Database Metrics
- 106. Setting Alarms with the Autonomous Database
- 107. Step 1 – Select the required compartment Setting Up Monitoring Page
- 108. Step 2 – Select the metric namespace for autonomous database Setting Up Monitoring Page
- 109. How To Create A New Alarm
- 124. Performance Hub on Autonomous Database
- 126. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
- 127. 127 Confidential – Oracle Internal Time Range selector is displayed on the top of the Performance Hub page Use the Select Duration field to set the time duration Default, Last 60 mins is selected • Specify to view Last 8 hours, Last 24 hours, Last week • Specify a custom time range Time Range field shows active sessions in chart form. • Active sessions chart displays avg number of active sessions broken down by CPU, User I/O, and Wait. Time Range field and time slider
- 128. 128 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted A time zone control has been added to Performance Hub. This will allow users to view data using an alternate time zone to UTC. They will have the choice of viewing data using either UTC (default), the time zone of their browser client, or the time zone of the database. Timezone Selector
- 130. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
- 131. 131 Confidential – Oracle Internal Shows Active Session History (ASH) analytics charts to explore ASH data Drill down into database performance across multiple dimensions such as Consumer Group, Wait Class, SQL ID, and User Name Select an Average Active Sessions dimension and view the top activity for that dimension for the selected time period. For information on ASH, see Active Session History (ASH) in Oracle Database Concepts. Active Session History (ASH) Analytics
- 132. 132 Confidential – Oracle Internal SQL only monitored if running for at least five seconds or run in parallel Displays monitored SQL statement executions by dimensions including Last Active Time, CPU Time, and Database Time Displays currently running SQL statements and SQL statements that completed, failed, or were terminated. Information includes Status, Duration, and SQL ID • Status column has the following icons: • A spinning icon indicates that the SQL statement is executing. • A green check indicates SQL statement completed • A red cross icon indicates that the SQL statement did not complete • A clock icon indicates that the SQL statement is queued SQL Monitoring
- 141. View of a slightly busier system…..
- 172. 172 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted Performance Hub has been expanded to include a new tab, "Workload". This can be used to monitor the workload in the database and helps the user identify performance spikes and bottlenecks. Workload Metrics on Performance Hub There are 4 regions: Each region contains a group of charts that indicate the characteristics of the workload and the distribution of the resources, as shown
- 173. Download AWR reports from Performance Hub
- 174. Copyright © 2019 Oracle and/or its affiliates. 1. Open the navigation menu. Under Database, click Autonomous Transaction Processing or Autonomous Data Warehouse. 2. Choose your Compartment. 3. In the list of Autonomous Databases, click the display name of the database you want to analyze using Performance Hub reports. 4. Click Performance Hub. 1) Navigate to Performance Hub in Oracle Cloud Infrastructure Console
- 175. 175 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted 2) Adjust the Time Range selector
- 176. 176 Confidential – © 2019 Oracle Internal/Restricted/Highly Restricted 2. Click the Reports drop-down to display a list of available reports. 2. Click AWR to display the Generate AWR Report dialog. 3. You can choose to generate a report either from two snapshots closest to the current time and date or from a custom time range of your choice. • If you choose the default time range, then select Use the two snapshots closest to date and click Download. • If you choose to generate a report from a custom time range, then select Custom and select start and end times for your range. Click Download. 2. Oracle Database generates a report named AWRReport_date_range.html that downloads to the default download folder for your browser. View the report after the download completes. 3) Download the AWR Report
- 177. SQL to access Tenancy Details in Cloud
- 178. 178 When you file a service request for Autonomous Database, you need to provide the tenancy details for your instance. Tenancy details for the instance are available on the Oracle Cloud Infrastructure console. However, if you are connected to the database, you can now obtain these details by querying the CLOUD_IDENTITY column of the V$PDBS view. For example: ...will generate something similar to the following: How to get the tenancy details for your instance SELECT cloud_identity FROM v$pdbs; {"DATABASE_NAME" : "DBxxxxxxxxxxxx", "REGION" : "us-phoenix-1", "TENANT_OCID" : "OCID1.TENANCY.REGION1..ID1", "DATABASE_OCID" : "OCID1.AUTONOMOUSDATABASE.OC1.SEA.ID2 ", "COMPARTMENT_OCID" : "ocid1.tenancy.region1..ID3"}
- 179. VP AIOps for the Autonomous Database Sandesh Rao KSAOUG – Oct 2021 Thank you @sandeshr https://www.linkedin.com/in/raosandesh/ https://www.slideshare.net/SandeshRao4