SlideShare a Scribd company logo

Cluster Health Advisor (CHA) Deep Dive by Mark Scardina

•
Markus Michalewicz, profile picture
Markus Michalewicz

The document provides an overview of Oracle's Cluster Health Advisor (CHA) within the Oracle Database 12c, detailing its monitoring capabilities for Real Application Clusters (RAC) and its functionality in diagnosing and resolving database issues. It covers the architecture of CHA, including its method of detecting potential issues, delivering notifications, and executing corrective actions. The content also explains using command line and GUI tools for operations, as well as guidance for modeling and analysis in a RAC environment.

Software
1 of 46
Downloaded 109 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Most read
19
20
21
22
23
24
Most read
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Most read
43
44
45
46
Cluster Health Advisor (CHA) Deep Dive by Mark Scardina
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Mark  V.  Scardina  –  Director  of  Product  Management   Oracle  Autonomous  Health  Framework  and     Oracle  Quality  of  Service  Management   Oracle  Database  12c  Rel.  2   Cluster  Health  Advisor  -­‐  Deep  Dive   How  it  Works  and  How  to  Use  It  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line   Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons     Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   3   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line   Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons     Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   4   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Oracle  Autonomous  Health  Framework   •  Integrates  next  generaMon  tools  running   as  components  -­‐  24/7     •  Discovers  PotenMal  Issues  and  NoMfies   or  takes  CorrecMve  AcMons   •  Speeds  up  Issue  Diagnosis  and  Recovery   •  Preserves  Database  and  Server   Availability  and  Performance   •  Autonomously  Monitors  and  Manages   resources  to  maintain  SLAs   5   Working  for  You  ConInuously  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Discovers  PotenMal  Cluster   &  DB  Problems  -­‐  NoMfies   with  CorrecMve  AcMons   6   Oracle  12c  Cluster  Health  Advisor  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Introducing  Oracle  12c  Cluster  Health  Advisor   •  Real  Mme  monitoring  of  Oracle  RAC  database  systems  and  their  hosts   •  Early  detecMon  of  impending  as  well  as  ongoing  system  faults   •  Diagnoses  and  idenMfies  the  most  likely  root  causes   •  Provides  correcMve  acMons  for  targeted  triage.   •  Generates  alerts  and  noMficaMons  for  rapid  recovery   7   ProacIve  Health  PrognosIcs  System    
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   8   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  (CHA)  Architecture  Overview   9   OS  Data   GIMR   ochad   DB  Data   CHM   Node   Health   PrognosIcs   Engine   Database   Health   PrognosIcs   Engine   OS   Model   DB   Model   •  cha  –  Cluster  node  resource     •  Single  Java  ochad  daemon  per  node   •  Reads  Cluster  Health  Monitor  data     directly  from  memory   •  Reads  DB  ASH  data  from  SMR  w/o  DB  connecMon   •  Uses  OS  and  DB  models  and  data  to  perform   prognosMcs   •  Stores  analysis  and  evidence  in  the  GI   Management  Repository   •  Sends  alerts  to  EMCC  Incident  Manager  per   target   EMCC   Alert  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  -­‐  Scope  of  Problem  DetecMon   •  Over  30  node  and  database  problems  have  been  modeled   •  Over  150  OS  and  DB  metric  predictors  idenMfied   •  Problem  DetecMon  in  12.2.0.1  includes   – Interconnect  ,  Global  Cache  and  Cluster  Problems   – Host  CPU  and  Memory  ,  PGA  Memory  stress     – IO  and  Storage  Performance  issues   – ReconfiguraMon  and  Recovery  issues   – Workload  and  Session  abnormal  variaMons     10   Best  Effort  Immediate  Guided  Diagnosis    
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   11   Data  Sources  and  Data  Points     Cluster  Health  Advisor   Time   CPU   ASM     IOPS     Network   %  uIl   Network_ Packets   Dropped   Log   file   sync   Log  file   parallel   write   GC    CR   request   GC  current     request   GC  current   block  2-­‐way   GC  current   block  busy   Enq:  CF     -­‐ conten Ion   … 15:16:00   0.90   4100   13%   0   2  ms   600  us   0   0   300  us   1.5  ms     0   A  CHA  Data  Point  contains  >  150  signals  (staMsMcs  and  events)  from  mul,ple  sources   OS,  ASM  ,  Network   DB  (  ASH,  AWR  session,  system  and  PDB  staIsIcs  )          StaMsMcs  are  collected  at  a  1  second  internal  sampling  rate  ,  synchronized,   smoothed  and  aggregated  to  a  Data  Point  every  5  seconds  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   12   Models  Capture  the  Dynamic  Behavior  of  all  Normal  Opera?on     Models  Capture  all  Normal  OperaMng  Modes   0   5000   10000   15000   20000   25000   30000   35000   40000   10:00   2:00   6:00   5100   9025   4024   2350   4100   22050   10000   21000   4400   2500   4900   800   IOPS   user  commits  (/sec)   log  file  parallel  write  (usec)   log  file  sync  (usec)   •  Release  ships  with  conservaMve  models  to  minimize  false  warnings   •  A  model  captures  the  normal  load  phases  and  their  staMsMcs  over  Mme,  and  thus  the  characterisMcs  for  all  load     intensiMes  and  profiles.  During  monitoring,  any  data  point  similar  to  one  of  the  vectors  is  NORMAL.     •  One  could  say  that  the  model  REMEMBERS  the  normal  opera?onal  dynamics  over  ?me   In-­‐Memory  Reference  Matrix   (Part  of  “Normality”  Model)     IOPS   ####   2500   4900   800   ####   User  Commits   ####   10000   21000   4400   ####   Log  File  Parallel   Write   ####   2350   4100   22050   ####   Log  File  Sync   ####   5100   9025   4024   ####   …   …   …   …   …   …  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   13   CHA  Model:  Find  Similarity  with  Normal  Values     Cluster  Health  Advisor   Observed  values   (Part  of  a  Data  Point)     CHA  esImator/predictor:  “based  on  my  normality  model,  the  value  of  IOPS  should  be  in  the   vicinity  of  ~  4900,  but  it  is  reported  as  10500,  this  is  causing  a  residual  of  ~  5600  in  magnitude”,   CHA  fault  detector:  “such  high  magnitude  of  residuals  should  be  tracked  carefully!  I’ll  keep  an  eye   on  the  incoming  sequence  of  this  signal  IOPS  and  if  it  remains  deviant  I’ll  generate  a  fault  on  it”. In-­‐Memory  Reference  Matrix   (Part  of  “Normality”  Model)     IOPS   ####   2500   4900   800   ####   User  Commits   ####   10000   21000   4400   ####   Log  File  Parallel   Write   ####   2350   4100   22050   ####   Log  File  Sync   ####   5100   9025   4024   ####   …   …   …   …   …   …   10500   20000   4050   10250   …   Residual  Values   (Part  of  a  Data  Point)     5600   -­‐1000   -­‐50   325   …        Observed  -­‐          Predicted  =  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  (CHA)  OperaMon  Overview   14   •  SRVCTL  lifecycle  daemon  management     •  Enabled  by  default  -­‐  AcMvates  when  1st   RAC  instance  starts   •  New  CHACTL  command  line  tool  for  all   local  operaMons     •  Java  GUI  Tool  available  on  OTN  soon   •  Integrated  into  EMCC  Incident  Manager   and  noMficaMons   •  Monitoring  has  no  impact  on     DB  performance  or  availability     CHACTL  Client   CHA  Java  GUI  Client   SRVCTL     OS  Data   GIMR   DB  Data   CHM   Node   Health   PrognosIcs   Engine   Database   Health   PrognosIcs   Engine   OS   Model   DB   Model   Local  to  Cluster   EM   Cloud   Control   CHADDriver  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   15   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Using  CHA  From  the  Command  Line   •  CHA  is  enabled  by  default     •  Autonomously  monitors  nodes  once  a  RAC  DB  starts  in  the  cluster   •  RAC  or  RAC  One  Node  Database  must  be  explicitly  monitored  –  opt  in   •  CHACTL  Command  Line  supports   – Start/Stop  Monitoring   – Diagnosing    Health  Issues  and  CorrecMve  AcMons   – Model  CalibraMon,    AcMvaMon  and  Lifecycle  Management     – CHA  Repository  Data  Lifecycle  Management   16   Overview    
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  –  Command  Line  OperaMons   17   Monitoring  Your  Databases  and  Nodes  with  CHACTL   Enable  CHA  monitoring  on  RAC  database  with  opMonal  model   $ chactl monitor database –db oltpacdb [-model model_name] Enable  CHA  monitoring  on  RAC  database  with  opMonal  verbose   $ chactl status –verbose monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CHA  Command  Line  OperaMons   18   Checking  for  Health  Issues  and  CorrecIve  AcIons  with  CHACTL  QUERY  DIAGNOSIS   $ chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15" 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected] 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected] 2016-10-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected] 2016-10-28 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected] Problem: DB Control File IO Performance Description: CHA has detected that reads or writes to the control files are slower than expected. Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were slow because of an increase in disk IO. The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance. Action: Separate the control files from other database files and move them to faster disks or Solid State Devices. Problem: DB Log File Switch Description: CHA detected that database sessions are waiting longer than expected for log switch completions. Cause: The Cluster Health Advisor (CHA) detected high contention during log switches because the redo log files were small and the redo logs switched frequently. Action: Increase the size of the redo logs.    
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  –  Command  Line  OperaMons   19   HTML  DiagnosIc  Health  Output  Available  (-­‐html  <file_name>)  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   20   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   ConfidenMal  –  Oracle   21   CHA  has  detected  a  service  degradaIon  due  to  higher  than  expected  I/O  latencies.  CHA has detected a service degradation due to higher than expected I/O latencies.! Cluster Health Advisor! CHA/DB Health! CHA detected a for service degradation due to higher than expected I/O latencies.! CHA/DB Health: I/O problem! CHA/DB Health: I/O problem!
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   22   CHA has detected a service degradation due to higher than expected I/O latencies.! Cluster Health Advisor! CHA/DB Health: I/O problem! CHA detected a for service degradation due to higher than expected I/O latencies.! CHA/DB Health! Cluster Health Advisor The degradation is caused by a higher than expected utilization of shared storage devices for this database. No evidence of significant increase in I/O demand on the local node.! Problem Confidence Action 95.17%! Validate whether there is increase in I/O demand on other nodes than the local and find I/O intensive SQL . Add more disks to disk group or move database to faster disks.! proddb_1 proddb_2
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Using  EMCC  for  Alerts  and  CorrecMve  AcMons   23  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line   Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   24   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   25   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   Overview     •  Standalone  Java  GUI  Client   •  Must  be  run  on  local  cluster  node   •  Can  be  run  against  live  GIMR  or  MDB   (dump)  file   chactl export repository -format mdb -start '2017-05-01 00:00:00' -end '2017-05-10 00:00:00' •  Used  internally  for  development   •  Will  be  available  and  maintained  on   Oracle  Technology  Network  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   26   svr01 svr02 svr03 mycluster CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) Cluster ‘mycluster’ Sep-16 21:20:25 Cluster ‘mycluster’ devdb prod testdb webdb Cluster mycluster, Hosts:3, DB’s:4
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   27   svr01 svr02 svr03 mycluster CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) devdb prod testdb webdb Cluster mycluster, Hosts:3, DB’s:4 Host ‘svr01’ Host ‘svr02’ Host ‘svr03’
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   28   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 svr02 Host svr02 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   29   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 DB prod in mycluster Sep-16 22:16:35 DB prod in mycluster mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   30   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   31   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   32  ConfidenMal  –  Oracle  Restricted   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   33   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   34   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   35   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   36   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster DB prod in mycluster Sep-16 22:16:35 DB prod in mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   37   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   38   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   “We  also  deployed  the  CHA  GUI  which  has  been  designed  specifically  to  visualize   abnormaliMes  in  OS  and  RDBMS  staMsMcs  data  streams  which  are  detected  and   diagnosed  by  CHA  in  real  Mme.  It  stands  out  as  a  successful  visualizaMon  of  the  real  Mme   and  historical  monitoring  data  and  diagnosMcs  in  our  Oracle  RAC  database  environment.”   –  Jens-­‐ChrisMan  Pokolm,     Head  of  Databases  Postbank  Systems  AG   39  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons     Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  Deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   40   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CalibraMng  CHA  to  your  RAC  Deployment   •  CalibraMon  Goal:  Increase  sensiMvity  and  accuracy  with  sufficient  warning   •  Release  ships  with  conservaMve  models  to  minimize  false  warnings   – DEFAULT_CLUSTER  for  each  cluster  node   – DEFAULT_DB  for  each  database  instance   •  Use  your  own  data  for  periods  of  “normal  operaMons”  to  increase   sensiMvity   – Recommended  minimum  6  hour  period     – Should  include  all  normal  workload  phases  for  that  model   •  Models  may  be  changed  dynamically  online  using  CHACTL     41   Overview    
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CalibraMng  CHA  to  your  RAC  deployment   42   Choosing  a  Data  Set  for  CalibraIon  –  Defining  “normal”   $ chactl query calibration –cluster –timeranges ‘start=2016-10-28 07:00:00,end=2016-10-28 13:00:00’ Cluster name : mycluster Start time : 2016-10-28 07:00:00 End time : 2016-10-28 13:00:00 Total Samples : 11524 Percentage of filtered data : 100% 1) Disk read (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.11 0.00 2.62 0.00 114.66 <25 <50 <75 <100 >=100 99.87% 0.08% 0.00% 0.02% 0.03% 2) Disk write (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.01 0.00 0.15 0.00 6.77 <50 <100 <150 <200 >=200 100.00% 0.00% 0.00% 0.00% 0.00% 3) Disk throughput (ASM) (IO/sec) MEAN MEDIAN STDDEV MIN MAX 2.20 0.00 31.17 0.00 1100.00 <5000 <10000 <15000 <20000 >=20000 100.00% 0.00% 0.00% 0.00% 0.00% 4) CPU utilization (total) (%) MEAN MEDIAN STDDEV MIN MAX 9.62 9.30 7.95 1.80 77.90 <20 <40 <60 <80 >=80 92.67% 6.17% 1.11% 0.05% 0.00%
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CalibraMng  CHA  to  your  RAC  deployment   •  Create  and  store  the  new  model   $ chactl query calibrate cluster –model daytime –timeranges ‘start=2016-10-28 07:00:00, end=2016-10-28 13:00:00’ •  Begin  using  the  new  model   $ chactl monitor cluster –model daytime •  Confirm  the  new  model  is  being  used   $ chactl status –verbose monitoring nodes svr01, svr02 using model daytime monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB 43   CreaIng  a  new  CHA  Model  with  CHACTL  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   44   5   6   7  
Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   For  Further  InformaMon   45   •  Oracle  12c  Autonomous  Health  Framework  User’s  Guide     •  Oracle  12c  Clusterware  AdministraMon  and  Deployment  Guide   •  Oracle  Autonomous  Health  Framework  on  OTN  
Cluster Health Advisor (CHA) Deep Dive by Mark Scardina

More Related Content

PDF
Tanel Poder - Scripts and Tools short
Tanel Poder
 
PDF
Oracle Performance Tuning Fundamentals
Enkitec
 
PPTX
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
 
PPTX
Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder
 
PDF
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
 
PPT
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
John Kanagaraj
 
PDF
Oracle RAC Internals - The Cache Fusion Edition
Markus Michalewicz
 
PDF
Oracle Enterprise Manager Cloud Control 13c for DBAs
Gokhan Atil
 
Tanel Poder - Scripts and Tools short
Tanel Poder
 
Oracle Performance Tuning Fundamentals
Enkitec
 
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
 
Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder
 
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
John Kanagaraj
 
Oracle RAC Internals - The Cache Fusion Edition
Markus Michalewicz
 
Oracle Enterprise Manager Cloud Control 13c for DBAs
Gokhan Atil
 

What's hot (20)

PPT
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
PPTX
SQL Plan Directives explained
Mauro Pagano
 
PDF
SQL Monitoring in Oracle Database 12c
Tanel Poder
 
PDF
Oracle statistics by example
Mauro Pagano
 
PDF
Oracle RAC 19c: Best Practices and Secret Internals
Anil Nair
 
PPTX
Oracle GoldenGate Performance Tuning
Bobby Curtis
 
PDF
Understanding oracle rac internals part 1 - slides
Mohamed Farouk
 
PDF
Analyzing and Interpreting AWR
pasalapudi
 
PDF
Chasing the optimizer
Mauro Pagano
 
PDF
Performance Tuning Using oratop
Sandesh Rao
 
PDF
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
 
PDF
Oracle db performance tuning
Simon Huang
 
PDF
Exadata master series_asm_2020
Anil Nair
 
PDF
Oracle Latch and Mutex Contention Troubleshooting
Tanel Poder
 
PPT
Wait events
anujkumarsingh08
 
PDF
Average Active Sessions RMOUG2007
John Beresniewicz
 
PDF
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
John Beresniewicz
 
PDF
What's new in Oracle Trace File Analyzer version 12.2.1.1.0
Sandesh Rao
 
PDF
Oracle data guard for beginners
Pini Dibask
 
PDF
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
 
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
SQL Plan Directives explained
Mauro Pagano
 
SQL Monitoring in Oracle Database 12c
Tanel Poder
 
Oracle statistics by example
Mauro Pagano
 
Oracle RAC 19c: Best Practices and Secret Internals
Anil Nair
 
Oracle GoldenGate Performance Tuning
Bobby Curtis
 
Understanding oracle rac internals part 1 - slides
Mohamed Farouk
 
Analyzing and Interpreting AWR
pasalapudi
 
Chasing the optimizer
Mauro Pagano
 
Performance Tuning Using oratop
Sandesh Rao
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
 
Oracle db performance tuning
Simon Huang
 
Exadata master series_asm_2020
Anil Nair
 
Oracle Latch and Mutex Contention Troubleshooting
Tanel Poder
 
Wait events
anujkumarsingh08
 
Average Active Sessions RMOUG2007
John Beresniewicz
 
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
John Beresniewicz
 
What's new in Oracle Trace File Analyzer version 12.2.1.1.0
Sandesh Rao
 
Oracle data guard for beginners
Pini Dibask
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
 
Ad

Similar to Cluster Health Advisor (CHA) Deep Dive by Mark Scardina (20)

PDF
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
Markus Michalewicz
 
PDF
AWR, ASH with EM13 at HotSos 2016
Kellyn Pot'Vin-Gorman
 
PDF
Analysis of Database Issues using AHF and Machine Learning v2 - SOUG
Sandesh Rao
 
PDF
Oracle RAC 12c Rel. 2 for Continuous Availability
Markus Michalewicz
 
PDF
Oracle Autonomous Health Service- For Protecting Your On-Premise Databases- F...
Sandesh Rao
 
PDF
Streamline it management
DLT Solutions
 
PDF
Magic With Oracle - Presentation
Francisco Alvarez
 
PDF
Best practices for_large_oracle_apps_r12_implementations
Ajith Narayanan
 
PDF
Make Your Application “Oracle RAC Ready” & Test For It
Markus Michalewicz
 
PDF
MAKING MAGIC WITH ORACLE (Francisco Slide)
TheGameZ
 
PDF
Getting optimal performance from oracle e business suite(aioug aug2015)
pasalapudi123
 
PDF
Getting optimal performance from oracle e business suite
aioughydchapter
 
PPTX
A-Team Mobile Persistence Accelerator Overview
Steven Davelaar
 
PDF
How to Use Oracle RAC in a Cloud? - A Support Question
Markus Michalewicz
 
PDF
VMworld 2014: Virtualizing Databases
VMworld
 
PDF
Analysis of Database Issues using AHF and Machine Learning v2 - AOUG2022
Sandesh Rao
 
PDF
Oracle Autonomous Health Framework (AHF) White Paper
Ankita Khandelwal
 
PDF
New Features in Oracle ORAchk & EXAchk 12.2.0.1.1
Gareth Chapman
 
PDF
OOW16 - Getting Optimal Performance from Oracle E-Business Suite [CON6711]
vasuballa
 
PPTX
Stream Analytics
Franco Ucci
 
AskTom: How to Make and Test Your Application "Oracle RAC Ready"?
Markus Michalewicz
 
AWR, ASH with EM13 at HotSos 2016
Kellyn Pot'Vin-Gorman
 
Analysis of Database Issues using AHF and Machine Learning v2 - SOUG
Sandesh Rao
 
Oracle RAC 12c Rel. 2 for Continuous Availability
Markus Michalewicz
 
Oracle Autonomous Health Service- For Protecting Your On-Premise Databases- F...
Sandesh Rao
 
Streamline it management
DLT Solutions
 
Magic With Oracle - Presentation
Francisco Alvarez
 
Best practices for_large_oracle_apps_r12_implementations
Ajith Narayanan
 
Make Your Application “Oracle RAC Ready” & Test For It
Markus Michalewicz
 
MAKING MAGIC WITH ORACLE (Francisco Slide)
TheGameZ
 
Getting optimal performance from oracle e business suite(aioug aug2015)
pasalapudi123
 
Getting optimal performance from oracle e business suite
aioughydchapter
 
A-Team Mobile Persistence Accelerator Overview
Steven Davelaar
 
How to Use Oracle RAC in a Cloud? - A Support Question
Markus Michalewicz
 
VMworld 2014: Virtualizing Databases
VMworld
 
Analysis of Database Issues using AHF and Machine Learning v2 - AOUG2022
Sandesh Rao
 
Oracle Autonomous Health Framework (AHF) White Paper
Ankita Khandelwal
 
New Features in Oracle ORAchk & EXAchk 12.2.0.1.1
Gareth Chapman
 
OOW16 - Getting Optimal Performance from Oracle E-Business Suite [CON6711]
vasuballa
 
Stream Analytics
Franco Ucci
 
Ad

More from Markus Michalewicz (20)

PDF
Achieving Continuous Availability for Your Applications with Oracle MAA
Markus Michalewicz
 
PDF
"It can always get worse!" – Lessons Learned in over 20 years working with Or...
Markus Michalewicz
 
PDF
Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud
Markus Michalewicz
 
PDF
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
Markus Michalewicz
 
PDF
HA, Scalability, DR & MAA in Oracle Database 21c - Overview
Markus Michalewicz
 
PDF
Oracle Cloud is Best for Oracle Database - High Availability
Markus Michalewicz
 
PDF
Oracle Database – Mission Critical
Markus Michalewicz
 
PDF
2020 – A Decade of Change
Markus Michalewicz
 
PDF
Standard Edition High Availability (SEHA) - The Why, What & How
Markus Michalewicz
 
PDF
Why Use an Oracle Database?
Markus Michalewicz
 
PDF
"Changing Role of the DBA" Skills to Have, to Obtain & to Nurture - Updated 2...
Markus Michalewicz
 
PDF
Oracle RAC 19c and Later - Best Practices #OOWLON
Markus Michalewicz
 
PDF
MAA for Oracle Database, Exadata and the Cloud
Markus Michalewicz
 
PDF
(Oracle) DBA and Other Skills Needed in 2020
Markus Michalewicz
 
PDF
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
 
PDF
Best Practices for the Most Impactful Oracle Database 18c and 19c Features
Markus Michalewicz
 
PDF
Oracle Database Availability & Scalability Across Versions & Editions
Markus Michalewicz
 
PDF
Oracle RAC 19c - the Basis for the Autonomous Database
Markus Michalewicz
 
PDF
From HA to Maximum Availability - A Holistic Historical Discussion
Markus Michalewicz
 
PDF
Why to Use an Oracle Database?
Markus Michalewicz
 
Achieving Continuous Availability for Your Applications with Oracle MAA
Markus Michalewicz
 
"It can always get worse!" – Lessons Learned in over 20 years working with Or...
Markus Michalewicz
 
Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud
Markus Michalewicz
 
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
Markus Michalewicz
 
HA, Scalability, DR & MAA in Oracle Database 21c - Overview
Markus Michalewicz
 
Oracle Cloud is Best for Oracle Database - High Availability
Markus Michalewicz
 
Oracle Database – Mission Critical
Markus Michalewicz
 
2020 – A Decade of Change
Markus Michalewicz
 
Standard Edition High Availability (SEHA) - The Why, What & How
Markus Michalewicz
 
Why Use an Oracle Database?
Markus Michalewicz
 
"Changing Role of the DBA" Skills to Have, to Obtain & to Nurture - Updated 2...
Markus Michalewicz
 
Oracle RAC 19c and Later - Best Practices #OOWLON
Markus Michalewicz
 
MAA for Oracle Database, Exadata and the Cloud
Markus Michalewicz
 
(Oracle) DBA and Other Skills Needed in 2020
Markus Michalewicz
 
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
 
Best Practices for the Most Impactful Oracle Database 18c and 19c Features
Markus Michalewicz
 
Oracle Database Availability & Scalability Across Versions & Editions
Markus Michalewicz
 
Oracle RAC 19c - the Basis for the Autonomous Database
Markus Michalewicz
 
From HA to Maximum Availability - A Holistic Historical Discussion
Markus Michalewicz
 
Why to Use an Oracle Database?
Markus Michalewicz
 

Recently uploaded (20)

PDF
49784907924775488180_LRN2959_Data_Pump_23ai.pdf
Abilash868456
 
PDF
49785682629390197565_LRN3014_Migrating_the_Beast.pdf
Abilash868456
 
PDF
Applitools Platform Pulse: What's New and What's Coming - July 2025
Applitools
 
DOCX
Can You Build Dashboards Using Open Source Visualization Tool.docx
Varsha Nayak
 
PDF
An Experience-Based Look at AI Lead Generation Pricing, Features & B2B Results
Thomas albart
 
PDF
WatchTraderHub - Watch Dealer software with inventory management and multi-ch...
WatchDealer Pavel
 
PPTX
Contractor Management Platform and Software Solution for Compliance
SHEQ Network Limited
 
PDF
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
PPTX
Can You Build Dashboards Using Open Source Visualization Tool.pptx
Varsha Nayak
 
PPTX
classification of computer and basic part of digital computer
ravisinghrajpurohit3
 
PPTX
Explanation about Structures in C language.pptx
Veeral Rathod
 
PDF
New Download MiniTool Partition Wizard Crack Latest Version 2025
imang66g
 
PDF
What to consider before purchasing Microsoft 365 Business Premium_PDF.pdf
Q-Advise
 
PDF
Generating Union types w/ Static Analysis
K. Matthew Dupree
 
PDF
ChatPharo: an Open Architecture for Understanding How to Talk Live to LLMs
ESUG
 
PDF
MiniTool Power Data Recovery Crack New Pre Activated Version Latest 2025
imang66g
 
PDF
Exploring AI Agents in Process Industries
amoreira6
 
PDF
Immersive experiences: what Pharo users do!
ESUG
 
PPTX
Visualising Data with Scatterplots in IBM SPSS Statistics.pptx
Version 1 Analytics
 
PPT
Why Reliable Server Maintenance Service in New York is Crucial for Your Business
Sam Vohra
 
49784907924775488180_LRN2959_Data_Pump_23ai.pdf
Abilash868456
 
49785682629390197565_LRN3014_Migrating_the_Beast.pdf
Abilash868456
 
Applitools Platform Pulse: What's New and What's Coming - July 2025
Applitools
 
Can You Build Dashboards Using Open Source Visualization Tool.docx
Varsha Nayak
 
An Experience-Based Look at AI Lead Generation Pricing, Features & B2B Results
Thomas albart
 
WatchTraderHub - Watch Dealer software with inventory management and multi-ch...
WatchDealer Pavel
 
Contractor Management Platform and Software Solution for Compliance
SHEQ Network Limited
 
Download iTop VPN Free 6.1.0.5882 Crack Full Activated Pre Latest 2025
imang66g
 
Can You Build Dashboards Using Open Source Visualization Tool.pptx
Varsha Nayak
 
classification of computer and basic part of digital computer
ravisinghrajpurohit3
 
Explanation about Structures in C language.pptx
Veeral Rathod
 
New Download MiniTool Partition Wizard Crack Latest Version 2025
imang66g
 
What to consider before purchasing Microsoft 365 Business Premium_PDF.pdf
Q-Advise
 
Generating Union types w/ Static Analysis
K. Matthew Dupree
 
ChatPharo: an Open Architecture for Understanding How to Talk Live to LLMs
ESUG
 
MiniTool Power Data Recovery Crack New Pre Activated Version Latest 2025
imang66g
 
Exploring AI Agents in Process Industries
amoreira6
 
Immersive experiences: what Pharo users do!
ESUG
 
Visualising Data with Scatterplots in IBM SPSS Statistics.pptx
Version 1 Analytics
 
Why Reliable Server Maintenance Service in New York is Crucial for Your Business
Sam Vohra
 

Cluster Health Advisor (CHA) Deep Dive by Mark Scardina

  • 2. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Mark  V.  Scardina  –  Director  of  Product  Management   Oracle  Autonomous  Health  Framework  and     Oracle  Quality  of  Service  Management   Oracle  Database  12c  Rel.  2   Cluster  Health  Advisor  -­‐  Deep  Dive   How  it  Works  and  How  to  Use  It  
  • 3. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line   Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons     Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   3   5   6   7  
  • 4. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line   Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons     Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   4   5   6   7  
  • 5. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Oracle  Autonomous  Health  Framework   •  Integrates  next  generaMon  tools  running   as  components  -­‐  24/7     •  Discovers  PotenMal  Issues  and  NoMfies   or  takes  CorrecMve  AcMons   •  Speeds  up  Issue  Diagnosis  and  Recovery   •  Preserves  Database  and  Server   Availability  and  Performance   •  Autonomously  Monitors  and  Manages   resources  to  maintain  SLAs   5   Working  for  You  ConInuously  
  • 6. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Discovers  PotenMal  Cluster   &  DB  Problems  -­‐  NoMfies   with  CorrecMve  AcMons   6   Oracle  12c  Cluster  Health  Advisor  
  • 7. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Introducing  Oracle  12c  Cluster  Health  Advisor   •  Real  Mme  monitoring  of  Oracle  RAC  database  systems  and  their  hosts   •  Early  detecMon  of  impending  as  well  as  ongoing  system  faults   •  Diagnoses  and  idenMfies  the  most  likely  root  causes   •  Provides  correcMve  acMons  for  targeted  triage.   •  Generates  alerts  and  noMficaMons  for  rapid  recovery   7   ProacIve  Health  PrognosIcs  System    
  • 8. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   8   5   6   7  
  • 9. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  (CHA)  Architecture  Overview   9   OS  Data   GIMR   ochad   DB  Data   CHM   Node   Health   PrognosIcs   Engine   Database   Health   PrognosIcs   Engine   OS   Model   DB   Model   •  cha  –  Cluster  node  resource     •  Single  Java  ochad  daemon  per  node   •  Reads  Cluster  Health  Monitor  data     directly  from  memory   •  Reads  DB  ASH  data  from  SMR  w/o  DB  connecMon   •  Uses  OS  and  DB  models  and  data  to  perform   prognosMcs   •  Stores  analysis  and  evidence  in  the  GI   Management  Repository   •  Sends  alerts  to  EMCC  Incident  Manager  per   target   EMCC   Alert  
  • 10. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  -­‐  Scope  of  Problem  DetecMon   •  Over  30  node  and  database  problems  have  been  modeled   •  Over  150  OS  and  DB  metric  predictors  idenMfied   •  Problem  DetecMon  in  12.2.0.1  includes   – Interconnect  ,  Global  Cache  and  Cluster  Problems   – Host  CPU  and  Memory  ,  PGA  Memory  stress     – IO  and  Storage  Performance  issues   – ReconfiguraMon  and  Recovery  issues   – Workload  and  Session  abnormal  variaMons     10   Best  Effort  Immediate  Guided  Diagnosis    
  • 11. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   11   Data  Sources  and  Data  Points     Cluster  Health  Advisor   Time   CPU   ASM     IOPS     Network   %  uIl   Network_ Packets   Dropped   Log   file   sync   Log  file   parallel   write   GC    CR   request   GC  current     request   GC  current   block  2-­‐way   GC  current   block  busy   Enq:  CF     -­‐ conten Ion   … 15:16:00   0.90   4100   13%   0   2  ms   600  us   0   0   300  us   1.5  ms     0   A  CHA  Data  Point  contains  >  150  signals  (staMsMcs  and  events)  from  mul,ple  sources   OS,  ASM  ,  Network   DB  (  ASH,  AWR  session,  system  and  PDB  staIsIcs  )          StaMsMcs  are  collected  at  a  1  second  internal  sampling  rate  ,  synchronized,   smoothed  and  aggregated  to  a  Data  Point  every  5  seconds  
  • 12. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   12   Models  Capture  the  Dynamic  Behavior  of  all  Normal  Opera?on     Models  Capture  all  Normal  OperaMng  Modes   0   5000   10000   15000   20000   25000   30000   35000   40000   10:00   2:00   6:00   5100   9025   4024   2350   4100   22050   10000   21000   4400   2500   4900   800   IOPS   user  commits  (/sec)   log  file  parallel  write  (usec)   log  file  sync  (usec)   •  Release  ships  with  conservaMve  models  to  minimize  false  warnings   •  A  model  captures  the  normal  load  phases  and  their  staMsMcs  over  Mme,  and  thus  the  characterisMcs  for  all  load     intensiMes  and  profiles.  During  monitoring,  any  data  point  similar  to  one  of  the  vectors  is  NORMAL.     •  One  could  say  that  the  model  REMEMBERS  the  normal  opera?onal  dynamics  over  ?me   In-­‐Memory  Reference  Matrix   (Part  of  “Normality”  Model)     IOPS   ####   2500   4900   800   ####   User  Commits   ####   10000   21000   4400   ####   Log  File  Parallel   Write   ####   2350   4100   22050   ####   Log  File  Sync   ####   5100   9025   4024   ####   …   …   …   …   …   …  
  • 13. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   13   CHA  Model:  Find  Similarity  with  Normal  Values     Cluster  Health  Advisor   Observed  values   (Part  of  a  Data  Point)     CHA  esImator/predictor:  “based  on  my  normality  model,  the  value  of  IOPS  should  be  in  the   vicinity  of  ~  4900,  but  it  is  reported  as  10500,  this  is  causing  a  residual  of  ~  5600  in  magnitude”,   CHA  fault  detector:  “such  high  magnitude  of  residuals  should  be  tracked  carefully!  I’ll  keep  an  eye   on  the  incoming  sequence  of  this  signal  IOPS  and  if  it  remains  deviant  I’ll  generate  a  fault  on  it”. In-­‐Memory  Reference  Matrix   (Part  of  “Normality”  Model)     IOPS   ####   2500   4900   800   ####   User  Commits   ####   10000   21000   4400   ####   Log  File  Parallel   Write   ####   2350   4100   22050   ####   Log  File  Sync   ####   5100   9025   4024   ####   …   …   …   …   …   …   10500   20000   4050   10250   …   Residual  Values   (Part  of  a  Data  Point)     5600   -­‐1000   -­‐50   325   …        Observed  -­‐          Predicted  =  
  • 14. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  (CHA)  OperaMon  Overview   14   •  SRVCTL  lifecycle  daemon  management     •  Enabled  by  default  -­‐  AcMvates  when  1st   RAC  instance  starts   •  New  CHACTL  command  line  tool  for  all   local  operaMons     •  Java  GUI  Tool  available  on  OTN  soon   •  Integrated  into  EMCC  Incident  Manager   and  noMficaMons   •  Monitoring  has  no  impact  on     DB  performance  or  availability     CHACTL  Client   CHA  Java  GUI  Client   SRVCTL     OS  Data   GIMR   DB  Data   CHM   Node   Health   PrognosIcs   Engine   Database   Health   PrognosIcs   Engine   OS   Model   DB   Model   Local  to  Cluster   EM   Cloud   Control   CHADDriver  
  • 15. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   15   5   6   7  
  • 16. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Using  CHA  From  the  Command  Line   •  CHA  is  enabled  by  default     •  Autonomously  monitors  nodes  once  a  RAC  DB  starts  in  the  cluster   •  RAC  or  RAC  One  Node  Database  must  be  explicitly  monitored  –  opt  in   •  CHACTL  Command  Line  supports   – Start/Stop  Monitoring   – Diagnosing    Health  Issues  and  CorrecMve  AcMons   – Model  CalibraMon,    AcMvaMon  and  Lifecycle  Management     – CHA  Repository  Data  Lifecycle  Management   16   Overview    
  • 17. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  –  Command  Line  OperaMons   17   Monitoring  Your  Databases  and  Nodes  with  CHACTL   Enable  CHA  monitoring  on  RAC  database  with  opMonal  model   $ chactl monitor database –db oltpacdb [-model model_name] Enable  CHA  monitoring  on  RAC  database  with  opMonal  verbose   $ chactl status –verbose monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
  • 18. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CHA  Command  Line  OperaMons   18   Checking  for  Health  Issues  and  CorrecIve  AcIons  with  CHACTL  QUERY  DIAGNOSIS   $ chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15" 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected] 2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected] 2016-10-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected] 2016-10-28 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected] Problem: DB Control File IO Performance Description: CHA has detected that reads or writes to the control files are slower than expected. Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were slow because of an increase in disk IO. The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance. Action: Separate the control files from other database files and move them to faster disks or Solid State Devices. Problem: DB Log File Switch Description: CHA detected that database sessions are waiting longer than expected for log switch completions. Cause: The Cluster Health Advisor (CHA) detected high contention during log switches because the redo log files were small and the redo logs switched frequently. Action: Increase the size of the redo logs.    
  • 19. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  Health  Advisor  –  Command  Line  OperaMons   19   HTML  DiagnosIc  Health  Output  Available  (-­‐html  <file_name>)  
  • 20. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   20   5   6   7  
  • 21. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   ConfidenMal  –  Oracle   21   CHA  has  detected  a  service  degradaIon  due  to  higher  than  expected  I/O  latencies.  CHA has detected a service degradation due to higher than expected I/O latencies.! Cluster Health Advisor! CHA/DB Health! CHA detected a for service degradation due to higher than expected I/O latencies.! CHA/DB Health: I/O problem! CHA/DB Health: I/O problem!
  • 22. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   22   CHA has detected a service degradation due to higher than expected I/O latencies.! Cluster Health Advisor! CHA/DB Health: I/O problem! CHA detected a for service degradation due to higher than expected I/O latencies.! CHA/DB Health! Cluster Health Advisor The degradation is caused by a higher than expected utilization of shared storage devices for this database. No evidence of significant increase in I/O demand on the local node.! Problem Confidence Action 95.17%! Validate whether there is increase in I/O demand on other nodes than the local and find I/O intensive SQL . Add more disks to disk group or move database to faster disks.! proddb_1 proddb_2
  • 23. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Using  EMCC  for  Alerts  and  CorrecMve  AcMons   23  
  • 24. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line   Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   24   5   6   7  
  • 25. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   25   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   Overview     •  Standalone  Java  GUI  Client   •  Must  be  run  on  local  cluster  node   •  Can  be  run  against  live  GIMR  or  MDB   (dump)  file   chactl export repository -format mdb -start '2017-05-01 00:00:00' -end '2017-05-10 00:00:00' •  Used  internally  for  development   •  Will  be  available  and  maintained  on   Oracle  Technology  Network  
  • 26. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   26   svr01 svr02 svr03 mycluster CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) Cluster ‘mycluster’ Sep-16 21:20:25 Cluster ‘mycluster’ devdb prod testdb webdb Cluster mycluster, Hosts:3, DB’s:4
  • 27. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   27   svr01 svr02 svr03 mycluster CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) devdb prod testdb webdb Cluster mycluster, Hosts:3, DB’s:4 Host ‘svr01’ Host ‘svr02’ Host ‘svr03’
  • 28. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   28   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 svr02 Host svr02 mycluster
  • 29. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   29   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 DB prod in mycluster Sep-16 22:16:35 DB prod in mycluster mycluster
  • 30. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   30   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 31. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   31   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 32. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   32  ConfidenMal  –  Oracle  Restricted   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 33. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   33   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 34. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   34   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 35. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   35   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 36. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   36   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster DB prod in mycluster Sep-16 22:16:35 DB prod in mycluster
  • 37. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   37   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 38. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   38   CHA (on SVR01) V0.57.6, Data V0.17 – OTN Version (on SVR01) svr01 svr02 svr03 mycluster
  • 39. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   “We  also  deployed  the  CHA  GUI  which  has  been  designed  specifically  to  visualize   abnormaliMes  in  OS  and  RDBMS  staMsMcs  data  streams  which  are  detected  and   diagnosed  by  CHA  in  real  Mme.  It  stands  out  as  a  successful  visualizaMon  of  the  real  Mme   and  historical  monitoring  data  and  diagnosMcs  in  our  Oracle  RAC  database  environment.”   –  Jens-­‐ChrisMan  Pokolm,     Head  of  Databases  Postbank  Systems  AG   39  
  • 40. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons     Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  Deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   40   5   6   7  
  • 41. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CalibraMng  CHA  to  your  RAC  Deployment   •  CalibraMon  Goal:  Increase  sensiMvity  and  accuracy  with  sufficient  warning   •  Release  ships  with  conservaMve  models  to  minimize  false  warnings   – DEFAULT_CLUSTER  for  each  cluster  node   – DEFAULT_DB  for  each  database  instance   •  Use  your  own  data  for  periods  of  “normal  operaMons”  to  increase   sensiMvity   – Recommended  minimum  6  hour  period     – Should  include  all  normal  workload  phases  for  that  model   •  Models  may  be  changed  dynamically  online  using  CHACTL     41   Overview    
  • 42. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CalibraMng  CHA  to  your  RAC  deployment   42   Choosing  a  Data  Set  for  CalibraIon  –  Defining  “normal”   $ chactl query calibration –cluster –timeranges ‘start=2016-10-28 07:00:00,end=2016-10-28 13:00:00’ Cluster name : mycluster Start time : 2016-10-28 07:00:00 End time : 2016-10-28 13:00:00 Total Samples : 11524 Percentage of filtered data : 100% 1) Disk read (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.11 0.00 2.62 0.00 114.66 <25 <50 <75 <100 >=100 99.87% 0.08% 0.00% 0.02% 0.03% 2) Disk write (ASM) (Mbyte/sec) MEAN MEDIAN STDDEV MIN MAX 0.01 0.00 0.15 0.00 6.77 <50 <100 <150 <200 >=200 100.00% 0.00% 0.00% 0.00% 0.00% 3) Disk throughput (ASM) (IO/sec) MEAN MEDIAN STDDEV MIN MAX 2.20 0.00 31.17 0.00 1100.00 <5000 <10000 <15000 <20000 >=20000 100.00% 0.00% 0.00% 0.00% 0.00% 4) CPU utilization (total) (%) MEAN MEDIAN STDDEV MIN MAX 9.62 9.30 7.95 1.80 77.90 <20 <40 <60 <80 >=80 92.67% 6.17% 1.11% 0.05% 0.00%
  • 43. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   CalibraMng  CHA  to  your  RAC  deployment   •  Create  and  store  the  new  model   $ chactl query calibrate cluster –model daytime –timeranges ‘start=2016-10-28 07:00:00, end=2016-10-28 13:00:00’ •  Begin  using  the  new  model   $ chactl monitor cluster –model daytime •  Confirm  the  new  model  is  being  used   $ chactl status –verbose monitoring nodes svr01, svr02 using model daytime monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB 43   CreaIng  a  new  CHA  Model  with  CHACTL  
  • 44. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Agenda   IntroducMon     CHA  Architecture  and  OperaMon  Details   Using  CHA  from  the  Command  Line     Using  CHA  from  EMCC  for  Alerts  and  CorrecMve  AcMons   Using  the  CHA  GUI  to  Perform  Root-­‐Cause  Analysis   CalibraMng  CHA  to  your  RAC  deployment   Q  &  A  –  Further  InformaMon     1   2   3   4   44   5   6   7  
  • 45. Copyright  ©  2016,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   For  Further  InformaMon   45   •  Oracle  12c  Autonomous  Health  Framework  User’s  Guide     •  Oracle  12c  Clusterware  AdministraMon  and  Deployment  Guide   •  Oracle  Autonomous  Health  Framework  on  OTN