AIOUG : ODEVCYathra 2018 - Oracle Autonomous Database What Every DBA should know

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Autonomous Database
Sandesh Rao
VP - Autonomous Health & Machine Learning

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Confidential – Oracle Restricted
2

Theme
1. Tools or features which provide some function
2. Automation around some of these tools or features
3. Components or products which use machine learning to solve some use-cases
4. Additional ML tools which can be used on 1,2 or the results of 3 to develop different
outcomes
1. People who know DataScience
2. People who want to use it – prebuilt models
C
o
n
f
i
d
e
n
t
i
a
l
–
O
r
a
c
l
e
R
e
s
t
r
i 3

Agenda
Journey to Autonomous Database
Machine learning basics & use cases
1
2
4

Oracle’s Vision for Autonomous Database
• Self-Driving
–User defines service levels, database makes them happen
• Self-Securing
–Protection from both external attacks and malicious internal users
• Self-Repairing
–Automated protection from all downtime
5
Autonomous
Database

Oracle Database 9i, 10g
• Automatic Storage Management (ASM)
• Automatic Memory Management
• Automatic DB Diagnostic Monitor (ADDM)
• Automatic Workload Repository (AWR)
• Automatic Undo tablespaces
• Automatic Segment Space Management
• Automatic Statistics Gathering
• Automatic Standby Management (Broker)
• Automatic Query Rewrite
Oracle Database 11g, 12c
• Automatic SQL Tuning
• Automatic Workload Replay
• Automatic Capture of SQL Monitor
• Automatic Data Optimization
• Automatic Storage Indexes
• Automatic Columnar Cache
• Automatic Diagnostic Framework
• Automatic Refresh of Database Cloning
• Autonomous Health Framework
6
• Oracle has been developing sophisticated database automation for decades

Database Operations Runtime Management
• Solving these challenges requires a holistic approach
– Prevent problems and optimize solutions in real-time
– Recover from failures and identify root cause quickly with minimal intervention
• Human reactions too late and do not scale
• Manual triage and floods of notifications do not scale
• Applied Machine learning techniques effectively respond in real-time and
without huge impact to operations
Confidential – Oracle Restricted 7
Prevention and Recovery Pillars

• Cloud enables Oracle to deliver a Fully Autonomous Database
– Expanded Database Automation
– Integrated with complete infrastructure automation
– With additional automation for operations, HA, security, etc.
8
Autonomous
Database

One Autonomous Database – Optimized by Use Case
9
Oracle Autonomous Database
Enterprise
OLTP,
Mixed
Workloads
Data
Warehousing
Departments,
Developers
2017 2018 Now

Autonomous Database Cloud For Data Warehouse
• Easy
– Automatically optimizes Analytic workloads
– Simply “load and go”
– Database tunes itself - No need to define indexes, partitions, materialized views, etc.
– Works with any BI analytics tool
• Fast
– Based on Exadata technology
– Performance matches or exceeds most hand-tuned Data Warehouses
• Elastic
– Instant scaling of compute or storage with no downtime
– Pay for compute when in use only
10
Expected CY 2017

Autonomous Database Cloud For OLTP or Mixed Workloads
11
Expected CY 2018
• Easy
– Configured for Mission Critical workloads
• Full Maximum Availability Architecture with scale-out clustering and disaster recovery
– Or Configured for Low Cost
• Single server for non-critical workloads or test/dev
• Fast
– Based on Exadata technology
• Elastic
– Instant scaling of compute or storage with no downtime

Full End-to-End Automation
• Must handle a large number of tasks
– Provisioning complex scale-out clusters with disaster recovery
– Patching, upgrading, and backing up online
– Monitoring, scaling, diagnosing performance, tuning, optimizing
– Testing and change management of complex applications and workloads
– Automatically handling failures and errors
• Autonomous Database brings difficult trade-offs:
– Best Performance vs. Consistent Performance
– Simplicity vs. Completeness
12
Autonomous
Database

Autonomous Health via Machine Learning
• Real-time Health Monitoring
of compliance, performance,
availability & capacity
2014
2016
2018+
Journey to Autonomous Database Cloud
Confidential – Gartner OPDBMS Vendor Briefing
• Automated analysis &
Anomaly detection
• Automated & targeted diagnostic
collections (50+ top areas & growing)
• Automated Health Checks
• Log masking, reduction &
diagnostic collections
• Automated
repair
2017
• Automated log
lifecycle management
• Preemptive fault
prediction &
correction
• Automated
environment
correlation for
fault prioritization
& flood control• Automated
workload
forecasting
2015
• Integration of database
support tools

Machine Learning Use Cases
Machine learning basics
Log reduction & Anomaly timeline
Maintenance slot identification
Detect Performance Problems
Problem Signatures from Event Paths
Discover Duplicate bugs correlated issues
1
2
3
14
4
5
6

3 Key Areas of Machine Learning
Analytics
Knowledge discovery
Machine Learning
Learn & get better from
experience
Artificial Intelligence
Simulate human
intelligence
15

Examples of Machine Learning Problem Types
Example: Classify if a particular log entry is
normal or not
Classifiers
Predict a label classification
Example: Predict when a system will run out
of memory
Regression
Predict a value
Example: Group incidents into collections of
similar ones, that share some common
attributes
Clustering
Form groups by discovering
reoccurring patterns
16

Machine Learning Categories
Supervised Learning
Predict future outcomes with the help of
training data provided by human experts
Semi-Supervised Learning
Discover patterns within raw data and make
predictions, which are then reviewed by human
experts, who provide feedback which is used to
improve the model accuracy
Unsupervised Learning
Find patterns without any external input other
than the raw data
Reinforcement Learning
Take decisions based on past rewards for this
type of action
17

Real-time Prevention
• Data Ingestion
– Kernel Smoothing and Moving Average
– Interpolation and Imputation
• Prediction and Pattern Recognition
– Multivariate and Auto-Associative Regression
– Clustering, Similarity Operators and Bayes Networks
• Fault and Anomaly Detection
– Sequential Probability Ratio Tests
– Conditional Probability Filters & Hidden Markov
Models
• Prognosis and Diagnosis
– Bayesian Belief Networks and Probabilistic Inference
– Remaining Useful Life Regression and GPM Models
Rapid Recovery
Autonomous Health Platform ML Technologies
• Data Ingestion
– ELK
– Lucene
• Prediction and Pattern Recognition
– TF-IDF and Bag-of-Words modelling
– Sequence Matcher
– K-nearest Neighbour
• Fault and Anomaly Detection
– Decision Trees and Random Forest
– Sequential Pattern Mining
• Prognosis and Diagnosis
– Recurrent neural Network
– Long short-term memory Predictive Analysis

Log reduction & Anomaly timeline
Remove the noise from thousands of log
events and metrics to identify key events
revealing what happened, in what order
and why
19

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 20
Knowledge
Base Indexing
Entry
Clustering
Model
Generation
Entry Feature
Creation
Log
Cleansing
1 2 3 4 5 6
Expert Input
Knowledge Base
Creation
FeedbackTraining Real-time
Log File
Processing
Timestamp
Correlation &
Ranking
8 97 Batch
Feedback
Anomaly
Detection

Anomaly Detection – High Level
21
Known normal log entry (discard)
Probable anomalous Line (collect)
Log
Collection
File
Type
1
File
Type
2
File
Type
n..
Log File
Anomaly Timeline
Probable
Anomalies

ML & Statistical Algorithms & Concepts Used
•Log entry features to build decision
tree of good / bad entries
•Entropy algorithm used feature
clustering
•SequenceMatcher library used for
clustering log entry clustering
•Expert input used in a feedback look
with ML output
•Functional rules defined for initial
good / bad mapping then feedback
only required for results with
standard deviation of > 2+-
•Features are extracted from log entries
and used for good / bad modeling
•TF-IDF used for weighted knowledge
matching & performance
Bag of
words
Semi-
supervised
ML
Decision
Tree
K-Nearest
Neighbour
22

Autonomous Health Analysis - Ex: Trace File Analyzer
Auto
Recommendation
Confidential – Oracle Restricted

Autonomous Health – TFA Anomaly Timeline
Confidential – Oracle
24

Find the next best window of time
maintenance can be performed
with minimal service impact
25

Maintenance Slot Identification
Apply log
transformation to
penalize higher
values more than
smaller
Trend using
varying
mean over
time
Calculate season
component as
average for each
period
Applying a
convolution
filter
Seasonal
decomposition
showing the cyclic
nature of seasonal
which helps identify
relative usage highs
and lows
30 days of
hourly average
active session
data
Calculate
periodicity using
pandas dataframe
Input
Output
Approach

• Use case
– Identify appropriate maintenance window for performing maintenance activity based
on historical workload patterns.
• Inputs (Training Data)
– The Average Active Sessions (metric is important because it's best representation of
your database system load) in sliding window format. Preferred last 30days data
points before making the prediction.
• AAS = (DB Time / Elapsed Time)
• In other words, AAS is a time-normalized DB Time
• From DB Tables :
– V$ACTIVE_SESSION_HISTORY => COUNT(*) = DB Time in seconds {Cyclic buffer ~4 Hours}
– DBA_HIST_ACTIVE_SESS_HISTORY => 10 * (COUNT(*)) = DB Time in seconds {Since one in 10 samples}

• Implementation
– Depending upon the size of maintenance window slot we identify the granularity with
which data is considered and length of prediction block
– Transformation is performed over the time series to reduce trend (Log, square root,
cube root )
– Seasonality of the time series is extracted
• Fourier Transformation helps in the identification of seasonality period (periodogram)
• Seasonal decomposition helps identify the nature of seasonality (curve)
• The cycle identified from this seasonality helps in identifying what are low and high workload periods
in the cycles.
• We use this information to identify the next best maintenance window.

• Seasonal Decomposition
– Using an observed time series extract a number of component series where each of
these has a certain characteristic or type of behavior.
– Time Series Decomposition
• Trend
– The trend component at time t, which reflects the long-term progression of the series (secular variation)
– A trend exists when there is a persistent increasing or decreasing direction in the data
• Seasonality
– The seasonal component at time t, reflecting seasonality (seasonal variation)
– Seasonality occurs over a fixed and known period (e.g., the quarter of the year, the month, or day of the
week)
• Residual
– The irregular component (or "noise") at time t, which describes random, irregular influences
– It represents the residuals or remainder of the time series after the other components have been removed.

Maintenance Slot Identification
START_TIME CNT
2018-04-11 15:00:00 290
2018-04-11 16:00:00 31120
2018-04-11 17:00:00 21530
2018-04-11 18:00:00 26240
2018-04-11 19:00:00 40520
2018-04-11 20:00:00 54270
2018-04-11 21:00:00 51460
2018-04-11 22:00:00 44310
2018-04-11 23:00:00 25690
START_TIME
2018-04-11 15:00:00 -0.226098
2018-04-11 16:00:00 -0.069821
2018-04-11 17:00:00 -0.350088
2018-04-11 18:00:00 -0.187483
2018-04-11 19:00:00 -0.513240
2018-04-11 20:00:00 0.019737
2018-04-11 21:00:00 0.059213
2018-04-11 22:00:00 -0.011312
2018-04-11 23:00:00 -0.179156
START_TIME
2018-04-11 15:00:00 5.669881
2018-04-11 16:00:00 10.345606
2018-04-11 17:00:00 9.977203
2018-04-11 18:00:00 10.175040
2018-04-11 19:00:00 10.609551
2018-04-11 20:00:00 10.901727
2018-04-11 21:00:00 10.848560
2018-04-11 22:00:00 10.698966
2018-04-11 23:00:00 10.153857
Current Date : 2018-05-12 15:00:00
Current Position in Seasonality : -0.22609829742533585
Best Maintenance Period in next Cycle : 2018-05-12 19:00:00
Worst Maintenance Period in next Cycle : 2018-05-13 08:00:00
Original observation data1 Apply convolution filter & average2 Calculate seasonality3
Use seasonality to
predict best
maintenance window
4

Anomaly Detection with OS and ASH Data
Detect performance
problems
31

Cluster Health Advisor – Applied Machine Learning
• Fault data driven model development
• Applied purpose-built Applied ML for
knowledge extraction
• Expert Dev team scrubs data
• Generates Bayesian Network-based
diagnostic root-cause models
• Uses BN-based run-time models to
perform real-time prognostics
32
Discovers Potential Cluster & DB Problems
CHA Dev Team
ASH
ML
Knowledge
Extraction
BN
Models
Expert
Supervision
CHA
Runtime
Model
Feedback
CHA
CHA
Scrub Data

Models Capture the Dynamic Behavior of all Normal Operation
Models Capture all Normal Operating Modes
0
5000
10000
15000
20000
25000
30000
35000
40000
10:00 2:00 6:00
5100
9025
4024
2350
4100
22050
10000
21000
4400
2500
4900
800
IOPS
user commits (/sec)
log file parallel write (usec)
log file sync (usec)
• Release ships with conservative models to minimize false warnings
• A model captures the normal load phases and their statistics over time, and thus the characteristics for all load
intensities and profiles. During monitoring, any data point similar to one of the vectors is NORMAL.
• One could say that the model REMEMBERS the normal operational dynamics over time
In-Memory Reference Matrix
(Part of “Normality” Model)
IOPS #### 2500 4900 800 ####
User Commits #### 10000 21000 4400 ####
Log File Parallel
Write
#### 2350 4100 22050 ####
Log File Sync #### 5100 9025 4024 ####
… … … … … …

Problem Signatures from Event Paths
Identify a series of events as
connected and representing the
signature of a problem
35

Longest Common Subsequence of Anomalous Entries
36
1. Start by classifying a problem such as an important
ORA or CRS error
2. Find occurrences of the problem across many different
log files
3. Identify anomalous entries and lifecycle events in
chronological order within a predefined time window
around the occurrence of the problem in all the logs
– Time window depends on frequency of message logging
(e.g. 10 mins window for Clusterware)
4. Compare the repeating anomalous / lifecycle entries
to identify the longest common subsequence of
anomalous entries
– These represent the problem signature
– Sequence of events are correlated by component, log file, host &
thread
Find the Finite State Automata(FSA)

Improvisation by constraining LCS identification
1. There are constraints over multiple features which are recorded while analysis:
• LCS must include the major checkpoints present in the knowledge base. Other CRS, ORA log
signatures which are non-fatal and generally appear in correlation with these fatal events must be
present.
• LCS are computed at two levels:
– Overall sequence of anomalous entries, i.e. in correlation with all components and products (sequentials
know-ids for ocssd, gipc, database alert log, asm alert log etc)
– Sequence of anomalies specific to file type.
• We record categorization of log entries specific to host.
• Pre defined knowledge of log structure also helps in optimizing the entries till thread level.
– E.g. Structure of Clusterware log file ([Timestamp]:[Component Name]:[Thread Id]:[Msg])
37

Example signatures and their analysis
• Sample Central Event : 2017-01-19 16:51:20.562 [OCSSD(24862)]CRS-1656: The CSS daemon is terminating due to a fatal error;
Details at (:CSSSC00012:) in /tools/list/grid/orabase/diag/crs/ur102ora3502c/crs/trace/ocssd.trc
38
Knowledge Id Sample Line (States in FSA for central event)
52CC1E8631FC2674E053B580E80AB08D 2016-10-16 21:22:36.520+CRS-5008: Invalid attribute value: en4 for the network interface
52CC1E8632082674E053B580E80AB08D
2016-10-16 21:25:11.516 [OCSSD(6816354)]CRS-1608: This node was evicted by node 3, rwsbs03; details at (:CSSNM00005:) in
/u01/app/crsusr/diag/crs/rwsbs02/crs/trace/ocssd.trc.
52CC1E8632212674E053B580E80AB08D 2016-10-16 21:25:17.927 [OCSSD(18219406)]CRS-1654: Clean up of CRSD resources finished successfully.
52CC1E8631EC2674E053B580E80AB08D 2016-10-16 21:25:17.927 [OCSSD(18219406)]CRS-1655: CSSD on node rwsbs01 detected a problem and started to shutdown.
52CC1E8632272674E053B580E80AB08D
2016-10-16 21:25:19.431 [OCSSD(18219406)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 18219406 experienced fatal signal or
exception code 6.
52CC1E8632202674E053B580E80AB08D
2016-10-16 21:25:21.788 [CRSD(44696012)]CRS-0805: Cluster Ready Service aborted due to failure to communicate with Cluster Synchronization Service with error
[3]. Details at (:CRSD00109:) in /u01/app/crsusr/diag/crs/rwsbs01/crs/trace/crsd.trc.
52CC1E86208C2674E053B580E80AB08D 2016-10-18 02:02:00.835 : CSSD:6684: (:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
52CC1E861F132674E053B580E80AB08D CLSB:6684: Oracle Clusterware infrastructure error in OCSSD (OS PID 12452524): Fatal signal 6 has occurred in program ocssd thread 6684; nested signal count is 1
52CC1E861E552674E053B580E80AB08D Incident 393 created, dump file: /u01/app/crsusr/diag/crs/rwsbs02/crs/incident/incdir_393/ocssd_i393.trc
52CC1E861F332674E053B580E80AB08D 2016-10-18 02:02:07.113 : SKGFD:5655: ERROR: -9(Error 27041, OS Error (IBM AIX RISC System/6000 Error: 47: Write-protected media
52CC1E86207C2674E053B580E80AB08D
2016-10-18 02:02:07.774 : CSSD:5655: clssnmvDiskCreate: Cluster guid ea34893b9442ef79ff642d70699aff9d found in voting disk /dev/rbs01_100G_asm1 does not
match with the cluster guid 7b63590c34fa5f44bf6944aefa4ee85d obtained from the GPnP profile
52CC1E863DB82674E053B580E80AB08D
2017-01-19 16:48:01.057 [OCSSD(24862)]CRS-1649: An I/O error occurred for voting file: /dev/rdsk/c1d16; details at (:CSSNM00059:) in
/tools/list/grid/orabase/diag/crs/ur102ora3502c/crs/trace/ocssd.trc.
52CC1E863DBC2674E053B580E80AB08D
2017-01-19 16:49:40.550 [OCSSD(24862)]CRS-1615: No I/O has completed after 50% of the maximum interval. Voting file /dev/rdsk/c1d16 will be considered not
functional in 99508 milliseconds

Example signatures and their analysis
5 Minute
before
Central
Event
5 Minute
After
Central
Event
Central
Event
52CC1E8
631FC26
74E053B
580E80A
B08D
52CC1E8
6207C26
74E053B
580E80A
B08D
52CC1E8
61F33267
4E053B5
80E80AB
08D
52CC1E8
61E5526
74E053B
580E80A
B08D
52CC1E8
61F13267
4E053B5
80E80AB
08D
52CC1E8
6208C26
74E053B
580E80A
B08D
52CC1E8
6322026
74E053B
580E80A
B08D
52CC1E8
6322726
74E053B
580E80A
B08D
52CC1E8
631EC26
74E053B
580E80A
B08D
52CC1E8
6322126
74E053B
580E80A
B08D
52CC1E8
6320826
74E053B
580E80A
B08D
52CC1E8
63DBC26
74E053B
580E80A
B08D
52CC1E8
63DB826
74E053B
580E80A
B08D
52CC1E8
6722C26
74E053B
580E80A
B08D

Generalizing event signatures over the scope of bug
•BugSignature
Repository 40
Event
Signature 35
Event
Signature 3435
Event
Signature 494
Event
Signature 3948
Event
Signature 292
Event Signature
434933
Node Eviction bug
243645 Timeline
Event
Signature 3434
Event
Signature 3435
Event
Signature 4344
Event
Signature 3048
Event
Signature 202
Event Signature
434983
Node Eviction bug
2747747 Timeline
Event Signature
35
Event Signature
3435
Event Signature
3048
Event Signature
3948
Event Signature
292
Event Signature
434933
New Signature
Check for weighted
probabilistic match

Bug Duplicate Identification
Discovers Duplicate Bugs,
Correlated Issues and Prioritizes
Based Upon Customer Impact
41

Applied Machine Learning – Adaptive Bug Search
• ABS is internally offered in BugDB and MOS
GUI for Dev and Support teams
• ABS helps find problems in the same space
• Allows engineers to get the full context of
past known problems
• Provides debugging clues to help diagnose the
reported problem
• Identifies developers who worked in this
space in the past
Discovers Duplicate Bugs and Correlated Issues

BUG
DB
Applied Machine Learning – Adaptive Bug Search
• Bugs are submitted from over 400 Oracle
products
• Performs ML Logistic Regression on
training set of bugs to generate model
• Displays up to 8 possible duplicates per
bug or SR
• Feedback improves model accuracy
– Direct from developers
– Indirect from bug updates
Discovers Duplicate Bugs and Correlated Issues
ABS Dev Team
Bugs
ML Logistic
Regression
Model
Generation
Expert
Supervision
ABS
Runtime
Model
Dev
Feedback
Bug
Submission Bug and
Duplicates
Together
ABS
Service
Scrub Data

Adaptive Bug Search (ABS) – High Level Flow
• Issues parsed into different features
– Error stack, Trace data, Problem description, etc.
• Issues represented as a cluster of features
– i.e. All bugs in a bug tree contribute towards the feature set
• Logistic Regression applied to build a model
– Model defines the significance of each feature
• Similarity between issues computed using the model
– Identifies the root of the cluster (aka bug tree)
• Feedback used to improve the model
– Feedback is automatically derived based on how the bug gets closed
44Confidential – Oracle Restricted

BUG
DB
Applied Machine Learning – ABS Bug Priority
• Analyze customer and internal bugs
submitted within past 5 years
• Performs ML Linear Regression on training
set of bugs to generate model
• Highlight bugs for Development Managers
to review
– Bugs affecting large numbers of customers
– Bugs with significant impact to high profile
customers
• Feedback improves model accuracy
– Direct from development manager
– Indirect from bug fixes
Discovers Bugs that have the most Customer Impact
ABS Dev Team
Cust.
Bugs
ML Linear
Regression
Model
Generation
Expert
Supervision
ABS BP
Model
Dev
Feedback
Customer files
SR and results
in a new Bug
Bug Priority
with Score
ABS BP
Service
Scrub Data

• Thank you for your feedback!!
• Please continue to reach out to us via
social media
Twitter @sandeshr
Linkedin
https://www.linkedin.com/in/raosandesh/
Questions ?

AIOUG : ODEVCYathra 2018 - Oracle Autonomous Database What Every DBA should know

More Related Content

What's hot (20)

Similar to AIOUG : ODEVCYathra 2018 - Oracle Autonomous Database What Every DBA should know (20)

More from Sandesh Rao (19)

Recently uploaded (20)

AIOUG : ODEVCYathra 2018 - Oracle Autonomous Database What Every DBA should know