DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals

Architect, Enterprise and Cloud Manageability
Oracle America
John Beresniewicz
Know your performance fundamentals
DB Time, Average Active Sessions, and ASH Math

The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making purchasing decisions. The
development, release, timing, and pricing of any features or functionality described for Oracle’s
products may change and remains at the sole discretion of Oracle Corporation.
Safe harbor statement
Copyright © 2020, Oracle and/or its affiliates2

4
3
2
1
Issues
ASH Math
Average Active Sessions
Database Time
Agenda

Copyright © 2020, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted4
Concepts
• The core ideas
Instrumentation
• Services supporting the core ideas
Visualization
• Presentation of concepts using instrumentation
Approach

Database Time

Performance is about work and time
Performance measures:
• Throughput = work / time
• Latency (response time) = time / work
Database time is about time spent doing database work
Database time is the foundation of Diagnostic and Tuning Packs:
• ADDM, AWR, SPA, SQL Monitor, EM Perf Page, Top Activity, ASH Analytics
Logic of DB Time

Time spent processing database client calls
Specifically:
• Foreground time spent executing database calls (SQL)
• Actively working (on CPU) or waiting to work (in wait)
Note:
• Idle wait time is not included in DB Time
• Background time is not included in DB Time
Definition of DB Time

DB Time is spent on CPU or in wait
DB Time clock starts upon client call to DB and stops with response to client
Model of DB Time
CPU WAIT
DB TIME

Timers built into code path to measure DB Time spent in certain operations:
• Call begin/exit (total time in call)
• Parsing
• PL/SQL compilation
CPU time fetched from OS
Can be tricky given call stack complexity
• V$SYS_TIME_MODEL
• DBA_HIST_SYS_TIME_MODEL
• V$SYSMETRIC_HISTORY
Instrumentation: Time Model

The following are critical Time Model measures:
DB Time
• Clearest and coarsest instrumentation points, highly reliable
• Includes run-queue time on CPU saturated systems
DB CPU
• Fetched from OS counters by foregrounds
• Includes only actual time on CPU, not time on run-queue
Also:
• background cpu time
• background elapsed
Critical Measures

DB call execution involves many queueing, serialization, and synchronization mechanisms
• Locks, latches, mutexes
• Cores / CPUs / instance caging
External I/O
DB code is instrumented to time these “wait events”
• V$WAITCLASSMETRIC_HISTORY (???)
• V$EVENT_HISTOGRAM
Instrumentation: Wait Model
DB CPU WAITUser I/O WAIT
DB Time

CPU saturation is an apex problem
• Processing capacity exhausted, no more work can be done
Instrumentation distortion
• Wait event timing
• Invisible / missing DB Time
Visibility issues, difficult to measure
Run queue: silent killer

Run-queue distortion: wait path
DB file scattered read Run queue CPU
actual event latency
measured event latency
1. Event finishes and foreground queues for CPU
2. End of wait timestamp requires CPU
3. Measured event latency includes run-queue time
end timestampbegin timestamp

DB Time exceeds total DB CPU time under CPU saturation
Run-queue not always reliably measured
• Check V$OSSTAT.STAT_NAME = ’RSRC_MGR_CPU_WAIT_TIME’
Run-queue distortion: CPU path
CPU Run queue CPU
DB Time
DB CPU DB CPU

Quantifying true CPU capacity increasingly difficult
• Hyper-threading
• Virtualization and hypervisors
• Containers and cloud
Core utilization is the used fraction of raw core time available
• When < 1 system is not CPU saturated
• When > 1 system exceeding raw core capacity through hyper-threading
Core Utilization
!" #$% + '()*+,-./0 )1.
23(1420 5672 ∗ )-,2 )-./5

Viz: EM DB Performance Page

Average Active Sessions

Labeling the Perf Page chart
DB Time per sec = “Centi-users per second”?

Over some time interval, e.g. an AWR snapshot
AAS is the rate at which DB Time is accumulates in the system
We also think of AAS as the load on the database
Typically, total sessions >> AAS (due to session idle time)
• However poor session management is a classic performance pitfall
Definition of AAS
!"#$%&# !'()"# *#++),-+ = !!* = /01 2)3#
#4%5+#6 ()3#

AAS = rate of change of DB Time over time
Integrating AAS = computing DB Time
Perf Page is literally a picture of DB Time
AAS and DB Time
!"#$%& '&''$()' # =
+ ,- .$/&
+#
0
12
13
!"#$%& '&''$()' # +# = ,- .$/&
#2
#1

AAS for comparisons:
• Across time periods – how does load vary?
• Across similar systems – which DB is busiest?
Per-core normalization
• Compare different systems’ profiles
• AWR1page script
Normalization and AAS
!""#(%&'()
*+,- *+./(
!""#(012)
*+,- *+./(

Q: Whose time is DB Time?
A: Business entities using database services to accomplish time-bound goals
• Interactive end-users
• Automated business processes
”Enterprise time” is the same regardless of which database or service is being invoked
• One second spent waiting for DB1 response is no different than one waiting for DB2
Enterprise DB Time can be thought of as a fungible quantity
Fungibility of DB Time

Viz: EM Enterprise Loadmap

Active session state sampled into memory every second (1000 msec)
• Active = in DB call and not in idle wait
Fixes-up values unknown at sampling time
• TIME_WAITED
Sampled session state is either ON CPU or in WAIT event
• Session in WAIT is known
• Session presumed ON CPU if not in WAIT
Wait event samples biased toward longer latency waits
Mechanism design is latch-less, efficient, reliable
Instrumentation: Active Session History
Conforms to the model of DB Time

Viz: EM Top Activity Page

Copyright © 2020, Oracle and/or its affiliates |29
ASH Math

Decompose DB Time by aggregating ASH sample rows
• Grouped over dimensions, classic fact table query
• Many analysis columns: 30+ in DB 10.2, 110+ in DB 12.2
Filter and Aggregate to isolate and expose how DB Time is spent
Normalize to AAS: divide DB Time by aggregation interval time
Estimating DB Time using ASH
!"#$% ∗ = () *+,- (/-0/)
select trunc(sample_time,'MI’), sql_id
,sum(usecs_per_row)/(60*power(10,6)) as AAS
from v$active_session_history
group by trunc(sample_time,'MI’), sql_id;

Summing samples is equivalent to computing Riemann estimate of DB Time integral
• !"#$% &'( )*+,-./ = *!%12.'.))1"$) %
• ∆% = )*+,-1$4 1$%.52*- = 1 ).!
The calculus of ASH Math
∆%
ASH
Sample
count
time
sessions
*!%12.'.))1"$) %

Both are literal pictures of DB Time:
• Perf Page is top-down instrumentation:
• Time Model & Wait model
• Top Activity is bottom-up instrumentation
• ASH
It’s the Fundamental Theorem of Calculus!
Perf Page ≡ Top Activity Page

Beware: the ASH sampler is biased toward longer latencies
• Higher probability of being sampled
Common mistakes:
• AVG(TIME_WAITED) ≠ average event latency
• SUM(TIME_WAITED) ≠ DB wait time
Bad ASH math

Viz: ASH Analytics Loadmap

IN_CONNECTION_MGMT VARCHAR2(1)
IN_PARSE VARCHAR2(1)
IN_HARD_PARSE VARCHAR2(1)
IN_PLSQL_RPC VARCHAR2(1)
IN_PLSQL_COMPILATION VARCHAR2(1)
IN_JAVA_EXECUTION VARCHAR2(1)
IN_BIND VARCHAR2(1)
ASH math can estimate DB Time spent in “un-timeable” operations
1. Timestamping may be relatively expensive for very low latency operations
2. Instead, set a session state bit ON when operation begins – very cheap
3. When operations completes, set the bit OFF
4. ASH sampler samples bit vector
5. Using ASH math: COUNT(samples with bit ON) = DB Time in operation
Bit vector magic

So what can TIME_WAITED tell us?
Recall: ASH sampling is independent of session activity and on strict schedule
TIME_WAITED: advanced ASH math
time
wait event E CPUCPU
1000 ms
200 ms ASH Sample time
Pr # $%&'()* = ,-./#_12.-#3
4%&'(567 .68)9:%(

• Questions:
• If a sampled 200 msec event counts for 1000 msec of DB Time, what about the other 800 msec?
• How can the likelihood of sampling an event provide an estimate for how often it occurs?
Estimated number of occurrences is number required to minimally cover sampling interval
TIME_WAITED: estimating event counts
!"#_%&'(#)*)+, = GREATEST
456789:; 9:<=>?58
#@A!_BC@#!D
, 1
1000 ms
200 ms
time
wait event E wait event E wait event E wait event E wait event E

Estimating average latency properly
select wait_class
,TRUNC(DBtime_usecs / est_count) as AvgLatency_usec
,TRUNC(bad_ash_math) as bad_ash_math
from
(select wait_class
,SUM(GREATEST(usecs_per_row / time_waited, 1)) est_count
,SUM(usecs_per_row) DBtime_usecs
,AVG(time_waited) bad_ash_math
from
v$active_session_history
where time_waited > 0
group by wait_class
)

Bad ASH math vs. properly estimated average latencies:
Results
WAIT_CLASS AVGLATENCY_USEC BAD_ASH_MATH
Concurrency 702 21139
User I/O 735 18380
Administrative 104910 135946
System I/O 543 855
Other 71 65951
Scheduler 9883 10901
Configuration 254143 330538
Cluster 155 5632
Application 277 26880
Commit 494 927
Network 399 51402
Idle 1004 1006

Issues

Copyright © 2020, Oracle and/or its affiliates
Database is only one of many layers in the Enterprise technology stack
Determine how much DB Time contributes to total Enterprise time
Beware of instrumentation infatuation
DB Time is not the only relevant time
Database Storage
Client Services
Micro
services
Database Storage
response time profile

Principal Database Engineer, AWS
Kevin Closson
“Everything is a CPU problem”

A curious case from @oradiag

CPU Wait on Performance Page is derived not measured
Time model instrumentation:
• Total DB Time
• Total DB CPU
Wait model instrumentation:
• Non-idle wait (I/O and everything else)
CPU Wait: estimating run-queue time
!"# $%&' = )* +&,- − )* !"# + $01+ DB CPU
I/O WAIT
WAIT
DB TIME
CPU Wait

Always some “CPU bleed” into measured wait time
• Timestamps
• Call stack traversal
CPU time under wait double-counted
• DB Time
• DB CPU
Shrinking event latencies exacerbate the problem
• May require instrumentation adjustments
CPU under wait
CPUWAIT
CPUWAIT
CPU under wait

AWR report from @kevinclosson
• “What’s up with % DB Time greater than 100?”
Close and painful scrutiny revealed CPU under wait
• 50% of I/O wait was CPU time
• AWR1page script developed
Identifying CPU under wait
Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Total Wait Wait % DB Wait
Event Waits Time (sec) Avg(ms) time Class
------------------------------ ----------- ---------- ---------- ------ --------
db file parallel read 139,819 469.8 3.36 78.1 User I/O
DB CPU 243 40.4
db file sequential read 201,921 52.4 0.26 8.7 User I/O
DB Time << CPU + Wait

@JBeresniewicz
john.beresniewicz@oracle.com
Thank you

DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals

DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals

More Related Content

What's hot (20)

Similar to DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals (20)

More from John Beresniewicz (12)

Recently uploaded (20)

DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals