Wait! What’s going on inside my database? (PASS 2023 Update)

Wait! What’s going on inside
my database?
PostgreSQL and the Science of
Database Performance
Jeremy Schneider

Jeremy
DB & Perf Engineer / AWS
Slack: pgtreats.info/slack-invite
Blog: ardentperf.com
Twitter/X: @jer_s
Organizer / Seattle PG User Group
• Engineering for Aurora and RDS Open Source: Aurora PostgreSQL
first GA release (2017), ICU & collation, data durability &
corruption protection, performance, fleet data collection and
analysis, early logical replication work, new major versions,
addressing complex operational issues, recruiting & training
• Launched an internal Amazon-wide dedicated PostgreSQL chat
channel in 2018 which now has thousands of members and
numerous technical discussions daily
• Founder of "RAC Attack" Community Driven Oracle Cluster
Database Workshop – almost 40 events across 15 countries
between 2011 and 2016
• Participant, speaker and/or organizer at Linux and Oracle
user groups & conferences since the early 2000’s. PostgreSQL
user groups & conferences since 2017.
• Database blog since 2007, Oracle ACE Alumni, Oak Table
Schneider

#PASSDataCommunitySummit
About PostgreSQL
1970: Mathematician Edgar F. Codd, working as researcher for
IBM, publishes “A Relational Model of Data for Large Shared Data
Banks”
1973: Michael Stonebraker and Eugene Wong at University of
California Berkeley seek funding and begin development of a
relational database called INGRES
1986: Michael Stonebraker and Lawrence A. Rowe at University of
California Berkeley publish “The Design of POSTGRES” – a new
database that is the successor to INGRES
1994: Andrew Yu and Jolly Chen at University of California
Berkeley add support for the SQL language
1996: Transition to non-university core team of volunteers, official
release under new name POSTGRESQL
1985

About PostgreSQL

About Database Performance

Delfador Chibi by Peileppe CC0
https://openclipart.org/detail/19811/delfador-chibi
1990’s Manager:
“Dear DBA: Expert consultants
have taught us that if the Buffer
Cache Hit Ratio (BCHR) is below
90% then the system
immediately needs an expensive
tuning engagement.
Please report any databases that
have BCHR < 90%.”

Nørgaard, Mogens et al. Oracle Insights: Tales of
the Oak Table. Berkeley, CA: Apress/OakTable
Press, 2004. p76-77.

Millsap, Cary V. Optimizing Oracle Performance.
Sebastopol, CA: OReilly, 2003. p225, 240, 258-259
R = S + W
“How long the
SQL takes to
run”
See also:
• Shallahamer, Craig.
Forecasting Oracle
Performance. Berkeley,
CA: Apress, 2007.

Active Session Sampling
(JB’s notebook, 2004)
Images & Quotes
Used With Permission

January 2022
March 2023

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mariinsky Theatre, St. Petersburg
by Sandra Cohen-Rose and Colin Rose (Montreal, Canada)
CC BY-SA

CC BY-SA
Wait Events

CC BY-SA
• 1990s: Database kernel instrumentation:
• Counters and tools to snapshot/compare them
• Events (log a message under certain circumstances)
• 1992: Unable to solve a performance problem, engineers added event
code in version 7.0.12 capable of emitting log messages when the
database waited for something
• First exposed in V$SESSION_WAIT and later in V$SESSION (equivalent
of pg_stat_activity)
• PostgreSQL recognized and built on concepts that had become
widespread across the industry
Wait Events

CC BY-SA
Wait Events

CC BY-SA
Wait Events
Millsap, Cary V. Optimizing Oracle Performance.
Sebastopol, CA: OReilly, 2003. p225, 240, 258-259
R = S + W
“How long the
SQL takes to
run”
See also:
• Shallahamer, Craig.
Forecasting Oracle
Performance. Berkeley,
CA: Apress, 2007.

CC BY-SA
Wait Events
Active Session Sampling
(JB’s notebook, 2004)
Images & Quotes
Used With Permission

CC BY-SA
“But why are these events called wait events?
…
In short, when a session is not using the CPU, it may be waiting for
a resource, an action to complete, or simply more work. Hence,
events that associated with all such waits are known as wait
events.”
Shee, Richmond, Kirtikumar Deshpande, and K. Gopalakrishnan. Oracle Wait Interface a Practical Guide to Performance
Diagnostics & Tuning. New York: London, 2004. p16
Wait Events

CC BY-SA
High-Level Idea:
Caveats:
• OS scheduling/runqueue
• Measurement overhead
• OS kernel or underlying library CPU time (e.g. I/O)
Wait Events
The database is WAITING any time when it’s not running on the CPU

CC BY-SA
Idle
Waiting
Cpu
Wait Events
psql> SELECT…

CC BY-SA
Wait Events
https://youtu.be/RpoC3UcG5YA

CC BY-SA
• Version 9.6
• Aa65de0 – 11 Sep 2015 – Autogenerate lwlocknames.[c|h]
• 53be0b1 – 10 Mar 2016 – Heavy/Lightweight Locks, Buffer Pins
Wait Events

CC BY-SA
• Version 9.6
• Aa65de0 – 11 Sep 2015 – Autogenerate lwlocknames.[c|h]
• 53be0b1 – 10 Mar 2016 – Heavy/Lightweight Locks, Buffer Pins
• Version 10
• 6f3bd98 – 4 Oct 2016 – Latches & Sockets, Clients, Main Loops
• 249cf07 – 18 Mar 2017 – I/O
• Fc70a4b – 26 Mar 2017 – Background and Auxiliary Processes
• Version 11
• 1804284 – 20 Dec 2017 – Parallel-Aware Hash Joins
Wait Events

CC BY-SA
• Version 12
• Add a wait event for fsync of WAL segments (Konstantin Knizhnik)
• Ensure that TimelineHistoryRead and TimelineHistoryWrite wait states are reported in
all code paths that read or write timeline history files (Masahiro Ikeda)
• Version 13
• Rename various wait events to improve consistency (Fujii Masao, Tom Lane)
• Report a wait event while creating a DSM segment with posix_fallocate() (Thomas
Munro)
• Add wait event VacuumDelay to report on cost-based vacuum delay (Justin Pryzby)
• Add wait events for WAL archive and recovery pause (Fujii Masao)
• The new events are BackupWaitWalArchive and RecoveryPause.
• Add wait events RecoveryConflictSnapshot and RecoveryConflictTablespace to monitor
recovery conflicts (Masahiko Sawada)
• Improve performance of wait events on BSD-based systems (Thomas Munro)
• Version 14
• Add wait event WalReceiverExit to report WAL receiver exit wait time (Fujii Masao)
• Wake up for latch events when the checkpointer is waiting between writes. This
improves responsiveness to backends sending sync requests. The change also creates a
proper wait event class for these waits. (Thomas Munro)
• Version 15
• Add wait events for local shell commands. The new wait events are used when calling
archive_command, archive_cleanup_command, restore_command and
recovery_end_command. (Fujii Masao)
• Correct the name of the wait event for SLRU buffer I/O for commit timestamps. This wait
event is named CommitTsBuffer according to the documentation, but the code had it as
CommitTSBuffer. Change the code to match the documentation, as that way is more
consistent with the naming of related wait events. (Alexander Lakhin)
• Re-activate reporting of wait event SLRUFlushSync. Reporting of this type of wait was
accidentally removed in code refactoring. (Thomas Munro)
• Version 16
• Add wait event SpinDelay to report spinlock sleep delays (Andres Freund)
• Create new wait event DSMAllocate to indicate waiting for dynamic shared memory
allocation. Previously this type of wait was reported as DSMFillZeroWrite, which was
also used by mmap() allocations. (Thomas Munro)
• Allow parallel application of logical replication. Wait events LogicalParallelApplyMain,
LogicalParallelApplyStateChange, and LogicalApplySendData were also added. Column
leader_pid was added to system view pg_stat_subscription to track parallel activity.
(Hou Zhijie, Wang Wei, Amit Kapila)
• Have wal_retrieve_retry_interval operate on a per-subscription basis. Previously the
retry time was applied globally. This also adds wait events >LogicalRepLauncherDSA
and LogicalRepLauncherHash. (Nathan Bossart)
• Version 17
• Support custom wait events for wait event type "Extension” (Masahiro Ikeda)
Wait Events

CC BY-SA
Wait Events
src/include/pgstat.h

CC BY-SA
Wait Events
doxygen.postgresql.org

CC BY-SA
Gaps after migrating to Open Source/Community PostgreSQL
• Wait Event Counters and Cumulative Times
• Wait Event Arguments (object, block, etc)
• Comprehensive tracking of CPU time (POSIX rusage)
• Ability to find previous SQL for COMMIT/ROLLBACK
• Needed to identify which transaction is committing. (SQL comments on commit?)
• On-CPU State
• SQL Execution Stage (parse/plan/execute/fetch)
• SQL Execution Plan Identifier in pg_stat_statements
• Current plan node
• Progress on long operations (e.g. large seqscan)
• Better runtime visibility into PLs
Wait Events

I can haz Wait Events?
Solving Problems with Wait Events in PostgreSQL
By Antony Griffiths (Flickr), CC BY

101
What’s happening right
now?

Solving Problems With Wait Events

401
What happened?
(It’s not happening now.)

Repository of Historical Performance Data (pg_stat_activity)
Top SQL
Top Wait Events
EXPLAIN ANALYZE with Buffers, IO timing, etc
Investigate WAIT EVENT & STEP Taking The Most TIME

Repositories of Historical Performance Data
(Active Session Sampling of Wait Events)
• https://wiki.postgresql.org/wiki/Monitoring
• Amazon RDS Performance Insights
• RDS for PostgreSQL 10+
• Aurora PostgreSQL-Compatible Edition 9.6+
(v10 Wait Events were backported)
• Rolling 7 days of history is free. Up to 2 years on paid tier.

(one of many options)

Aurora PostgreSQL-Compatible:
• AWS Documentation covers
Aurora-Specific Wait Events
• Shares Code With Community
PostgreSQL (and merges
regularly)

Thank you
aws.amazon.com/rds/postgresql
Jeremy Schneider
Twitter/X: @jer_s

Stuffed Elephant Store
• A unique service that produces on-
demand stuffed elephants
• Multiple sizes with long or short fur
• Any color the customer wants, as
long as its blue
Breibeest (Flickr), CC BY

See the query text
and the wait events
by query
Look back 7 days or
as much as 2 years
to find activity
Amazon RDS Performance Insights

explain.depesz.com

Explains how
Postgres plans to
execute a query
Shows the type of
operation, the
estimated cost,
and the estimated
number of rows
Execution Plans

Contains the
structure of all
objects in the
database
Statistics views
shows usage of the
objects
System Catalogs

PostgreSQL has a
rich set of index
types
Base functionality
can be enhanced
by specialized
extensions
Indexes

It’s so simple!

Wait! What’s going on inside my database? (PASS 2023 Update)

Session evaluation
Your feedback is important to us
Evaluate this session at:
www.PASSDataCommunitySummit.com/evaluation

Thank you
aws.amazon.com/rds
Jeremy Schneider
Twitter/X: @jer_s

Wait! What’s going on inside my database? (PASS 2023 Update)

More Related Content

Similar to Wait! What’s going on inside my database? (PASS 2023 Update) (20)

Recently uploaded (20)

Wait! What’s going on inside my database? (PASS 2023 Update)