Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl

@LukasFittl
Monitoring
PostgreSQL
At Scale

@LukasFittl
Statistics That Matter
Two Tables To Remember
Breaking Down High-Level Statistics
Log Events Worth Knowing
Fingerprinting & Tracing Queries

@LukasFittl
PostgresStatistics
Tables

@LukasFittl
1“Block”=8kB
(usually, check block_size to confirm)

@LukasFittl
StatisticsAreOftenCounters
* except when reset / overrun
Counts only go up*, 
calculate diffs!

@LukasFittl
pg_stat_user_tables
relname: name of the table
seq_scan: # of sequential scans
idx_scan: # of index scans
n_tup_(ins/del/upd): # of rows modified
n_live_tup: live rows
n_dead_tup: dead rows
last_(auto)vacuum: last VACUUM
last_(auto)analyze: last ANALYZE
…

@LukasFittl
SELECT relname, n_live_tup, seq_scan + idx_scan,
100 * idx_scan / (seq_scan + idx_scan)
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC
IndexHitRate
Target: >= 95% on large, active tables

@LukasFittl
pg_statio_user_tables
relname: name of the table
heap_blks_read: blocks from disk / OS cache
heap_blks_hit: blocks from buffer cache
idx_blks_read: index blks from disk
idx_blks_hit: index blks from buffer cache
…

@LukasFittl
SELECT sum(heap_blks_hit) /
nullif(sum(heap_blks_hit + heap_blks_read),0)
FROM pg_statio_user_tables
TableCacheHitRate
Target: >= 99%

@LukasFittl
pg_stat_activity
pid: process ID
backend_type: “client backend”
vs internal processes
state: idle/active/idle in transaction
state_change: time of state change
query: current/last running query
backend_start: process start time
xact_start: TX start time
query_start: query start time
wait_event: what backend is waiting
for (e.g. Lock, I/O, etc)
…

@LukasFittl
#ofConnectionsByState
SELECT state,
backend_type,
COUNT(*)
FROM pg_stat_activity 
GROUP BY 1, 2

@LukasFittl
LongestRunningQuery
SELECT now() - query_start,
query
WHERE state = ‘active’
ORDER BY 1
LIMIT 1

@LukasFittl
AgeOfOldestTransaction
SELECT MAX(now() - xact_start) 
WHERE state <> ‘idle’

@LukasFittl
pg_stat_activity
lock information
https://github.com/postgrespro/pg_wait_sampling

1. Install postgresql contrib package (if not installed)
2. Enable in postgresql.conf 
shared_preload_libraries = ‘pg_stat_statements’
3. Restart your database
4. Create the extension 
CREATE EXTENSION pg_stat_statements;
pg_stat_statements

queryid | 1720234670
query | SELECT * FROM x WHERE y = ?
calls | 5
total_time | 15.249
Query+No.ofCalls+AvgTime

shared_blks_hit | 2447215
shared_blks_read | 55335
Avg.SharedBufferHitRate
hit_rate = shared_blks_hit /
(shared_blks_hit + shared_blks_read)
97.78% Cache Hit Rate

blk_read_time | 14.594
blk_write_time | 465.661
Timespentreading/writingtodisk
track_io_timing = on

pg_qtop
Simple top-like tool that shows
pg_stat_statements data
https://github.com/lfittl/pg_qtop

AVG | QUERY
--------------------------------------------------------------------------------
10.7ms | SELECT oid, typname, typelem, typdelim, typinput FROM pg_type
3.0ms | SET time zone 'UTC'
0.4ms | SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid),
a.attnotnull, a.atttypid, a.atttypmod FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid
= d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = ?::regclass AND a.attnum > ? AND NOT
a.attisdropped ORDER BY a.attnum
0.2ms | SELECT pg_stat_statements_reset()
0.1ms | SELECT query, calls, total_time FROM pg_stat_statements
0.1ms | SELECT attr.attname FROM pg_attribute attr INNER JOIN pg_constraint cons ON attr.attrelid
= cons.conrelid AND attr.attnum = cons.conkey[?] WHERE cons.contype = ? AND cons.conrelid = ?:
:regclass
0.0ms | SELECT COUNT(*) FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE
c.relkind in (?,?) AND c.relname = ? AND n.nspname = ANY (current_schemas(?))
0.0ms | SELECT * FROM posts JOIN users ON (posts.author_id = users.id) WHERE users.login = ?;
0.0ms | SET client_min_messages TO 'panic'
0.0ms | set client_encoding to 'UTF8'
0.0ms | SHOW client_min_messages
0.0ms | SELECT * FROM ad_reels WHERE id = ?;
0.0ms | SELECT * FROM posts WHERE guid = ?;
0.0ms | SELECT ?
0.0ms | SET client_min_messages TO 'warning'
0.0ms | SET standard_conforming_strings = on
0.0ms | SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT ?
0.0ms | SHOW TIME ZONE
pg_qtop -d testdb

AVG | QUERY
--------------------------------------------------------------------------------
0.0ms | SELECT * FROM posts JOIN users ON (posts.author_id = users.id) WHERE users.login = ?;
0.0ms | SELECT * FROM posts WHERE guid = ?;
0.0ms | SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT ?
pg_qtop -d testdb -t posts

AVG | CALLS | HIT RATE | QUERY
--------------------------------------------------------------------------------
0.1ms | 1 | 100.0 | SELECT * FROM users;
0.1ms | 1 | - | SELECT * FROM databases;
0.0ms | 1 | - | SELECT * FROM invoices;
0.0ms | 1 | - | SELECT * FROM query_snapshots;
pg_qtop -d testdb -s select

@LukasFittl
LockStatistics
pg_locks
pid: process ID
(JOIN to pg_stat_activity.pid!)
locktype: type of object being locked
mode: locking type (e.g. AccessExclusive)
granted: Lock Granted vs Being Waited For
…

@LukasFittl
LockStatistics
pg_locks
SELECT *
FROM pg_locks
WHERE NOT granted

@LukasFittl
LockStatistics
pg_locks
SELECT locktype,
mode,
COUNT(*)
FROM pg_locks
WHERE granted
GROUP BY 1, 2

@LukasFittl
CheckpointStatistics
pg_stat_bgwriter
checkpoints_timed: # of scheduled checkpoints
checkpoints_req: # of requested checkpoints
1. TimeBetweenCheckpoints
2. %ofTimedCheckpoints

@LukasFittl
autovacuum
=> SELECT pid, query FROM pg_stat_activity
WHERE query LIKE 'autovacuum: %';
10469 | autovacuum: VACUUM ANALYZE public.schema_columns
12848 | autovacuum: VACUUM public.replication_follower_stats
28626 | autovacuum: VACUUM public.schema_index_stats
| (to prevent wraparound)
(3 rows)
pg_stat_activity

@LukasFittl
autovacuum
pg_stat_activity

@LukasFittl
autovacuum
pg_stat_progress_vacuum
relid: OID of the table
phase: current VACUUM phase
heap_blks_total: Heap Blocks Total
heap_blks_scanned: Heap Blocks Scanned
heap_blks_vacuumed: Heap Blocks Vacuumed
…

@LukasFittl
autovacuum
pg_stat_progress_vacuum

@LukasFittl
pg_stat_replication
client_addr: ip address of the follower
backend_start: replication start time
state: replication state
(ideally = streaming)
replay_location: WAL location

@LukasFittl
pg_stat_replication
SELECT client_addr,
pg_wal_lsn_diff(
pg_current_wal_lsn(),
replay_location)
FROM pg_stat_replication
Replication Lag in Bytes, Per Follower

@LukasFittl
pg_stat_replication
Replication Lag in Bytes, Per Follower

@LukasFittl
CPU&I/OUtilization

@LukasFittl
“We had an outage yesterday at
10am - whathappened?”

@LukasFittl
Keeping Historic
Statistics Data
IsEssential

@LukasFittl
DIYMonitoringHack:
Save pg_stat_activity and
pg_stat_database
every 10 seconds
into a separate monitoring database

@LukasFittl
pg_stat_activity
- Number & State of Connections
- Oldest Query Still Running
- Oldest Transaction Still Open
- Blocked Queries

@LukasFittl
pg_stat_database
- Transactions Per Second
- Data Read Per Second
- Rows Updated/etc Per Second
- Deadlocks Per Second
- …

@LukasFittl
Ability to Drill Down
From “HighCPUUtilization”
To Specific Set of Queries

@LukasFittl
CPUUtilization
pg_stat_statements.total_runtime

@LukasFittl
I/OUtilization
pg_stat_statements.blk_read_time
pg_stat_statements.blk_write_time

@LukasFittl
CacheHitRatio%
pg_stat_statements.shared_blks_hit
pg_stat_statements.shared_blks_read
pg_stat_database.blks_hit
pg_stat_database.blks_read

@LukasFittl
TemporaryFilesWritten
pg_stat_statements.temp_blks_written
pg_stat_database.temp_bytes

@LukasFittl
LOG: duration: 4079.697 ms execute <unnamed>:
SELECT * FROM x WHERE y = $1 LIMIT $2
DETAIL: parameters: $1 = 'long string', $2 = ‘1'
SlowQueries
log_min_duration_statement
= 1000 ms

@LukasFittl
auto_explain
logs the query plan 
for specific slow queries

@LukasFittl
CancelledQueries
ERROR: canceling statement due to
statement timeout
STATEMENT: SELECT 1
ERROR: canceling statement due to 
user request
STATEMENT: SELECT 1
…

@LukasFittl
log_lock_waits = on
LOG: process 20679 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 after 1000.115 ms
LOG: process 20678 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 after 1000.126 ms
LOG: process 15533 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.129 ms
LOG: process 15758 still waiting for ShareLock on transaction 250175899 after 1000.073 ms
LockWaits

@LukasFittl
archive_commandFailures
LOG: archive command failed with
exit code 1
DETAIL: The failed archive command
was: /my_backup_script.sh pg_xlog/
0000000100025DFA00000023

@LukasFittl
OutofMemory
ERROR: out of memory
DETAIL: Failed on request of size 408028.
QUERY: SELECT 1 ...

@LukasFittl
OutofConnections
FATAL: remaining connection slots
are reserved for non-replication
superuser connections

@LukasFittl
ServerCrash/Segfault
LOG: server process (PID 660) was
terminated by signal 6: Aborted
DETAIL: Failed process was running:
SELECT pg_advisory_lock(1, 2);
LOG: terminating any other active
server processes
WARNING: terminating connection
because of crash of another server
process
…

@LukasFittl
TXIDWraparound
WARNING: database “mydb" must
be vacuumed within 938860
transactions
HINT: To avoid a database
shutdown, execute a full-
database VACUUM in
“mydb".

@LukasFittl
TXIDWraparound
ERROR: database is not accepting
commands to avoid wraparound
data loss in database “mydb”
HINT: Stop the postmaster and use a
standalone backend to vacuum
that database. You might also
need to commit or roll back
old prepared transactions.

@LukasFittl
Fingerprinting
Identifying & Grouping Queries

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ‘value’
A

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ‘value’
SELECT a, b
FROM public.test
WHERE col = ‘other_value’
A
B

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ?
SELECT a, b
FROM public.test
WHERE col = ?
A
A

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ?
SELECT a, b —— COMMENT
FROM public.test
WHERE col = ?
A
B

@LukasFittl
# SELECT queryid, query FROM pg_stat_statements;
queryid | query
------------+----------------------------------------
1115711211 | SELECT a, b FROM public.test WHERE col = $1
(1 row)
pg_stat_statements

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ?
SELECT a, b —— COMMENT
FROM public.test
WHERE col = ?
A
queryid=1115711211
queryid=1115711211
A

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ?
SELECT b, a —— COMMENT
FROM public.test
WHERE col = ?
A
queryid=1115711211
queryid=2511327719
B

@LukasFittl
irb> PgQuery.fingerprint(
‘SELECT a, b FROM public.test WHERE col = $1’)
=> 0254f1e78f2d47b258d7b022f3dfa5794351a75128
pg_query

@LukasFittl
SELECT a, b
FROM public.test
WHERE col = ?
SELECT b, a /* COMMENT */
FROM public.test
WHERE col = ?
A
fingerprint=0254f1e78f2d47b258d7b022f3dfa5794351a75128
fingerprint=0254f1e78f2d47b258d7b022f3dfa5794351a75128
A

@LukasFittl
PgQuery.fingerprint
- Based on Postgres Parsetree
- Table names, not OIDs
- Identical across servers 
& Postgres versions
https://github.com/lfittl/libpg_query/wiki/Fingerprinting

@LukasFittl
Tracing Queries
Based OnTheir QueryOrigin

@LukasFittl
application: pganalyze
controller: graphql
action: graphql
line: /app/graphql/organization_type.rb …
graphql: getOrganizationDetails.logVolume24h
request_id: 44bd562e-0f53-453f-831f-498e61ab6db5

@LukasFittl
github.com/basecamp/marginalia
Automatic
QueryAnnotationsForRubyonRails

@LukasFittl
3Take-Aways
1. Collect Historic Metrics
2. Focus on Drill-Down To Query Level
3. Annotate Your Queries With
Their Origin

@LukasFittl
Scale Your Postgres:
citusdata.com
Thanks!

Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl

More Related Content

What's hot (20)

Similar to Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl (20)

More from Citus Data (20)

Recently uploaded (20)

Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl