Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
0 likes128 views
The document discusses best practices for monitoring PostgreSQL databases at scale, highlighting important statistics to track, including user tables and query workload. It emphasizes the need for historic metrics collection, query performance analysis, and effective use of PostgreSQL's monitoring tools, like pg_stat_activity and pg_stat_statements. The conclusion suggests annotating queries for better tracking and diagnosing performance issues.
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
- 3. @LukasFittl Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Three Log Events Worth Knowing Query Annotations
- 4. @LukasFittl Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Three Log Events Worth Knowing Query Annotations
- 6. @LukasFittl 1“Block”=8kB (usually, check block_size to confirm)
- 8. @LukasFittl StatisticsAreOftenCounters * except when reset / overrun Counts only go up*, calculate diffs!
- 10. @LukasFittl pg_stat_user_tables relname: name of the table seq_scan: # of sequential scans idx_scan: # of index scans n_tup_(ins/del/upd): # of rows modified n_live_tup: live rows n_dead_tup: dead rows last_(auto)vacuum: last VACUUM last_(auto)analyze: last ANALYZE …
- 11. @LukasFittl SELECT relname, n_live_tup, seq_scan + idx_scan, 100 * idx_scan / (seq_scan + idx_scan) FROM pg_stat_user_tables ORDER BY n_live_tup DESC IndexHitRate Target: >= 95% on large, active tables
- 12. @LukasFittl pg_statio_user_tables relname: name of the table heap_blks_read: blocks from disk / OS cache heap_blks_hit: blocks from buffer cache idx_blks_read: index blks from disk idx_blks_hit: index blks from buffer cache …
- 13. @LukasFittl SELECT sum(heap_blks_hit) / nullif(sum(heap_blks_hit + heap_blks_read),0) FROM pg_statio_user_tables TableCacheHitRate Target: >= 99%
- 15. @LukasFittl pg_stat_activity pid: process ID backend_type: “client backend” vs internal processes state: idle/active/idle in transaction state_change: time of state change query: current/last running query backend_start: process start time xact_start: TX start time query_start: query start time wait_event: what backend is waiting for (e.g. Lock, I/O, etc) …
- 16. @LukasFittl #ofConnectionsByState SELECT state, backend_type, COUNT(*) FROM pg_stat_activity GROUP BY 1, 2
- 17. @LukasFittl LongestRunningQuery SELECT now() - query_start, query FROM pg_stat_activity WHERE state = ‘active’ ORDER BY 1 LIMIT 1
- 18. @LukasFittl AgeOfOldestTransaction SELECT MAX(now() - xact_start) FROM pg_stat_activity WHERE state <> ‘idle’
- 21. @LukasFittl 1. Install postgresql contrib package (if not installed) 2. Enable in postgresql.conf shared_preload_libraries = ‘pg_stat_statements’ 3. Restart your database 4. Create the extension CREATE EXTENSION pg_stat_statements; Enablingpg_stat_statements
- 22. @LukasFittl SELECT * FROM pg_stat_statements; userid | 10 dbid | 1397527 query | SELECT * FROM x WHERE calls | 5 total_time | 15.249 rows | 0 shared_blks_hit | 451 shared_blks_read | 41 shared_blks_dirtied | 26 shared_blks_written | 0 local_blks_hit | 0 pg_stat_statements
- 24. @LukasFittl queryid | 1720234670 query | SELECT * FROM x WHERE y = ? calls | 5 total_time | 15.249 Query+No.ofCalls+AvgTime
- 25. @LukasFittl shared_blks_hit | 2447215 shared_blks_read | 55335 Avg.SharedBufferHitRate hit_rate = shared_blks_hit / (shared_blks_hit + shared_blks_read) 97.78% Cache Hit Rate
- 26. @LukasFittl blk_read_time | 14.594 blk_write_time | 465.661 Timespentreading/writingtodisk track_io_timing = on
- 27. @LukasFittl LockStatistics pg_locks pid: process ID (JOIN to pg_stat_activity.pid!) locktype: type of object being locked mode: locking type (e.g. AccessExclusive) granted: Lock Granted vs Being Waited For …
- 28. @LukasFittl LockStatistics pg_locks SELECT * FROM pg_locks WHERE NOT granted
- 29. @LukasFittl LockStatistics pg_locks SELECT locktype, mode, COUNT(*) FROM pg_locks WHERE granted GROUP BY 1, 2
- 31. @LukasFittl autovacuum => SELECT pid, query FROM pg_stat_activity WHERE query LIKE 'autovacuum: %'; 10469 | autovacuum: VACUUM ANALYZE public.schema_columns 12848 | autovacuum: VACUUM public.replication_follower_stats 28626 | autovacuum: VACUUM public.schema_index_stats | (to prevent wraparound) (3 rows) pg_stat_activity
- 33. @LukasFittl autovacuum pg_stat_progress_vacuum relid: OID of the table phase: current VACUUM phase heap_blks_total: Heap Blocks Total heap_blks_scanned: Heap Blocks Scanned heap_blks_vacuumed: Heap Blocks Vacuumed …
- 35. @LukasFittl Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Three Log Events Worth Knowing Query Annotations
- 36. @LukasFittl “We had an outage yesterday at 10am - whathappened?”
- 38. @LukasFittl DIYMonitoringHack: Save pg_stat_activity and pg_stat_database every 10 seconds into a separate monitoring database
- 39. @LukasFittl pg_stat_activity - Number & State of Connections - Oldest Query Still Running - Oldest Transaction Still Open - Blocked Queries
- 40. @LukasFittl pg_stat_database - Transactions Per Second - Data Read Per Second - Rows Updated/etc Per Second - Deadlocks Per Second - …
- 41. @LukasFittl Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Three Log Events Worth Knowing Query Annotations
- 42. @LukasFittl Ability to Drill Down From “HighCPUUtilization” To Specific Set of Queries
- 43. @LukasFittl
- 44. @LukasFittl
- 45. @LukasFittl
- 49. @LukasFittl TemporaryFilesWritten pg_stat_statements.temp_blks_written pg_stat_database.temp_bytes
- 50. @LukasFittl Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Three Log Events Worth Knowing Query Annotations
- 51. @LukasFittl LOG: duration: 4079.697 ms execute <unnamed>: SELECT * FROM x WHERE y = $1 LIMIT $2 DETAIL: parameters: $1 = 'long string', $2 = ‘1' SlowQueries log_min_duration_statement = 1000 ms
- 52. @LukasFittl
- 53. @LukasFittl
- 54. @LukasFittl auto_explain logs the query plan for specific slow queries
- 55. @LukasFittl
- 56. @LukasFittl
- 57. @LukasFittl log_lock_waits = on LOG: process 20679 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 after 1000.115 ms LOG: process 20678 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 after 1000.126 ms LOG: process 15533 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.129 ms LOG: process 20663 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.100 ms LOG: process 15537 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.130 ms LOG: process 15536 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.222 ms LOG: process 20734 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.130 ms LOG: process 15538 still waiting for ExclusiveLock on tuple (566,1) of relation 16421 1000.136 ms LOG: process 15758 still waiting for ShareLock on transaction 250175899 after 1000.073 ms LockWaits
- 58. @LukasFittl Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Three Log Events Worth Knowing Query Annotations
- 59. @LukasFittl
- 60. @LukasFittl
- 61. @LukasFittl application: pganalyze controller: graphql action: graphql line: /app/graphql/organization_type.rb … graphql: getOrganizationDetails.logVolume24h request_id: 44bd562e-0f53-453f-831f-498e61ab6db5
- 63. @LukasFittl 3Take-Aways 1. Collect Historic Metrics 2. Focus on Drill-Down To Query Level 3. Annotate Your Queries With Their Origin
- 64. @LukasFittl Monitor YourPostgres: pganalyze.com Scale Your Postgres: citusdata.com Thanks!