Elegant database tuning
- 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Postgres Conf US - April, 2018
- 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Tuning • Application • Data Lifecycle • Database configuration
- 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tuning Methodology 1. Run unit tests 2. Track database statistics 3. Baseline Config 4. Run Load 5. Track database statistics 6. Update config 7. GOTO 4
- 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What should we track? • Activity • Database Utilization • Table Utilization • Index Utilization • Locking
- 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How do we track it? • PG Catalog • Schema in every postgres db (pg_catalog) • Appended to search_path • “Data Dictionary”
- 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog SELECT pn.nspname, pc.relkind, count(1) FROM pg_namespace pn, pg_class pc WHERE pn.oid = pc.relnamespace AND pn.nspname = 'pg_catalog' GROUP BY pn.nspname, pc.relkind ORDER BY 1 DESC; nspname | relkind | count ------------+---------+------- pg_catalog | i | 115 pg_catalog | r | 62 pg_catalog | v | 59 i = indexes r = tables v = views
- 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog functions select count(1) FROM ( select pn.nspname, pp.proname FROM pg_namespace pn, pg_proc pp WHERE pn.oid = pp.pronamespace AND pn.nspname = 'pg_catalog' ) foo; count ----- 2882
- 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog functions SELECT pn.nspname, pp.proname FROM pg_namespace pn, pg_proc pp WHERE pn.oid = pp.pronamespace AND pn.nspname = 'pg_catalog’; nspname | proname ------------+-------------- pg_catalog | boolin pg_catalog | boolout pg_catalog | byteain pg_catalog | byteaout pg_catalog | charin pg_catalog | charout pg_catalog | namein pg_catalog | nameout ...
- 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog • Structural • Performance • Informational
- 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog – translation! • Relation = table, index, sequence, etc… • Attribute = column • Namespace = schema
- 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog - oid • Object Identifier datatype • 4 byte unsigned int • Hidden column on catalog tables • Primary / Foreign keys throughout pg_catalog schema
- 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog - oid postgres=# select * from pg_namespace; nspname | nspowner | nspacl --------------------+----------+---------------------------- pg_toast | 10 | pg_temp_1 | 10 | pg_toast_temp_1 | 10 | pg_catalog | 10 | {meads=UC/meads,=U/meads} public | 10 | {meads=UC/meads,=UC/meads} information_schema | 10 | {meads=UC/meads,=U/meads} (6 rows)
- 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog - oid postgres=# select oid, * from pg_namespace; oid | nspname | nspowner | nspacl -------+--------------------+----------+---------------------------- 99 | pg_toast | 10 | 11816 | pg_temp_1 | 10 | 11817 | pg_toast_temp_1 | 10 | 11 | pg_catalog | 10 | {meads=UC/meads,=U/meads} 2200 | public | 10 | {meads=UC/meads,=UC/meads} 12349 | information_schema | 10 | {meads=UC/meads,=U/meads} (6 rows)
- 14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. pg_catalog – List all tables in a schema SELECT pn.nspname, pc.relkind, count(1) FROM pg_namespace pn, pg_class pc WHERE pn.oid = pc.relnamespace AND pn.nspname = 'pg_catalog' GROUP BY pn.nspname, pc.relkind ORDER BY 1 DESC;
- 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What’s this have to do with tuning??
- 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tuning Methodology 1. Run unit tests 2. Track database statistics 3. Baseline Config 4. Run Load 5. Track database statistics 6. Update config 7. GOTO 4
- 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What should we track? • Activity • Database Utilization • Table Utilization • Index Utilization • Locking
- 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Where is the data? • pg_stat_activity • pg_stat_database • pg_stat_all_tables / pg_stat_user_tables • pg_stat_all_indexes / pg_stat_user_indexes • pg_locks
- 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Activity • pg_stat_activity • Current activity in the database • Snapshot of that instant • Starting point for tuning • State • Idle • Active • Idle in transaction select datname, usename, client_addr, wait_event, now() - state_change as runtime, state, query from pg_stat_activity ;
- 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Activity - Tuning • Application • Connections • Concurrency • Data Lifecycle • Query runtimes • WAL
- 21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Activity - Tuning • Server configuration • max_connections • shared_buffers • work_mem
- 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Statistics • pg_stat_database • Utilization stats for each DB in an instance • Global view of all DBs select datname, numbackends, xact_commit, tup_returned, tup_fetched, tup_inserted, tup_updated, tup_deleted, pg_database_size(datname), pg_size_pretty(pg_database_size(datname)) FROM pg_stat_database;
- 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Statistics • pg_stat_all_tables / pg_stat_user_tables • Cumulative view of table-level stats • Basis for workload profile • pg_stat_all_indexes / pg_stat_user_indexes • Cumulative view of index-level stats • Index utilization stats select schemaname, relname, n_tup_ins, n_tup_upd, n_tup_del, idx_scan, seq_scan FROM pg_stat_user_tables;
- 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Where are my hot-spots? • What’s the most modified (INS / UPD / DEL)? • What’s the most read ? • What’s the largest?
- 25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Stats -Tuning • Application • Partitioning • SQL • Stored Procedures
- 26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table Stats - Tuning • Data Lifecycle • Datamodel • Indexing • Workflow (Truncate vs Delete) • Database configuration • Vacuum • Shared_buffers
- 27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Locking • pg_locks • Status of all locks in a given database • Look for conflicting locks WHERE granted = ‘f’
- 28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Locking - Tuning • Application • Workflow • Data Lifecycle • Data Model • Normalization • ETL • Maintenance
- 29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tracking vs Storing • Cloud Watch • Performance Insights • POWA • PGWatch2 • Snapshot scripts • Grafana • ELK Max CPU
- 30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Learn more.. aws.amazon.com/rds aws.amazon.com/rds/aurora
- 31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Scott Mead [email protected]