Best Practices for Becoming an Exceptional Postgres DBA
25 likes18,521 views
The document outlines best practices for PostgreSQL Database Administrators (DBAs), focusing on monitoring, maintenance, capacity planning, and configuration tuning. It emphasizes the importance of architectural health checks, performance tuning, and effective backup strategies to ensure database reliability and efficiency. Additionally, it shares insights from customer case studies and offers recommendations for addressing common issues such as performance problems and replication failures.
Best Practices for Becoming an Exceptional Postgres DBA
- 1. DBA Best Practices – Becoming an Exceptional Postgres DBA Marc Linster | August 27, 2014 © 2014 EDB All rights reserved. 1
- 2. Agenda • EnterpriseDB Overview • Best Practice Expertise • The Postgres DBA Role − Monitoring − Maintenance − Capacity Planning − Configuration Tuning − Deployment Planning • Professional Development • Summary and Resources • Q&A © 2014 EnterpriseDB Corporation. All rights reserved. 2 We will take one quick-poll during the webinar to gather group data on best practices of interest
- 3. Brief EDB Overview © 2014 EnterpriseDB Corporation. All rights reserved. 3
- 4. POSTGRES innovation Services & training © 2014 EnterpriseDB Corporation. All rights reserved. 4 ENTERPRISE reliability 24/7 support Enterprise-class features & tools Indemnification Product road-map Control Thousands of developers Fast development cycles Low cost No vendor lock-in Advanced features Enabling commercial adoption of Postgres
- 5. Postgres Plus Advanced Server Postgres Plus Management Performance © 2014 EnterpriseDB Corporation. All rights reserved. 5 Cloud Database High Availability REMOTE DBA 24x7 SUPPORT PROFESSIONAL SERVICES TRAINING EDB Serves All Your Postgres Needs PostgreSQL Security
- 6. EDB Customers EDB currently has over 2,500 total customers including 50 of the Fortune 500 and 98 of the Forbes Global 2000 © 2014 EnterpriseDB Corporation. All rights reserved. 6
- 7. Best Practice Expertise © 2014 EnterpriseDB Corporation. All rights reserved. 7
- 8. Hundreds of Postgres Instances Supported • Remote DBA Services • Architectural Health Checks • Postgres Support Tickets • Postgres Training © 2014 EnterpriseDB Corporation. All rights reserved. 8 When • ~70% support calls are at deployment, post-deployment • Remote DBA and AHC are post-deployment What • How to perform DBA tasks • Which product tools or features to use
- 9. When and Where do Users Need Help? 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% POC Dev Deployment Maintenance © 2014 EnterpriseDB Corporation. All rights reserved. 9 Tuning Product Awareness How to Enhancements Corruption Bug
- 10. Where Do Users Need Help # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Connectors 0% 2% 1% 0% 3% Bug 0% 0% 0% 0% 1% How to 0% 1% 0% 0% 1% Product Awareness 0% 1% 0% 0% 1% Database 10% 9% 17% 37% 73% Bug 1% 2% 1% 2% 6% Corruption 0% 0% 0% 2% 2% Enhancements 0% 0% 0% 0% 1% How to 4% 3% 6% 10% 23% Product Awareness 4% 3% 8% 15% 31% Tuning 0% 0% 1% 7% 9% Replication 1% 4% 4% 2% 12% Bug 0% 1% 0% 0% 2% How to 0% 0% 3% 1% 4% Product Awareness 1% 2% 1% 0% 4% Tuning 0% 0% 0% 1% 1% Utilities 1% 3% 6% 3% 12% Bug 0% 1% 1% 0% 2% Enhancements 0% 0% 0% 0% 0% How to 1% 1% 3% 1% 6% Product Awareness 0% 1% 3% 1% 5% Grand Total 12% 17% 28% 42% 100% © 2014 EnterpriseDB Corporation. All rights reserved. 10
- 11. Select Top Support and RDBA Questions (not in rank order) Deployment/Database/ Product Awareness & How To • How to set up connection pooling? • How to set up hot standby and streaming replica? • How to add table spaces? • How to partition? • How to upgrade from major version? • What should I monitor? • How do I reduce downtime? © 2014 EnterpriseDB Corporation. All rights reserved. 11 Maintenance/Database/ Product Awareness & How To • How to find bloat in tables & indexes? • How to enable auditing selectively without performance impact? • What are the effects of changes to the values of postgresql.conf file parameters wal_writer_delay, bgwriter_delay? • What is the correct configuration of hba_conf?
- 12. Building Blocks of the DBA Role Performance Tuning Monitoring Maintenance © 2014 EnterpriseDB Corporation. All rights reserved. 12 Capacity Management Security Deployment Training & Professional Development
- 13. The Postgres DBA Role © 2014 EnterpriseDB Corporation. All rights reserved. 13
- 14. Taking on the Postgres DBA Role • Postgres is very reliable and stable • You can translate your existing DBA skills to Postgres • Key areas for Postgres DBAs − Bloat management − Vacuum setup − OS configuration − Storage sub-system configuration − Backup – logical and physical − HA Setup − pg_hba.conf and other security settings − Monitoring © 2014 EDB All rights reserved. 14
- 15. Postgres DBA Role – An Approach • Begin with assessing the health of the DB - Architectural Health Check (AHC) − Performance − Maintenance Tasks – index management, bloat reduction − Availability – backup strategies, recovery, replication and failover − Scalability and support for business and data growth − Upgrade planning • Establish monitoring • Focus on list of items uncovered in health check © 2014 EDB All rights reserved. 15
- 16. Architectural Health Check Process • Technical architecture – OS, application, database • Indexes and bloat • Configuration tuning – OS, database − Memory − Write-ahead log − Background worker − Query − Logging − Checkpoint − Vacuum − Query performance − Security © 2014 EDB All rights reserved. 16 • DR plan − Backup strategy – logical and physical − HA solutions – warm standby/log shipping, hot standby, streaming and cascading replication • Performance tuning − Reindexing − Partitioning • Resource management − Connections − CPU • Hardware and OS configuration − File system options
- 17. Monitoring and Maintenance © 2014 EnterpriseDB Corporation. All rights reserved. 17
- 18. Maintenance Highlights from 20+ AHCs Bloated Tables © 2014 EnterpriseDB Corporation. All rights reserved. 18 Unused Indexes 39% Bloated Indexes 30% 13% Missing Indexes 9% Missing PK 9% Bloat management (43%) is a key production problem specific to Postgres
- 19. General Monitoring Recommendations Description Frequency Alerting Criteria / Parameters monitored Load average, disk space and checking for .ready files Every 10 minutes (i) When load average is above threshold (ii) When disk space is below threshold (iii) When .ready(pg_xlog/archive_status/*.ready) files count is beyond threshold Postgres instance is running and connection count Every minute (i) When postgres instance is not running (ii) When number of connection is reached to value of max_connections (iii) When number of connections crosses threshold Long running and waiting queries in database Every 10 minutes (i) When any query is running for longer than threshold (ii) When any query is in state of "idle in transaction" for more than threshold (iii) When any query is waiting for more than threshold Bloats for tables and indexes Once a day (i) When database tables and/or indexes individually bloated more than critical threshold (ii) When transaction ID wrap around is approaching Is connection pooler running Every 10 minutes If pgbouncer or pgPool process is not running Is streaming replication in sync with primary Every 5 minutes (i) When lag detected on streaming cluster exceeds threshold © 2014 EnterpriseDB Corporation. All rights reserved. 19
- 20. Monitoring Tools (if you can’t use PEM) • Open source plug-in check_postgres.pl! − http://bucardo.org/ • PostgreSQL Statistics Collector - pg_stat_xxx views − pg_stat_activity - details of open connections and running transactions − pg_stat_database - details of databases − pg_stat_user_* - details of tables, indexes and functions − pg_locks - list of current locks being held • Database object size functions − pg_*_size - disk space used by a tablespace, database, relation or total_relation (includes indexes and toasted data) © 2014 EnterpriseDB Corporation. All rights reserved. 20
- 21. Postgres Enterprise Manager • Postgres Enterprise Manager (PEM) Tool − Monitor real-time alerts − Historical reports − Identify poorly running SQL statements • Top Features − Capacity Manager − Audit Manager − Postgres Expert − SQL Profiler − Alert Manager − Log Manager − Web Client − Graphical Dashboards for I/O, Memory & user activity © 2011 EnterpriseDB . All rights reserved. © 2014 EnterpriseDB Corporation. All rights reserved. 21
- 22. PEM Features • Global dashboards with up-to-date status (up/down/ performance) of all your servers • Ability to create performance thresholds for each key metric e.g. memory, storage, etc. • Any threshold violation results in an alert being sent to a centralized dashboard • All key performance-related statistics are automatically collected and retained • Forecast resource usage in the future • Identify and tune poorly running SQL statements • SNMP integration • Audit Logging for Postgres Plus Advanced Server instances © 2011 EnterpriseDB . All rights reserved. © 2014 EnterpriseDB Corporation. All rights reserved. 22
- 23. Capacity Management © 2014 EnterpriseDB Corporation. All rights reserved. 23
- 24. Capacity Management • Use cases: − Consumption projection: − At current consumption rates when will I drop below 100MB of disk? − At the consumption rate of the past 3 days when will I drop below 1 GB of disk space? − Point in Time Activity − Users were complaining of performance problems from 3 to 4 PM today. What was going on in the system, the database, how many users were connected? − We get a load spike every day at 11. What was happing at the time of the load spike? − Root cause for slower Query Performance − Queries that access the employ table seem to be taking longer and longer. View the growth of the number of dead tuples in the table, see what kind of scans have been executing against the table (index or sequential) − Activity monitoring − Which database on the server is the most active? What kind of growth is it seeing? © 2014 EnterpriseDB Corporation. All rights reserved. 24
- 25. Sample PEM Report – Disk Space Available © 2011 EnterpriseDB . All rights reserved. 25 © 2014 EnterpriseDB Corporation. All rights reserved. 25 Extrapolated Actual Data Data
- 26. Security © 2014 EnterpriseDB Corporation. All rights reserved. 26
- 27. Security Auditing • Conceptual Layers − Authentication − Authorization − Auditing − Data Security − SQL Injection Attacks © 2014 EnterpriseDB Corporation. All rights reserved. 27 • Approach − Secure physical access − Network access limitation − Host access limitation − Database access limitation − Data access limitation
- 28. Security Best PracSetcuiricty Beest sPra ctficeos forr PoPstgreoSQsL atndg Porstgeres SPlusQ AdvaLnce d Server http://info.enterprisedb.com/rs/enterprisedb/images/ Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf © 2014 EnterpriseDB Corporation. All rights reserved. 28 Table of Contents Executive Summary ................................................................3 Introduction ….........................................................................4 Postgres Security Features within the Above Framework ......6 Authentication .........................................................6 Authorization ….......................................................7 Accounting/Auditing ................................................8 Data Security ….......................................................9 SQL Injection Attacks ............................................10 Postgres Plus Advanced Server Security Features ..............11 Further Reading and Useful Links ........................................12 About EnterpriseDB ..............................................................12 Disclaimer
- 29. Quick Poll Database Priorities © 2014 EnterpriseDB Corporation. All rights reserved. 29
- 30. Audience Quick Poll #1 • Question: What is your current database priority? • Answer choices: a. Resolving a current performance issue b. Architecture – migration, capacity planning, upgrade c. Preparing for production deployment d. Monitoring for ongoing database health e. Establishing routine maintenance practices © 2014 EnterpriseDB Corporation. All rights reserved. 30
- 31. Parameter Tuning for Performance © 2014 EnterpriseDB Corporation. All rights reserved. 31
- 32. Lessons From AHC and RDBA Performance Improvement Recommendations in: • Optimizing memory usage − Adjusting shared_buffers − work_mem * max_connections • vm.dirty_bytes and vm.dirty_background_bytes • Understanding the storage − effective_io_concurrency • Identifying problem queries - pgbadger and SQL Profiler The following analysis is based on a set of 20 AHCs executed in 2013 © 2014 EnterpriseDB Corporation. All rights reserved. 32
- 33. 100%# 90%# 80%# 70%# 60%# 50%# 40%# 30%# 20%# 10%# 0%# Top Database Parameter Adjustments – Ordered by Frequency effec0ve_cache_size# Top$Database$Parameter$Adjustments$3$Ordered$by$Frequency$ wal_buffers# maintenance_work_mem# shared_buffers# bgwriter_lru_maxpages# work_mem# random_page_cost# checkpoint_segments# cpu_tuple_cost# checkpoint_comple0on_target# bgwriter_lru_mul0plier# Red bars: experts expected top parameters, including vacuum_cost_limit, log_line_prefix, log_min_duration_statement, log_checkpoints checkpoint_warning# autovacuum_vacuum_cost_delay# autovacuum_vacuum_threshold# log_min_dura0on_statement# autovacuum_vacuum_scale_factor# checkpoint_0meout# © 2014 EnterpriseDB Corporation. All rights reserved. 33 log_line_prefix# log_lock_waits# autovacuum_nap0me# log_autovacuum_min_dura0on#
- 34. Kernel Parameters 16 14 12 10 8 6 4 2 0 © 2014 EnterpriseDB Corporation. All rights reserved. 34
- 35. Customer Case Study Use Case: Supply Chain Logistics IT Provider SaaS Warehouse Management System Issue: • Performance issues - speed Root Causes • Slow, memory-hungry queries, improper indexes Results © 2014 EnterpriseDB Corporation. All rights reserved. 35 Change Areas • Indexing improvements • Query tuning or rewrites
- 36. Customer Case Study Use Case: Global MLS Platform – Reporting System Issue: • Intermittent timeouts Root Cause • Parameter configuration • Enabled − vm.dirty_bytes − vm.dirty_background_bytes − vm.zone_reclaim_mode • Disabled − vm.overcommit_memory − vm.swappiness − redhat_transparent_hugepage © 2014 EnterpriseDB Corporation. All rights reserved. 36 Change Areas • Kernel parameters • Database memory parameters • Increased − work_mem − maintenance_work_mem • Reduced − shared_buffers • wal_buffers 32MB • wal_level “archive”
- 37. Deployment Planning © 2014 EnterpriseDB Corporation. All rights reserved. 37
- 38. Backup and Recovery Strategies • Why do you need backup and recovery? Backup and Recovery Strategies protect you in case of: − Catastrophic Device Failure − Site Failure − Maintenance − Operator Error − Compliance − Data Corruption • Multiple components: − Logical backup provides granularity in objects (tables, table spaces, databases) − Physical backups provide granularity in time for PITR − Cold/off-line backups − Hot/on-line backup, with WAL archiving © 2014 EnterpriseDB Corporation. All rights reserved. 38
- 39. Backup and Standby Snapshot Backup Approach © 2014 EnterpriseDB Corporation. All rights reserved. 39 Standby Approach hot 47% None 18% cold 12% warm 23% Physical Only 36% Logical Only 29% FS 7% Physical and Logical 14% None 14% 43% of customers could not execute PITR (prior to AHC) – some had NO backup in place 30% of customers would have been slow or unable to recover from failure
- 40. Backup and Recovery Best Practices • Backup Strategy Framework − Logical backup after structural changes or major updates of reference/meta data − Physical backup (daily…) − WAL file archiving − Combine Backup/Recovery and Streaming Replication − Uses replica to offload hot backup and logical backup • Consider − Allowable PITR timeframe − Data retention policy − Test, test, test − Periodic backup validation • Advanced solutions, such as RHCS, Veritas Clustering , EFM and EDB BART augment the basic capabilities © 2014 EnterpriseDB Corporation. All rights reserved. 40
- 41. Customer Case Study Use Case: on-line advertising service with hosted chat Stage: post-production Primary Issue: • Outage due to replication failure - insufficient log info to determine true root cause Potential Pain Points • Concerns have reached limits of scale and meeting anticipated growth demands © 2014 EnterpriseDB Corporation. All rights reserved. 41 Recommendations • Backup and recovery strategies − Implement continuous archive recovery in addition to streaming replication − Revised backup strategy with PITR − Tune auto-vacuum settings and manually vacuum large tables • Indexes – create concurrent indexes, REINDEX to reduce bloat • Parameter tuning – various for kernel, memory settings and WALs
- 42. File System and Storage Subsystem • Types of Storage Systems − Direct Attached Storage (DAS) − Storage Area Network (SAN) w. Fiber Channel − Storage Area Network (SAN) w. iSCSI − Network File System (NFS) • DAS: Fast and low latency • SAN: Fast, expensive (w. Fiber Channel), scalable, can include redundancy and smart file system operations • NFS: Not an optimal solution for PostgreSQL data files © 2014 EnterpriseDB Corporation. All rights reserved. 42
- 43. Summary and Useful Resources © 2014 EnterpriseDB Corporation. All rights reserved. 43
- 44. Summary • Postgres DBAs need to focus on − Config file settings, especially understanding memory configs − Storage subsystem configuration − OS Configuration − Bloat & vacuum − Backup - logical and physical − HA setup − Monitoring - see the Freight Train before it hits you! • Tools, tools, tools – Get the right tools • Most problems occur when moving to production − Get ready early − Get trained before MTP − New Postgres DBA -- consider RDBA to help you get started © 2014 EnterpriseDB Corporation. All rights reserved. 44
- 45. What You Can Do • Training & Certification − Classroom Comprehensive Admin - Bangalore – Sept 15-19 − Comprehensive Admin Live Online – NA- with Bruce Momjian – PostgreSQL Guru and community leader – Oct 6-10 − Live Online EMEA – Adv. PostgreSQL Admin – with 9.4 Highlights – Oct 13-15 • Get the team (developer, DBA, Infrastructure) ready to thrive with Postgres • EDB Knowledge Base • Blogs − Robert Haas – EDB Chief Architect Database Server (http://rhaas.blogspot.com/) − Vibhor Kumar – EDB Database Consultant (http://vibhorkumar.wordpress.com/) • User groups and Meetups – almost all major cities − http://www.postgresql.org/community/user-groups/ − http://postgresql.meetup.com/ • PostgresOpen 2014 - Sept 17-19 Chicago • PostgreSQL Conference Europe 2014 - Oct 21-24 Madrid © 2014 EnterpriseDB Corporation. All rights reserved. 45
- 46. Architectural Health Check • Review of database, OS, env. settings • Guidance on tuning, backup, high availability strategies Remote DBA Services • 24 x 7 monitoring • Supplemental staff • Architectural health check • Access to Postgres expertise © 2014 EnterpriseDB Corporation. All rights reserved. 46 Kick Start • Training • Flex consulting • Developer subscription Training and Certification • Flexible formats: on-demand, live virtual, on-site • Continuing education and Postgres certification How We Can Help Whitepapers and Recorded Webinars http://www.enterprisedb.com/resources-community
- 47. Special Offers for Attendees For all attendees we are providing a special offer on related DBA services including training. Please complete an inquiry form on-line and mention this webinar, or speak to your account manager − EDB’s regional sales offices found on enterprisedb.com: − US +1 781-357-3390 or 1-877-377-4352 − EMEA +31 (0) 70 36 11 774 − JAPAN +81-50-5532-7038 − KOREA +82-2-6007-2500 − PAKISTAN +92-51-8358874 − INDIA +91-20-30589500/01 − Send email to [email protected] − http://www.enterprisedb.com/general-inquiry-form © 2014 EnterpriseDB Corporation. All rights reserved. 47
- 48. Q&A © 2014 EnterpriseDB Corporation. All rights reserved. 48