EnterpriseDB's Best Practices for Postgres DBAs
3 likes1,999 views
This document provides an agenda and overview for a presentation on best practices for PostgreSQL database administrators (DBAs). The presentation covers EnterpriseDB's expertise in PostgreSQL, the key responsibilities of a PostgreSQL DBA including monitoring, maintenance, capacity planning and configuration tuning. It also discusses deployment planning, professional development resources, and takes questions. Examples from architectural health checks and remote DBA services illustrate common issues found like index bloat and lack of backups. The document recommends performance monitoring and security tools and techniques for PostgreSQL.
EnterpriseDB's Best Practices for Postgres DBAs
- 1. © 2014 EDB All rights reserved. 1 DBA Best Practices – Becoming an Exceptional Postgres DBA
- 2. © 2014 EnterpriseDB Corporation. All rights reserved. 2 • EnterpriseDB Overview • Best Practice Expertise • The Postgres DBA Role − Monitoring − Maintenance − Capacity Planning − Configuration Tuning − Deployment Planning • Professional Development • Summary and Resources • Q&A Agenda
- 3. © 2014 EnterpriseDB Corporation. All rights reserved. 3 Brief EDB Overview
- 4. © 2014 EnterpriseDB Corporation. All rights reserved. 4 POSTGRES innovation ENTERPRISE reliability 24/7 support Services & training Enterprise-class features & tools Indemnification Product road-map Control Thousands of developers Fast development cycles Low cost No vendor lock-in Advanced features Enabling commercial adoption of Postgres
- 5. © 2014 EnterpriseDB Corporation. All rights reserved. 5 Postgres Plus Advanced Server Postgres Plus Cloud Database High Availability PerformanceManagement REMOTE DBA 24x7 SUPPORT PROFESSIONAL SERVICES TRAINING EDB Serves All Your Postgres Needs PostgreSQL Security
- 6. © 2014 EnterpriseDB Corporation. All rights reserved. 6 Why EDB? More than 3,000 Enterprises and Governments
- 7. © 2014 EnterpriseDB Corporation. All rights reserved. 7 EnterpriseDB is a Market Leader The Gartner report, Magic Quadrant for Operational Database Management Systems, by Donald Feinberg, Merv Adrian and Nick Heudecker, was published October 16, 2014.
- 8. © 2014 EnterpriseDB Corporation. All rights reserved. 8 EDB is an Open Source Community Leader Amit Kapila Ashesh Vashi Bruce Momjian Dave Page Devrim Gunduz Jan Wieck Kevin Grittner Korry Douglas Muhammad Usama Robert M Haas Thom Brown
- 9. © 2014 EnterpriseDB Corporation. All rights reserved. 9 Best Practice Expertise
- 10. © 2014 EnterpriseDB Corporation. All rights reserved. 10 • Remote DBA Services • Architectural Health Checks • Postgres Support Tickets • Postgres Training Hundreds of Postgres Instances Supported When • ~70% support calls are at deployment post-deployment • Remote DBA and AHC are post-deployment What ? • How to perform DBA tasks • Which product tools or features to use
- 11. © 2014 EnterpriseDB Corporation. All rights reserved. 11 When and Where do Users Need Help? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% POC Dev Deployment Maintenance Tuning Product Awareness How to Enhancements Corruption Bug
- 12. © 2014 EnterpriseDB Corporation. All rights reserved. 12 Where Do Users Need Help # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Connectors 0% 2% 1% 0% 3% Bug 0% 0% 0% 0% 1% How to 0% 1% 0% 0% 1% Product Awareness 0% 1% 0% 0% 1% Database 10% 9% 17% 37% 73% Bug 1% 2% 1% 2% 6% Corruption 0% 0% 0% 2% 2% Enhancements 0% 0% 0% 0% 1% How to 4% 3% 6% 10% 23% Product Awareness 4% 3% 8% 15% 31% Tuning 0% 0% 1% 7% 9% Replication 1% 4% 4% 2% 12% Bug 0% 1% 0% 0% 2% How to 0% 0% 3% 1% 4% Product Awareness 1% 2% 1% 0% 4% Tuning 0% 0% 0% 1% 1% Utilities 1% 3% 6% 3% 12% Bug 0% 1% 1% 0% 2% Enhancements 0% 0% 0% 0% 0% How to 1% 1% 3% 1% 6% Product Awareness 0% 1% 3% 1% 5% Grand Total 12% 17% 28% 42% 100%
- 13. © 2014 EnterpriseDB Corporation. All rights reserved. 13 Building Blocks of the DBA Role Monitoring Maintenance Performance Tuning Capacity Management Security Deployment Training & Professional Development
- 14. © 2014 EnterpriseDB Corporation. All rights reserved. 14 The Postgres DBA Role
- 15. © 2014 EDB All rights reserved. 15 • Postgres is very reliable and stable • You can translate your existing DBA skills to Postgres • Key areas for Postgres DBAs − Bloat management − Vacuum setup − OS configuration − Storage sub-system configuration − Backup – logical and physical − HA Setup − pg_hba.conf and other security settings − Monitoring Taking on the Postgres DBA Role
- 16. © 2014 EDB All rights reserved. 16 • Begin with assessing the health of the DB - Architectural Health Check (AHC) − Performance − Maintenance Tasks – index management, bloat reduction − Availability – Backup strategies, recovery, replication and failover − Scalability and support for business and data growth − Upgrade Planning • Establish monitoring • Focus on list of items uncovered in health check Postgres DBA Role – An Approach
- 17. © 2014 EDB All rights reserved. 17 • Technical architecture – OS, application, database • Indexes and bloat • Configuration tuning – OS, database − Memory − Write-ahead log − Background worker − Query − Logging − Checkpoint − Vacuum − Query performance − Security Architectural Health Check Process • DR plan − Backup strategy – logical and physical − HA solutions – warm standby/log shipping, hot standby, streaming and cascading replication • Performance tuning − Reindexing − Partitioning • Resource management − Connections − CPU • Hardware and OS configuration − File system options
- 18. © 2014 EnterpriseDB Corporation. All rights reserved. 18 Monitoring and Maintenance
- 19. © 2014 EnterpriseDB Corporation. All rights reserved. 19 Maintenance Highlights from 20+ AHCs Unused Indexes 39% Bloated Indexes 30% Bloated Tables 13% Missing Indexes 9% Missing PK 9% Bloat management (43%) is a key production problem specific to Postgres
- 20. © 2014 EnterpriseDB Corporation. All rights reserved. 20 General Monitoring Recommendations Description Frequency Alerting Criteria / Parameters monitored Load average, disk space and checking for .ready files. Every 10 minutes (i) When load average is above threshold (ii) When disk space is below threshold (iii) When .ready(pg_xlog/archive_status/*.ready) files count is beyond threshold Postgres instance is running and connection count Every minute (i) When postgres instance is not running (ii) When number of connection is reached to value of max_connections (iii) When number of connections crosses threshold Long running and waiting queries in database Every 10 minutes (i) When any query is running for longer than threshold (ii) When any query is in state of "idle in transaction" for more than threshold (iii) When any query is waiting for more than threshold Bloats for tables and indexes. Once a day (i) When database tables and/or indexes individually bloated more than critical threshold (ii) When transaction ID wrap around is approaching Is connection pooler running Every 10 minutes If pgbouncer or pgPool process is not running Is streaming replication in sync with primary. Every 5 minutes (i) When lag detected on streaming cluster exceeds threshold
- 21. © 2014 EnterpriseDB Corporation. All rights reserved. 21 • Open source plug-in check_postgres.pl − http://bucardo.org/ • PostgreSQL Statistics Collector - pg_stat_xxx views − pg_stat_activity - details of open connections and running transactions − pg_stat_database - details of databases − pg_stat_user_* - details of tables, indexes and functions − pg_locks - list of current locks being held • Database object size functions − pg_*_size - disk space used by a tablespace, database, relation or total_relation (includes indexes and toasted data) Monitoring Tools (if you can’t use PEM)
- 22. © 2014 EnterpriseDB Corporation. All rights reserved. 22 © 2011 EnterpriseDB . All rights reserved. Postgres Enterprise Manager • Postgres Enterprise Manager (PEM) Tool − Monitor real-time alerts − Historical reports − Identify poorly running SQL statements • Top Features − Capacity Manager − Audit Manager − Postgres Expert − SQL Profiler − Alert Manager − Log Manager − Web Client − Graphical Dashboards for I/O, Memory & user activity
- 23. © 2014 EnterpriseDB Corporation. All rights reserved. 23 © 2011 EnterpriseDB . All rights reserved. PEM Features • Global dashboards with up-to-date status (up/down/ performance) of all your servers • Ability to create performance thresholds for each key metric e.g. memory, storage, etc. • Any threshold violation results in an alert being sent to a centralized dashboard • All key performance-related statistics are automatically collected and retained • Forecast resource usage in the future • Identify and tune poorly running SQL statements • SNMP integration • Audit Logging for Postgres Plus Advance server instances
- 24. © 2014 EnterpriseDB Corporation. All rights reserved. 24 Capacity Management
- 25. © 2014 EnterpriseDB Corporation. All rights reserved. 25 • Use cases: − Consumption projection: − At current consumption rates when will I drop below 100MB of disk? − At the consumption rate of the past 3 days when will I drop below 1 GB of disk space? − Point in Time Activity − Users were complaining of performance problems from 3 to 4 PM today. What was going on in the system, the database, how many users were connected? − We get a load spike every day at 11. What was happing at the time of the load spike? − Root cause for slower Query Performance − Queries that access the employ table seem to be taking longer and longer. View the growth of the number of dead tuples in the table, see what kind of scans have been executing against the table (index or sequential) − Activity monitoring − Which database on the server is the most active? What kind of growth is it seeing? Capacity Management
- 26. © 2014 EnterpriseDB Corporation. All rights reserved. 26 Sample PEM Report – Disk Space Available 26© 2011 EnterpriseDB . All rights reserved. Extrapolated DataActual Data
- 27. © 2014 EnterpriseDB Corporation. All rights reserved. 27 Security
- 28. © 2014 EnterpriseDB Corporation. All rights reserved. 28 • Conceptual Layers • Authentication • Authorization • Auditing • Data Security • SQL Injection Attacks Security Auditing • Approach • Secure physical access • Network access limitation • Host access limitation • Database access limitation • Data access limitation
- 29. © 2014 EnterpriseDB Corporation. All rights reserved. 29 Security Best Practices for PostgreSQLSecurity Best Practices for PostgreSQL and Postgres Plus Advanced Server Executive Summary ................................................................3 Introduction ….........................................................................4 Postgres Security Features within the Above Framework ......6 Authentication .........................................................6 Authorization ….......................................................7 Accounting/Auditing ................................................8 Data Security ….......................................................9 SQL Injection Attacks ............................................10 Postgres Plus Advanced Server Security Features ..............11 Further Reading and Useful Links ........................................12 About EnterpriseDB ..............................................................12 Table of Contents http://info.enterprisedb.com/rs/enterprisedb/images/ Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf
- 30. © 2014 EnterpriseDB Corporation. All rights reserved. 30 Parameter Tuning for Performance
- 31. © 2014 EnterpriseDB Corporation. All rights reserved. 31 Performance Improvement Recommendations in: • Optimizing memory usage − Adjusting shared_buffers − work_mem * max_connections • vm.dirty_bytes and vm.dirty_background_bytes • Understanding the storage − effective_io_concurrency • Identifying problem queries - pgbadger and SQL Profiler The following analysis is based on a set of 20 AHCs executed in 2013 Lessons From AHC and RDBA
- 32. © 2014 EnterpriseDB Corporation. All rights reserved. 32 0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%# effec0ve_cache_size# shared_buffers# bgw riter_lru_m axpages# w al_buffers# m aintenance_w ork_m em # bgw riter_lru_m ul0plier# w ork_m em # random _page_cost# checkpoint_segm ents# cpu_tuple_cost# checkpoint_com ple0on_target# checkpoint_0m eout# log_line_prefix# log_lock_w aits# log_m in_dura0on_statem ent# autovacuum _vacuum _scale_factor# checkpoint_w arning# autovacuum _vacuum _cost_delay# autovacuum _vacuum _threshold# autovacuum _nap0m e# log_autovacuum _m in_dura0on# Top$Database$Parameter$Adjustments$3$Ordered$by$Frequency$ Red bars: experts expected top parameters, including vacuum_cost_limit, log_line_prefix, log_min_duration_statement, log_checkpoints Top Database Parameter Adjustments – Ordered by Frequency
- 33. © 2014 EnterpriseDB Corporation. All rights reserved. 33 Kernel Parameters 0 2 4 6 8 10 12 14 16
- 34. © 2014 EnterpriseDB Corporation. All rights reserved. 34 Deployment Planning
- 35. © 2014 EnterpriseDB Corporation. All rights reserved. 35 • Why do you need backup and recovery? Backup and Recovery Strategies protect you in case of: − Catastrophic Device Failure − Site Failure − Maintenance − Operator Error − Compliance − Data Corruption • Multiple components: − Logical backup provides granularity in objects (tables, table spaces, databases) − Physical backups provide granularity in time for PITR − Cold/off-line backups − Hot/on-line backup, with WAL archiving Backup and Recovery Strategies
- 36. © 2014 EnterpriseDB Corporation. All rights reserved. 36 Backup and Standby hot 47% warm 23% None 18% cold 12% Standby Approach Physical Only 36% Logical Only 29% Physical and Logical 14% FS Snapshot 7% None 14% Backup Approach 43% of customers could not execute PITR (prior to AHC) – some had NO backup in place 30% of customers would have been slow or unable to recover from failure
- 37. © 2014 EnterpriseDB Corporation. All rights reserved. 37 • Backup and Recovery Strategy − Many deployments lack good backup & recovery (43%) • OS Considerations − (dirty_background_ratio & dirty_ratio) are important (70% & 45%) • Storage Systems − key deployment consideration - DAS/SAN/NFS • High Availability and Standby Approach − 30% of customers would have been slow or unable to recover from failure • Plan for Ongoing Maintenance − Bloat and vacuum setup is a problem (43%) Deployment Planning Observations from AHCs
- 38. © 2014 EnterpriseDB Corporation. All rights reserved. 38 • Backup Strategy Framework − Logical backup after structural changes or major updates of reference/meta data − Physical backup (daily…) − WAL file archiving − Combine Backup/Recovery and Streaming Replication − Uses replica to offload hot backup and logical backup • Consider − Allowable PITR timeframe − Data retention policy − Test, test, test − Periodic backup validation • Advanced solutions, such as RHCS, Veritas Clustering, EFM and EDB BART augment the basic capabilities Backup and Recovery Best Practices
- 39. © 2014 EnterpriseDB Corporation. All rights reserved. 39 • Types of Storage Systems − Direct Attached Storage (DAS) − Storage Area Network (SAN) w. Fiber Channel − Storage Area Network (SAN) w. iSCSI − Network File System (NFS) • DAS: Fast and low latency • SAN: Fast, expensive (w. Fiber Channel), scalable, can include redundancy and smart file system operations • NFS: Not an optimal solution for PostgreSQL data files File System and Storage Subsystem
- 40. © 2014 EnterpriseDB Corporation. All rights reserved. 40 • OLTP Infrastructure − ACID Compliant − High Performance − Scalable − Reliable • High Availability − Protect against hardware failure − Protect against software failure • Disaster Recovery − Protect against site failure − Protect against operator error • Data Integration − Integration with other Line of Business systems − Heterogeneous integration with Microsoft, Oracle, etc • Monitoring and Management − Capacity Planning − Event management and alerting • Security − Authentication and Authorization − Encryption and data protection Key Components of a Enterprise Data Management Architecture
- 41. © 2014 EnterpriseDB Corporation. All rights reserved. 41 PEM Client Oracle/SQL Server/PG Replication DB MasterWitness Offsite Replica Virtual IP DB Client Backup & Recovery PEM ServerHigh Availability Disaster Recovery HA Replica Read/HA Replica Robust, Resilient, Scalable Postgres Enterprise Architecture Monitoring DataIntegration Simple, development focused setup
- 42. © 2014 EnterpriseDB Corporation. All rights reserved. 42 Summary and Useful Resources
- 43. © 2014 EnterpriseDB Corporation. All rights reserved. 43 • Postgres DBAs need to focus on − Config file settings, especially understanding memory configs − Storage subsystem configuration − OS Configuration − Bloat & vacuum − Backup - logical and physical − HA setup − Monitoring - see the Freight Train before it hits you! • Tools, tools, tools – Get the right tools • Most problems occur when moving to production − Get ready early − Get trained before MTP − New Postgres DBA -- consider RDBA to help you get started Summary
- 44. © 2014 EnterpriseDB Corporation. All rights reserved. 44 • Training & Certification − Online classes and certifications http://www.enterprisedb.com/products-services-training/training • Get the team (developer, DBA, Infrastructure) ready to thrive with Postgres • EDB Knowledge Base • Blogs − EDB’s Blog – http://www.enterprisedb.com/postgres-plus-edb-blog − Robert Haas – EDB Chief Architect Database Server (http://rhaas.blogspot.com/) − Vibhor Kumar – EDB Database Consultant (http://vibhorkumar.wordpress.com/) • User groups and Meetups – almost all major cities − http://www.postgresql.org/community/user-groups/ − http://postgresql.meetup.com/ What You Can Do