Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
1 like1,404 views
The document discusses the automation and management of PostgreSQL using ClusterControl, a tool that simplifies deployment, monitoring, and performance management. Key features include backup management, high availability setups, and automatic failover, all aimed at alleviating the labor-intensive processes typically required for database management. Additionally, it highlights the complexities of manual DBA operations and emphasizes the need for reliable automation tools in PostgreSQL management.
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
- 1. January 2019 Automation & Management of PostgreSQL with ClusterControl Sebastian Insausti Presenter [email protected]
- 2. Copyright 2018 Severalnines AB I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: [email protected] during or after the webinar. Your host & some logistics
- 5. Copyright 2018 Severalnines AB Free to download Initial 30 days Enterprise trial Converts into free Community Edition Enterprise / paid versions available
- 6. Copyright 2018 Severalnines AB Automation & Management Deployment (Free Community) ● Deploy a Cluster in Minutes ○ On-Prem ○ Cloud (AWS/Azure/Google) - paid Monitoring (Free Community) ● Systems View with 1 sec Resolution ● DB / OS stats & Performance Advisors ● Configurable Dashboards ● Query Analyzer ● Agentless & agent-based Management (Paid Features) ● Backup Management ● Upgrades & Patching ● Security & Compliance ● Operational Reports ● Automatic Recovery & Repair ● Performance Management ● Automatic Performance Advisors
- 7. Copyright 2018 Severalnines AB Supported Databases
- 8. Copyright 2018 Severalnines AB Our Customers
- 9. Poll - multiple choice Copyright 2018 Severalnines AB Which method do you use to perform your DBA tasks? ● Manual actions ● Custom scripts ● External tools and utilities ● ClusterControl ● Others (please specify in the chat box)
- 10. January 2019 Automation & Management of PostgreSQL with ClusterControl Sebastian Insausti Presenter [email protected]
- 11. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Managing PostgreSQL ● Requirements for managing production databases “The Old Way” ● Labour intensive, manual operations ● Scripts, tools and utilities ● What happens when the DBA leaves? Agenda Automation and Management with ClusterControl ● Deployment ● Monitoring & Alerting ● Performance Management ● Backup management ● High Availability setups ● Single VIP endpoint with Load Balancer ● Automatic Failover Live Demo
- 12. Copyright 2018 Severalnines AB Copyright 2018 Severalnines AB Requirements for managing production databases
- 13. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Deployment ● Install ○ The first step to begin the game ● Secure ○ Keep your data safe and secure ● Configure ○ Environment requirements
- 14. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Monitoring & Alerting ● Define metrics ○ Database ○ Operating System ● Configure monitoring ○ Custom Monitoring ○ System Monitoring ● Configure alerts ○ Email alerts ○ Tool Integrations
- 15. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Important but complex task ● PostgreSQL is configured with compatibility and stability by default ● Depends on the hardware and on the system itself ● Performance queries Performance Management
- 16. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Backups Methods: ○ Logical backup: Plain text files with SQL statements ○ Physical backup: A copy of the datafiles ● Backup tasks: ○ Schedule ○ Rotation / Retention ○ Secure Backup ○ Verify Backup ○ Point-in-time recovery backup ○ Upload to external datacenter, e.g., cloud storage ○ Reporting Backup Management
- 17. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Define topology ○ Master-Slave ○ Master-Master ● Define replication type ○ Synchronous: There is no loss of data even if our master node is lost. The commits in the master must wait for a confirmation from the slave. ○ Asynchronous: There is a possibility of data loss in case we lose our master node. If the replica is not updated at the time of the incident, the information that has not been copied may be lost. ○ Logical Replication: Performed via SQL statements. ○ Physical Replication: Performed via internal data structure. ● Failover ● Single endpoint for application High Availability setups
- 18. Poll - multiple choice Copyright 2018 Severalnines AB According to you, what are your most challenging tasks in production? ● A well tuned and secure configuration ● Monitoring uptime & Alerting ● Managing performance / queries ● Backup management ● Failover & High Availability
- 19. Copyright 2018 Severalnines AB Copyright 2018 Severalnines AB "The Old Way"
- 20. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Deployment ● Installation: ○ Repositories or packages ○ Some packages dependencies could be required ● Secure installation: ○ Configuration files (pg_hba.conf or postgresql.conf) ○ Authentication ● Basic configuration: ○ For example: Listen address or wal_level ○ Require DBA or Sysadmin skills
- 21. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB What to monitor ● Database ○ Queries ○ Sessions ○ Locks ○ IO ○ Replication ○ Backups ● System ○ CPU ○ Memory ○ Load Average ○ Network Monitoring & Alerting
- 22. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Performance Management ● Time consuming task ● There is not a standard configuration for this task ● Check charts ○ e.g: max_connections ● Database commands ○ EXPLAIN: Check the query plan ○ VACUUM: For several maintenance tasks ● Parameters ○ shared_buffers: Amount of memory that the database uses for shared memory ○ work_mem: Amount of memory used by internal operations like ORDER BY, DISTINCT, JOIN, among others ● Require deep DBA knowledge
- 23. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Backup Management ● Typically done using custom scripts ● Some just copy the datafiles (!) ● Schedule using cron ● Manual rotation ● No backup verification or PITR backup
- 24. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Master-Slave Topology ● Asynchronous replication ● Manual setup of slave ○ Install a new server ○ Copy data from master ○ Configure replication High Availability Setups
- 25. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Automatic Failover ● We need to know when we need the failover ● PostgreSQL itself does not include an automatic failover mechanism ● After failover, the applications need to know about the new master
- 26. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Load Balancer: ○ Balancing traffic ○ Redirect traffic to available nodes ○ Specify ports for different roles ○ It's an external component that requires installation, configuration and monitoring ● Virtual IP: ○ Keepalived is a linux service that allows to configure a virtual IP within an active/passive group of servers ○ Another external component that requires installation, configuration and monitoring Single VIP endpoint with Load Balancer
- 27. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ● Where is the documentation? ● How do all these scripts hang together? ● It is often hard to know how the database is managed ● There is nobody to take over what you did ● It can be a mess What happens when the DBA leaves?
- 28. Copyright 2018 Severalnines AB Copyright 2018 Severalnines AB ClusterControl - PostgreSQL automation and management
- 29. Copyright 2018 Severalnines AB ClusterControl
- 30. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Deployment
- 31. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Monitoring & Alerting
- 32. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Monitoring & Alerting
- 33. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Performance Management
- 34. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Backup Management
- 35. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB High Availability Setups
- 36. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Single VIP endpoint with Load Balancer
- 37. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB Automatic Failover
- 38. Copyright 2018 Severalnines ABCopyright 2018 Severalnines AB ClusterControl CLI $ s9s cluster --list --long ID STATE TYPE OWNER GROUP NAME COMMENT 1 FAILURE galera system admins Galera1 Cluster failed to start. 2 STARTED postgresql_single system admins PostgreSQL1 All nodes are operational. 5 FAILURE replication system admins MySQL1 Cluster failed to start. Total: 3 $ s9s node --list --cluster-id=2 --long STAT VERSION CID CLUSTER HOST PORT COMMENT coC- 1.7.1.2965 2 PostgreSQL1 192.168.100.110 9500 Up and running. ?o-- 2.3.2 2 PostgreSQL1 192.168.100.110 9090 Process 'prometheus' is running. ho-- 1.5.18 2 PostgreSQL1 192.168.100.114 9600 Process 'haproxy' is running. koM- 1.3 2 PostgreSQL1 192.168.100.114 112 Process 'keepalived' is running. ho-- 1.5.18 2 PostgreSQL1 192.168.100.115 9600 Process 'haproxy' is running. ko-- 1.3 2 PostgreSQL1 192.168.100.115 112 Process 'keepalived' is running. poM- 10.5 2 PostgreSQL1 192.168.100.123 5432 Up and running. poS- 10.5 2 PostgreSQL1 192.168.100.124 5432 Up and running. poS- 10.5 2 PostgreSQL1 192.168.100.125 5432 Up and running. Total: 9
- 39. Copyright 2018 Severalnines AB Copyright 2018 Severalnines AB Live Demo
- 40. ● Deployment ● Add replication slave ● Add load balancer ● Deploy keepalived ● Monitoring ● Performance ● Backup ● Topology view ● CLI Cluster Control PostgreSQL Management Demo Copyright 2018 Severalnines AB
- 41. Poll - multiple choice (choose one of below) Copyright 2018 Severalnines AB Where do you need the most improvements in your environment? ● Monitoring & Alerting ● Performance Management ● Backups ● High Availability & Automatic Failover ● Other (please specify in the chat box)
- 42. Copyright 2018 Severalnines AB Copyright 2018 Severalnines AB Q & A
- 43. Links & Resources ● ClusterControl for PostgreSQL ● Download / install ClusterControl ● ClusterControl Community Edition Page ● Contact us: [email protected]