How to Replicate PostgreSQL Database
0 likes223 views
This document provides an overview of replicating a PostgreSQL database. It discusses setting up a primary server for reads and writes and standby servers that are kept in sync with the primary to serve as backups. The primary server writes data to its write-ahead log (WAL) files, which are streamed in real-time to the standby servers via WAL shipping. This allows the standby servers to keep an identical copy of the database. The document also covers configuration of both the primary and standby servers for replication as well as tools for testing the replication setup.
![Replication Testing
postgres=# select * from pg_stat_replication ;
-[ RECORD 1 ]----+---------------------------------
pid | 1114
usesysid | 16384
usename | repuser
application_name | walreceiver
client_addr | 127.0.0.1
client_hostname |
client_port | 52444
backend_start | 08-MAR-20 19:54:05.535695 -04:00
state | streaming
..
sync_priority | 0
sync_state | async
ps ef | grep postgres
[Master] wal sender process check
[Standby] wal receiver & startup process check
SELECT * FROM pg_stat_replication; Query execution on the Primary/Standby nodes](https://image.slidesharecdn.com/howtoreplicatepostgresqldatabase-200721142614/85/How-to-Replicate-PostgreSQL-Database-10-320.jpg)
How to Replicate PostgreSQL Database
- 1. Service Platform Architect Brandon Kang [email protected] https://tech.brandonkang.net July 2020 How to Replicate PostgreSQL Database / / 0 2 / . 2 : / 2 . / A @ /: / / /
- 2. Replication Overall Primary Server • A server for both READ and WRITE Standby Server = Slave server, Backup server, Replica, etc. • A server where the data is kept in sync with the master continuously • A warm standby server can’t be connected until it is promoted to primary • A hot standby server accepts connections and serves read-only queries • Data is written to the master server and propagated to the slave servers • When primary server has an issue, a standby server will take over and continue to take ‘WRITE” process
- 3. WAL Shipping Replication WAL(Write-Ahead Logging) • A log file where all the modifications to the database • WAL is used for recovery after a database crash, ensuring data integrity • WAL is used in database systems to achieve atomicity and durability WAL based Replication Streaming WAL Records • Write-ahead log records are used to keep the data in sync • WAL record chunks are streamed by database servers to synchronize • Standby server connects to the master to receive the WAL chunks • WAL records are streamed as they are generated • The streaming of WAL records need not wait for the WAL file to be filled
- 4. Streaming Replication PRIMARY Server Standby Server WAL Sender WAL Receiver WAL Record READ ONLYREAD/WRITE pg_wal directory WAL File pg_wal directory WAL File Synchronize - WAL Record1 - WAL Record2 - WAL Record3 ……
- 5. Replication Configuration Primary Node pg_hba.conf /PG_HOME/DATA/pg_hba.conf # Allow replication connections from localhost, # by a user with the replication privilege. # TYPE DATABASE USER ADDRESS METHOD host replication repl_user IPaddress(CIDR) md5 Restart PostgreSQL service on the primary node Internal IP ranges Firewall Open for this.
- 6. Replication Configuration Primary Node /PG_HOME/DATA/postgres.conf # Possible values are replica|minimal|logical wal_level = hot_standby archive_mode = on #Syncronize with Primany node using wal archive archive_command = ‘cp %p /PG_TBS/backup/archive/%f’ #psql_superuser archive backup # required for pg_rewind capability when standby goes out of sync with master wal_log_hints = on # sets the maximum number of concurrent connections from the standby servers. max_wal_senders = 2 # The below parameter is used to tell the master to keep the minimum number of # segments of WAL logs so that they are not deleted before standby consumes them. # each segment is 16MB wal_keep_segments = 16
- 7. Replication Configuration Secondary Node /PG_HOME/DATA/postgres.conf hot_standby = on Secondary Node /PG_HOME/DATA/recovery.conf standby_mode = ‘on’ restore_command = ‘cp %p /PG_TBS/backup/archive/%f “%p”’ Primary_conninfo = ‘host=… ports=… user=postgres password= application_name=slave1’ # Explaining a few options used for pg_basebackup utility # -X option is used to include the required transaction log files (WAL files) in the # backup. When you specify stream, this will open a second connection to the server # and start streaming the transaction log at the same time as running the backup. # -c is the checkpoint option. Setting it to fast will force the checkpoint to be # created soon. # -W forces pg_basebackup to prompt for a password before connecting # to a database. pg_basebackup -D <data_directory> -h <master_host> -X stream -c fast -U repl_user -W
- 8. Replication Configuration Secondary Node pg_basebackup utility # Explaining a few options used for pg_basebackup utility # -X option is used to include the required transaction log files (WAL files) in the # backup. When you specify stream, this will open a second connection to the server # and start streaming the transaction log at the same time as running the backup. # -c is the checkpoint option. Setting it to fast will force the checkpoint to be # created soon. # -W forces pg_basebackup to prompt for a password before connecting # to a database. pg_basebackup -D <data_directory> -h <master_host> -X stream -c fast -U repl_user -W Example) $pg_basebackup h localhost p 5170 -u postgres D /PG_TBS/cluster -checkpoint=fast --progress
- 9. Replication Configuration Secondary Node Replication configuration file # Specifies whether to start the server as a standby. In streaming replication, # this parameter must be set to on. standby_mode = ‘on’ # Specifies a connection string which is used for the standby server to connect # with the primary/master. primary_conninfo = ‘host=<master_host> port=<postgres_port> user=<replication_user> password=<password> application_name=”host_name”’ # Specifies recovering to a particular timeline. The default is to recover along the # same timeline that was current when the base backup was taken. # Setting this to latest recovers to the latest timeline found # in the archive, which is useful in a standby server. recovery_target_timeline = ‘latest’ Start PostgreSQL service on the Standby node
- 10. Replication Testing postgres=# select * from pg_stat_replication ; -[ RECORD 1 ]----+--------------------------------- pid | 1114 usesysid | 16384 usename | repuser application_name | walreceiver client_addr | 127.0.0.1 client_hostname | client_port | 52444 backend_start | 08-MAR-20 19:54:05.535695 -04:00 state | streaming .. sync_priority | 0 sync_state | async ps ef | grep postgres [Master] wal sender process check [Standby] wal receiver & startup process check SELECT * FROM pg_stat_replication; Query execution on the Primary/Standby nodes
- 11. Replication Testing pg_is_in_recovery(): A function to check if standby server is still in recovery mode or not postgres=# select pg_is_in_recovery(); pg_is_in_recovery ------------------- t (1 row) pg_last_xact_replay_timestamp(): A function shows time stamp of last replica transaction postgres=# select pg_last_xact_replay_timestamp(); pg_last_xact_replay_timestamp ---------------------------------- 08-MAR-20 20:54:27.635591 -04:00 (1 row) SELECT CASE WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location() THEN 0 ELSE EXTRACT (EPOCH FROM now() - pg_last_xact_replay_timestamp()) END AS log_delay; Calculating logs in seconds
- 12. Scaling PostgreSQL PRIMARY Server Standby Server WAL Sender WAL Receiver WAL Record WRITE Synchronize HAProxy / NGINX / F5 READ WRITE READ pgBouncer primary_db standby_db Application
- 13. - Thank You. - Service Platform Architect Brandon Kang [email protected] https://tech.brandonkang.net