Out of the Box Replication in Postgres 9.4(PgCon)

/
Out of the Box Replication
In
Postgres 9.4
Denish Patel
Lead Database Architect

Who am I ?
• Database Architect with OmniTI for last 8+ years
• Expertise in PostgreSQL , Oracle, MySQL, NoSQL
• Contact : denish@omniti.com
• Twitter: @DenishPatel
• Blog: http://www.pateldenish.com
• Providing Solutions for business problems to deliver
• Scalability
• Reliability
• High Availability
• Security
We are hiring!!
Apply @ l42.org/lg
1

Agenda
! What is WAL?
! Postgres Replication History
! How you setup replication now?
! What’s missing ?
! Why replication slots?
! Demo
I’m NOT going to discuss …
! SLONY or other 3rd party replication tools
! 3rd party replication management tools
2

WAL - Write Ahead Log
• Roll-forward recovery aka REDO
• flushed to disk to guarantee commit durability
• sequential writes
• lower cost than flushing page cache
• Allows us to do cool things
• Crash recovery
• Binary backups
• Point-In Time Recovery
• Replication
3

WAL Internals (Basic)
• Automatically enabled; no action required
• Make sure to have enough space on server
• Stored under pg_xlog directory
• Normally, 16MB in size
" --with-wal-segsize config option at build
• Each segment is divided into pages (8 kB page)
" --with-wal-blocksize config option at build
• Segment name starts with..
000000010000000000000000
4

What is logged?
select * from pg_settings where name='wal_level';
name | wal_level
setting | hot_standby
unit |
category | Write-Ahead Log / Settings
short_desc | Set the level of information written to the WAL.
extra_desc |
context | postmaster
vartype | enum
source | configuration file
min_val |
max_val |
enumvals | { minimal, archive, hot_standby, logical }
boot_val | minimal
reset_val | hot_standby
sourcefile | /var/lib/pgsql/9.4/data/postgresql.auto.conf
sourceline | 4
5

What’s NOT logged?
Almost everything is replicated, but...
! unlogged tables (As name suggests)
! temporary tables
! hash indexes? (generally don’t use?)
6

Postgres Replication History
Postgres 7.0: WAL
Postgres 8.0: PITR (Point-In-Time-Recovery)
Postgres 8.2: pg_standby
Postgres 9.0: Hot_standby, Streaming replication
Postgres 9.1: pg_basebackup, Synchronous replication
Postgres 9.2: Cascading Replication
Postgres 9.3: Standby can switch timeline to follow new
master
Postgres 9.4: Replication Slots , Logical decoding
7

Basic Steps to Setting up Replication
initdb
8

Postgresql.conf
! max_wal_senders=10
! wal_level=hot_standby
! hot_standby=on (on standby)
9

pg_hba.conf
#TYPE DATABASE USER ADDRESS METHOD
host replication replication 10.0.0.1/32 md5
10

Restart database …
pg_ctl restart
11

Create replication user
CREATE ROLE replication WITH LOGIN
REPLICATION;
password replication
12

Take Backup
! Take file system level backups
! What tools you are using for backups?
13

Recovery.conf
primary_conninfo = 'host=primaryhost user=replication
password=replication'
standby_mode = on
14

Startup standby db
pg_ctl start
15

Not done yet!
Have you configured archiving?
16

Don’t forget about this setting?
wal_keep_segments
17

Setup archiving . . .
! postgresql.conf
archive_mode = on
archive_command = 'cp %p /some/where/%f'
18

Setup restore command
! recovery.conf
restore_command = 'cp /some/where/%f %p'
19

Archiving Options
! local or remote copy
! Scp
! Rsync
! NFS
! pg_archivecleanup
20

What if you have more standby machines?
archive_command = 'rsync %p standby1::pg/%f && rsync
%p standby2::pg/%f'
archive_command = 'echo standby1 standby2 ...
| xargs -d" " -I{} -n1 -P0 -r rsync %p {}::pg/%f'
21

Replication management tools?
! OmniPITR
! WAL-E
! Repmgr
! Pgbarman
! Skytools
! A lot of Custom scripts
22

What about fsync?
! fsync capabilities
! cp: no
! dd: GNU coreutils
! SSH: OpenSSH 6.5 sftp-server (Jan 2014)
! rsync: patch or wrapper
! NFS: Supported
23

Postgres 9.4 ; Enter Replication Slots
24

Postgresql.conf
max_replication_slots = 8
25

Create Slot
SELECT * FROM
pg_create_physical_replication_slot('name');
26

Recovery.conf
primary_slot_name = ‘name'
28

Replication slots benefits
! Keep necessary WAL files
! Each standby can have different WAL apply status
! Single access control setup
! fsync on receiving side
29

pg_basebackup
pg_basebackup
-h primaryhost
-U replication
-D $PGDATA
-X stream
–P –v -R
30

How about archiving?
! Meet pg_receivexlog
! (Available since Postgers 9.1!)
pg_receivexlog
-D archivedir
--slot archivingslot
-h primaryhost -U replication
! -- synchronous option in 9.5
31

Postgres 9.4 – Out of the Box Replication
Are you ready to setup replication without any external
tools?
1. pg_basebackup
2. streaming with Replication Slots
3. pg_receivexlog
32

Tutorial – Let’s bring up VM!
! Google Drive: https://drive.google.com/open?
id=0BxnXwkT5PRBdeVByQVYySkhIemc
! Login: pgtraining/pgcon
! pgtraining user has sudo access
! You can access terminal on the desktop
! Internet should be working within VM
! Copy/paste should work between VM and Host
! Take snapshot along the process so you can rollback
easily
33

Installing Postgres
! Remove Postgres 8.4 version
sudo yum erase postgresql.*
! Setup yum repo for your desired version
sudo yum install http://yum.postgresql.org/9.4/redhat/rhel-6-
x86_64/pgdg-redhat94-9.4-1.noarch.rpm
! Install PostgreSQL 9.4 & Contrib modules
sudo yum install postgresql94-server postgresql94-contrib
! Create postgres cluster & initial automatic startup
sudo service postgresql-9.4 initdb
sudo chkconfig postgresql-9.4 on
sudo service postgresql-9.4 start
34

Create Role and Database
! Become postgres system user
sudo su - postgres
! Log into database using psql (? to see all available commands)
! Create a role & database for yourself
CREATE ROLE pgtraining WITH LOGIN SUPERUSER;
CREATE DATABASE pgtraining;
! You can login as pgtraining user (psql –U pgtraining –d pgtraining)
! Create replication role for later
CREATE ROLE replication WITH LOGIN REPLICATION;
! Set password (”replication”)
password replication
35

Configure pg_hba.conf
! http://www.postgresql.org/docs/9.4/static/auth-pg-
hba-conf.html
! Find location of hba_file
postgres=# show hba_file;
! Add following entry for replication user
! Try to avoid trust authentication
! [pgtraining@localhost ~]$ sudo vi /var/lib/pgsql/9.4/
data/pg_hba.conf
host replication replication 127.0.0.1/32 md5
36

Prepare Primary DB server
alter system set wal_level = hot_standby;
alter system set archive_mode=on;
alter system set max_replication_slots=8;
alter system set archive_timeout = 60;
alter system set max_wal_senders = 8;
alter system set wal_keep_segments=100;
alter system set logging_collector=on;
37

Restart Primary DB server
! Restart database
sudo service postgresql-9.4 restart
! Verify settings
psql=# show max_wal_senders;
38

Create replication slot
SELECT * FROM pg_create_physical_replication_slot('standby1');
39

Let’s take backup
sudo su - postgres
pg_basebackup -h 127.0.0.1 -U replication -D /var/
lib/pgsql/9.4/slave -R -Xs -P -v
40

Prepare standby db server …
cd /var/lib/pgsql/9.4/slave
! Edit Standby postgresql.conf file
port = 5433
hot_standby = on
! Edit Standby recovery.conf file
standby_mode = 'on'
primary_conninfo = 'user=replication password=replication
host=127.0.0.1 port=5432'
primary_slot_name='standby1'
trigger_file = '/var/lib/pgsql/9.4/slave/finish.recovery'
recovery_target_timeline='latest'
41

Configure Standby with init
! Copy existing init file
sudo cp /etc/init.d/postgresql-9.4 /etc/init.d/postgresql-9.4-5433
! Edit config file to change (as root):
PGDATA=/var/lib/pgsql/9.4/slave
PGLOG=/var/lib/pgsql/9.4/pgstartup-5433.log
PGUPLOG=/var/lib/pgsql/$PGMAJORVERSION/
pgupgrade-5433.log
! Register service, start up slave
sudo chkconfig postgresql-9.4-5433 on
sudo service postgresql-9.4-5433 start
42

What about archiving?
! Create slot for archiving:
SELECT * FROM
pg_create_physical_replication_slot('archiver1');
! Create archive directory
mkdir /var/lib/pgsql/9.4/archive
! Start archiving process in background
/usr/pgsql-9.4/bin/pg_receivexlog -h 127.0.0.1 -p 5432 -U
replication –S 'archiver1' -n -v -D /var/lib/pgsql/9.4/
archive
! Put under init.d for continuous run
! Switch xlog : primary_db_sever# select pg_switch_xlog();
45

Monitoring
! Monitor Disk space
! Monitor slave lag
select pg_xlog_location_diff(sent_location, write_location) AS
byte_lag
from pg_stat_replication
where application_name='pg_receivexlog';
! Monitor WAL archive process
select pg_xlog_location_diff(sent_location, write_location) AS
byte_lag
from pg_stat_replication
where application_name='walreceiver';
46

Monitoring
• Number of pg_xlogs on master
• Monitor pg_recievexlog process
• Make sure archive location has new files
• pg_basebackup log files for successful backup… look
for “pg_basebackup: base backup completed”
47

Thanks to….
! OmniTI for travel sponsorships
! Pgcon conference committee
! Peter Eisentraut
! You!!
48

Questions?
denish@omniti.com
Twitter: DenishPatel
49

Out of the Box Replication in Postgres 9.4(PgCon)

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Out of the Box Replication in Postgres 9.4(PgCon) (20)

More from Denish Patel (11)

Out of the Box Replication in Postgres 9.4(PgCon)