Advanced mysql replication techniques

Advanced MySQL
Replication Techniques

Giuseppe Maxia, QA Director, Continuent, Inc

Facts. And Demos. Possibly fun
Monday, April 11, 2011 1

about me - Giuseppe Maxia
• a.k.a. The Data Charmer
• QA Director at Continuent, Inc
• Long time hacking with MySQL features
• Formerly, community manager,db consultant, designer,
coder.
• A passion for QA and open source
• Blogger
• http://datacharmer.blogspot.com


Agenda
• Replication basics
• Replication blues
• Master switch and failover
• Slave lagging
• Gotchas
• Replication power techniques and features
• row-based, semi-synch, delayed
• Circular, bi-directional
• leveraging replication
• Tools and tips
• Replication outside the box
•
Monday, April 11, 2011
seamless failover, parallel, multi-master
3

DA
AG
EN Replication basics

• Principles
• How to set up replication
• Monitoring replication


DA
AG
EN Principles


database server

a simple web
application
scheme

r/w requests

web server

clients
6

database server

scaling web
requests

r/w requests

web servers

load balancer

clients
7

read/write
master a web
application
read/only
slaves
scheme with
replication
load balancer
R/W

R/O

web servers

load balancer

clients
8

client master
transaction

binary log

reads
slave
IO thread

relay log replication
concepts
SQL thread reads
9

DA
AG
EN replication setup


1 SHUT DOWN THE DATABASE SERVER

Master

11

2 MAKE A BACKUP COPY

Master

12

3 ENABLE THE MASTER

Master

Configuration file
[mysqld]
log-bin=mysql-bin
server-id=1

13

4 RESTART THE MASTER

Master

14

5 CREATE REPLICATION USER

Master

SQL command
GRANT REPLICATION SLAVE ON *.*
to 'slave_user@'10.10.100.%'
IDENTIFIED BY 'slave_pass';

15

6 INSTALL MySQL on the slave

Slave 1

Make sure that:
• You're using the same version of MySQL
• You have the same directory structure
• The server is not started yet

16

7 COPY THE MASTER DATA to the slave

Slave 1

17

8 ENABLE THE SLAVE

Slave 1

Configuration file
[mysqld]
server-id=2
relay-log=mysql-relay
read-only
# optional:
log-bin=mysql-bin 18

9 START THE SLAVE SERVER

Slave 1

19

10 INITIALIZE THE SLAVE

Slave 1

SQL command
CHANGE MASTER TO
MASTER_HOST=master_IP,
MASTER_PORT=3306,
MASTER_USER=slave_user,
MASTER_PASSWORD='slave_pwd';
20

11 START THE SLAVE SERVICE

Slave 1

SQL command
START SLAVE;

21

12 CHECK THE SLAVE

Slave 1

SQL command
SHOW SLAVE STATUS G
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
22

Troubleshooting

• SHOW SLAVE STATUS says
SLAVE_IO_RUNNING=No
• Make sure that the slave host can connect
to the master

• Make sure that master and slave have
different Server-id

• Check the error log of both master and slave


Testing the slave

• Create a table in the master.
• Make sure that the slave has replicated
the table.
• Insert data in the master
• read that data in the slave


DA
AG
EN Monitoring


Sample get slave
status
monitoring
slave

Running? No

Yes
master
Get master
binlog and
position

Same or later
No
Yes
binlog/position?

check table
contents
alert 26

monitoring replication
master> SHOW MASTER STATUS

slave> SHOW SLAVE STATUS

FULL SCRIPTS:

http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans-
mysql.html

http://forge.mysql.com/tools/tool.php?id=6


show master status
File: mysql-bin.000002
Position: 78045744
Binlog_Do_DB:
Binlog_Ignore_DB:


show slave status
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 27371
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 78045744
Relay_Log_File: mysql_sandbox27372-relay-bin.000055
Relay_Log_Pos: 78045889
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes


show slave status
...
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
...


show slave status
...
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 78045744
Relay_Log_Space: 78046100
...
Seconds_Behind_Master: 0
...
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:


DA
AG
EN Replication blues

• Master switch and failover
• Slave lagging
• Gotchas


DA
AG
EN Master switch and failover


Replacing the Master
master crashe
s

Let all slaves
catch up with
execution STOP
replication in
all slaves

FIND the most make it the
up to date slave master

FIND which run missing
connect all
transactions are transactions
slaves to the
missing from to other
new master
other slaves slaves

Stop
34

Planned master switch

• Stop accepting writes
• Wait until all slaves have caught up
• Stop replication in all slaves
• Promote a slave to master
• Point all slaves to the new master


Changing a failed master

• Pre-requisite:
• log_bin and log_slave_updates must be
enabled in all the slaves
• If not, there is more manual labor


Changing a failed master (1)

• Wait until all slaves have caught up
• Identify the most advanced slave
• Make that slave the new master
• ... so far, so good


• For each remaining slave:
• Find the missing statements
• Find the LAST statement replicated by the slave

• Find the same statement in the new master
binlog (*)

• get the position of the NEXT statement

(*) if log_slave_updates was not enabled, you
need to convert the relay log statements to SQL
and do the next step manually



• For each remaining slave:
• Apply the missing statements
• CHANGE MASTER TO
master_host=”new_master_hostname”,
master_port=new_master_port,
master_log_file=”mysql-bin.xxxxxx”,
master_log_pos=YYYY


Reasons for complexity

• No global transaction ID


What is a global transaction ID

• A unique identifier of a transaction
• Unique for the whole cluster, not for each node
• Generated by the ultimate source (the master)
• Does not change when the transaction goes
through an intermediate master


Why should you care

• Failure recovery
• MySQL DOES NOT have it


Deﬁning the problem
Slave 3

Master Replication Slave 2

Slave 1
Binary log

Transactions
Relay log Binary log
--------
-------- Transactions Transactions
-------- -------- ========
-------- -------- ========
-------- ========
-------- ========


The master crashes
Slave 3

Master Replication Slave 2

Slave 1
Binary log

Transactions
Relay log Binary log
--------
-------- Transactions Transactions
-------- -------- ========
-------- -------- ========
-------- ========
-------- ========


Where do the slaves
stand?
Slave 1 Slave 2 Slave 3

Binary log Binary log Binary log

Transactions Transactions Transactions
======== ======== ========
======== ======== ========
======== ======== ========
======== ======== ========


Slave 3 becomes
master
New master

Slave 1 Slave 2
Binary log

Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ========
======== ========


Need to synch the
missing transactions
New master

Slave 1 Slave 2
Binary log

Transactions
========
========
======== ========
======== ========
======== ========
======== ========


Transaction position in
the new master binlog
New master

Slave 1 Slave 2
Binary log

Transactions
========
========
======== ========
======== ========
======== ? ======== ?
======== ========


An existing solution

• Google has made a patch to implement global
transaction IDs
• HOWEVER
• It only works with 5.0
• It doesn't work with row based replication


Hack
ahead!


CREATE TABLE Global_Trans_ID (

number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY

) ENGINE = MyISAM;

CREATE TABLE Last_Exec_Trans (

server_id INT UNSIGNED,

trans_id INT UNSIGNED

) ENGINE = InnoDB;


CREATE PROCEDURE commit_trans ()

BEGIN

DECLARE trans_id, server_id INT UNSIGNED;

SET SQL_LOG_BIN = 0;

INSERT INTO global_trans_id() values ();

SELECT LAST_INSERT_ID() INTO trans_id,

@@server_id INTO server_id;

DELETE FROM global_trans_id where number < trans_Id;

SET SQL_LOG_BIN = 1;

INSERT INTO last_exec_trans(server_id, trans_id)

VALUES (server_id, trans_id);

COMMIT;

END

Hacking
the hack


Simplifying the scheme

• No MyISAM table and LAST_INSERT_ID()
• Use UUID_SHORT() instead
• Replace the InnoDB table with a BlackHole one


Advantages

• 10 times faster
• No additional tables and storage required


Disadvantages

• The master SERVER ID must be < 256


Another solution

• We’ll see that in the second part of the
talk


DA
AG
EN Slave lagging


Slave lagging

• Common causes:
• Excess of traffic

• Slave restart

• Long DDL operations


Slave lagging
transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction

BINARY LOG

ns act act tio n
tra csio s ac ctio
ac ion ion n
an n s an
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa

n
n
tio
t
tra

tra

REPLICATION

MySQL replication
is SINGLE THREAD
transaction
transaction

MySQL
DBMS


Slave lagging remedies

• row-based replication (helps in some
cases)
• Expensive query split
• Double apply (dangerous)
• Slave query prefetch (requires external
app)
• Parallel replication (requires external
app)


statement-based replication
master> set global binlog_format=‘statement’;
master> insert into test.t1 select count(*) from
information_schema.columns;
master> show global status like 'opened_tables';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Opened_tables | 41 |
+---------------+-------+
slave> show global status like 'opened_tables';
+---------------+-------+
+---------------+-------+
+---------------+-------+

row-based replication
master> set global binlog_format=‘row’;
master> insert into test.t1 select count(*) from
information_schema.columns;
master> show global status like 'opened_tables';
+---------------+-------+
+---------------+-------+
+---------------+-------+
slave> show global status like 'opened_tables';
+---------------+-------+
+---------------+-------+
+---------------+-------+

Expensive query split
# instead of this:
insert into t1 select benchmark(10000000, sha('abc'));
Query OK, 1 row affected (4.19 sec)
Records: 1 Duplicates: 0 Warnings: 0

# you may do this
set @result=(select benchmark(10000000, sha('abc')));
Query OK, 0 rows affected (4.19 sec)

insert into t1 values (@result);


Expensive query split
# at 471
#110410 8:21:21 server id 1 end_log_pos 518
User_var
SET @`result`:=0/*!*/;
# at 518
#110410 8:21:21 server id 1 end_log_pos 612 Query
thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1302448881/*!*/;
insert into t1 values (@result)


Slave query prefetch

• What is it?
• a technique that fetches queries from the
relay logs, and runs them as SELECT
statements, to warm up the indexes

• How do you do it?
• mk-slave-prefetch (http://www.maatkit.org)

• Slave readahead http://sourceforge.net/
projects/slavereadahead/


DA
AG
EN Gotchas


Gotchas

• When you thought that you had figured
out everything ...


Gotchas: LOAD DATA

• LOAD DATA INFILE is replicated as a
temporary file
• The slave needs three times the size of
the data:
• the relay log

• the temporary file

• the data itself


Gotchas: default engine

• If you define storage_engine in the
master, it does not get replicated


Gotchas: binlog format

• In circular replication, and relay slaves
• Changes to binlog_format are not
propagated


Gotchas: binlog format

server id: 101 insert into t1
format: statement values (@@server_id) 101

server id: 102
format: row
102

server id: 103 102
format: row


Gotchas: set global not really global

• set global binlog_format=row
• Works for all new connections after the
change
• It does not work for the event
scheduler!


Gotchas: triggers and row replication

• In statement based replication, triggers
are fired both on master and slave
• in row-based replication, triggers are
fired on master only, and the resulting
rows are moved to the slave


DA
AG
EN Replication power techniques
and features

• row-based
• semi-synch
• delayed
• Circular
• bi-directional
• Leveraging replication


DA
AG
EN row-based replication



• Available in 5.1 and later
• can be changed using “binlog_format”
• row

• mixed

• statement (default)



• DO
• when you have expensive queries in the
master

• when you are using non-deterministic
functions

• DON’T
• When you are updating a lot of rows


DA
AG
EN semi-synch replication


semi-synchronous
replication

• Available in 5.5 and higher
• Makes sure that at least one slave has
copied the data.
• Increases reliability


client
master transaction
1 with regular
commit replication

execute 2

slave
returns
4 to client
binary log 3

5 replication
81

client transaction
master
1 with semi-
commit synchronous
replication

execute 2

slave
7 returns
to client
binary log 3

sends
4 transaction relay log 5
to slave

gets
6 acknowledgement
82

semi-synchronous
replication in practice

• installation:
• it’s a plugin.

• Actually, two plugins


semi-synch replication install
# in the master
plugin-load=rpl_semi_sync_master=semisync_master.so
rpl_semi_sync_master_enabled=1

# in each slave
plugin-load=rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled=1

# restar all servers


semi-synch replication check
# in the master
show variables like 'rpl_semi%';
+------------------------------------+-------+
+------------------------------------+-------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_no_slave | ON |
+------------------------------------+-------+


semi-synch replication check
show status like "rpl_semi_%tx";
+-----------------------------+-------+
| variable_name | value |
+-----------------------------+-------+
| RPL_SEMI_SYNC_MASTER_NO_TX | 0 |
| RPL_SEMI_SYNC_MASTER_YES_TX | 0 |
+-----------------------------+-------+


semi-synch replication test
master> create table t1 ( i int);
Query OK, 0 rows affected (0.01 sec)

master> show status like "rpl_semi_%tx";
+-----------------------------+-------+
+-----------------------------+-------+
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_yes_tx | 1 |
+-----------------------------+-------+


disabling semi-synch
# for each slave

set global rpl_semi_sync_slave_enabled=0;
stop slave io_thread;
start slave io_thread;


disabled semi-synch replication test
master> insert into t1 values (1);

+-----------------------------+-------+
+-----------------------------+-------+
+-----------------------------+-------+
2 rows in set (0.00 sec)


disabled semi-synch replication test

+-----------------------------+-------+
+-----------------------------+-------+
+-----------------------------+-------+


re-enabling semi-synch
# in one slave

set global rpl_semi_sync_slave_enabled=1;
stop slave io_thread;
start slave io_thread;


reenabled semi-synch replication test

+-----------------------------+-------+
+-----------------------------+-------+
+-----------------------------+-------+


DA
AG
EN delayed replication


delayed replication in practice
STOP SLAVE;
change master to master_delay=60;
START SLAVE;


delayed replication

• DEMO


DA
AG
EN Circular replication


Circular
A
A MySQL B
DBMS MySQL
DBMS B
master
master

MySQL

DD
DBMS MySQL
DBMS
C
C
Monday, April 11, 2011
master 97

DA
AG
EN bi-directional replication


bi-directional replication

MySQL MySQL
DBMS DBMS


bi-directional replication

MySQL MySQL
DBMS DBMS

Los Angeles New York

MySQL
MySQL MySQL DBMS
DBMS DBMS
MySQL
DBMS


DA
AG
EN Leveraging replication


Overview of MySQL partitions


Partition pruning
1a - unpartitioned table - SINGLE RECORD

I
D select *
N from table_name
A
D where colx =
T
E 120
A
X


Partition pruning
2a - table partitioned by colx - SINGLE REC
1-99

100-199 select *
from table_name
200-299
where colx =
120
300-399

400-499

500-599

Partition pruning
1a - unpartitioned table - RANGE

select *
I from
D
N table_name
A where colx
D between 120
T
E and 230
A
X


Partition pruning
2a - table partitioned by colx - RANGE
1-99

select *
100-199 from
table_name
200-299 where colx
between 120
and 230
300-399

400-499

500-599

Storage comparison

engine storage
(GB)
innodb (with PK) 330
myisam (with PK) 141
archive 13
innodb partitioned (no PK) 237
myisam partitioned (no PK) 107
archive partitioned 13


Benchmarking results
engine 6 month range
InnoDB 4 min 30s
MyISAM 25.03s
Archive 22 min 25s
InnoDB partitioned by month 13.19
MyISAM partitioned by year 6.31
MyISAM partitioned by month 4.45
Archive partitioned by year 16.67
Archive partitioned by month 8.97


Partitions limits
§ Little concurrency
§ Need of using the partitioning column


The ARCHIVE storage engine


Size matters

Innodb

Archive


REPLICATION PARTITIONING

large
data
ARCHIVE


Replication
=
let the slave do the dirty work


master
backup

slaves

STOP SLAVE

remove slave
perform
from load
backup
balancer

attach slave
START Let slave
to load
SLAVE catch up
balancer

master
make
summary
tables
slaves

STOP SLAVE

calculate remove slave
summary from load
tables balancer

attach slave
Let slave
START SLAVE to load
catch up
balancer

master
Partitions
for heavy innodb
non partitioned
statistics

slave slave

innodb
innodb non partitioned
partitioned by range

slave

MyISAM

Simulating master
multiple innodb
non partitioned
dimensions
slave

slave

innodb
non partitioned
ARCHIVE
(date)
slave
slave
ARCHIVE
ARCHIVE (location)
(product)

DA
AG
EN Tools and tips
• MySQL Sandbox
• Replaying binary logs
• Filtering binary logs
• Baron Schwartz’s Maatkit
• Shlomi Noach’s openark kit
• Facebook’s Online Schema Change
• Measuring replication speed
• Detecting if replication is on

DA
AG
EN MySQL Sandbox


MySQL Sandbox
http://mysqlsandbox.net
• Free software (Perl under GPL)
• One (unix) host
• Many database servers
• Single or multiple sandboxes
• Customized scripts to use the servers
• Standard or circular replication
• Installs IN SECONDS

overview

MySQL MySQL
server server
Data DB1 Data DB1

DB2 DB3
DATA DIRECTORY DB2 DB3

PORT

SOCKET


overview

MySQL MySQL
server server
Data DB1
SAME Data DB1

DB2 DB3
DATA DB2 DB3

DIRECTORY?
/var/lib/mysql /var/lib/mysql

DATA CORRUPTION

overview

MySQL MySQL
server server

SAME
PORT or
SOCKET?
3306 3306

/tmp/mysql.sock /tmp/mysql.sock

DOES NOT START

The hard way (1)


the hard way (2)


The easy way
$ make_sandbox
/path/to/mysql-5.1.54_linux.tar.gz

# it should work always


The easier way
$ make_sandbox 5.1.54

# Needs some preliminary work


The easiest way
$ sb 5.1.54

# Needs the same preliminary work


MySQL Sandbox
VERSION
MySQL
server
Data DB1

DB2 DB3
$SANDBOX_HOME/msb_VERSION/data

VERSION

/tmp/mysql_VERSION.sock


MySQL Sandbox
5.1.54
MySQL
server
Data DB1

DB2 DB3
$SANDBOX_HOME/msb_5_1_54/data

5154

/tmp/mysql_5154.sock


MySQL Sandbox
5.5.9
MySQL
server
Data DB1

DB2 DB3
$SANDBOX_HOME/msb_5_5_09/data

5509

/tmp/mysql_5509.sock


Single Sandbox
MySQL customized scripts
server

start
stop
restart
status
clear
send_kill
use


Multiple Sandbox
MySQL customized scripts
server

start_all
stop_all
restart_all m n1
status_all s1 n2
clear_all s2 n3
send_kill_a
ll
use_all

Where do you get it

•from CPAN
sudo cpan MySQL::Sandbox

•from launchpad
http://launchpad.net/mysql-sandbox


The easy replication way
$ make_replication_sandbox
/path/to/mysql-5.1.54_linux.tar.gz

# or, after some preparation

$ make_replication_sandbox 5.1.54


default architecture
$HOME

/sandboxes opt
expanded
tarballs
$SANDBOX_HOME
mysql

$SANDBOX_BINARY
installed
sandboxes


default architecture
$HOME

/sandboxes opt

msb_5_0_91
mysql
msb_5_1_48
5.0.91
rsandbox_5_1_48
5.1.45
master
5.1.48
node1
5.5.4
node2

creating a single sanbox
make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz


using a single sanbox
# after
# make_sandbox
# /path/to/mysql-X.X.XX-OS.tar.gz

$ cd $SANDBOX_HOME/msb_X_X_XX
$ ./use


creating a single sanbox
with a speciﬁc options ﬁle

make_sandbox
/path/to/mysql-X.X.XX-OS.tar.gz
--my_file=/path/to/my.cnf


easily create a sandbox after the ﬁrst
one
The long way
$ cd $HOME/opt/mysql # $SANDBOX_BINARY
$ gunzip -c
/path/to/mysql-5.1.34-osx10.5-x86.tar.gz
| tar -xf -
$ mv mysql-5.1.34-osx10.5-x86 5.1.34
$ make sandbox 5.1.34


easily create a sandbox after the ﬁrst
one
The short way
$ make_sandbox
path/to/mysql-5.1.34-osx10.5-x86.tar.gz
--export_binaries


starting a single sanbox
$ ./start


starting a single sanbox
with temporary options
$ ./start --option=value

$ ./restart --option=value

$ ./start --key-buffer=20000000


creating a sandbox with custom port
and directory
$ make_sandbox 5.1.34
--sandbox_port=7800
--sandbox_directory=mickeymouse


creating a sandbox with automatic
port checking
$ make_sandbox 5.1.34 --check_port

# if 5.1.34 is free
# port=5134
# directory=msb_5_1_34
# else
# port=5135 (or the first free)
# directory=msb_5_1_34_a


create a replication sandbox



create a circular replication sandbox

--circular=4


changing port to an existing sandbox

$ sbtool -o port
-s /path/to/source/sandbox
--new_port=XXXX


installing the innodb plugin

$ sbtool -o plugin
--plugin=innodb
-s /path/to/source/sandbox


creating a replication sandbox with
new base port

--replication_directory=newwdir
--check_base_port 5.0.79

# Creates a replication directory under
# $SANDBOX_HOME/newdir
# The previous one is preserved.
# No conflicts happen


DA
AG
EN Replaying binary logs


DA
AG
EN Filtering binary logs


DA
AG
EN Maatkit


maatkit
• mk-heartbeat Monitor MySQL replication delay.
• mk-purge-logs Purge binary logs on a master based on purge rules.
• mk-slave-delay Make a MySQL slave server lag behind its master.
• mk-slave-find Find and print replication hierarchy tree of MySQL
slaves.
• mk-slave-move Move a MySQL slave around in the replication
hierarchy.
• mk-slave-prefetch Pipeline relay logs on a MySQL slave to pre-warm
caches.
• mk-slave-restart Watch and restart MySQL replication after errors.
• mk-table-checksum Perform an online replication consistency check,
or checksum MySQL tables efficiently on one or many servers.
• mk-table-sync Synchronize MySQL table data efficiently.
• http://www.maatkit.org/

Facebook Online Schema Change
DA
N
AGE


Facebook Online Schema Change

• Based on one of Shlomi Noach OpenArk utilities
• Developed in PHP by Facebook

• http://www.facebook.com/note.php?note_id=430801045932


DA
AG
EN OpenArk


openark

• oak-get-slave-lag: print slave replication lag and
terminate with respective exit code.

• oak-online-alter-table: perform a non-blocking ALTER
TABLE operation.

• oak-purge-master-logs: purge master logs, depending
on the state of replicating slaves.

• oak-show-replication-status: show how far behind are
replicating slaves on a given master.

• http://code.openark.org/forge/openark-kit


DA
AG
EN Check replication speed


check replication speed

• Write a time to the master
• Retrieve it from the slave
• Compare times
• http://forge.mysql.com/tools/tool.php?
id=5


DA
AG
EN Replication outside the box

• Seamless failover
• Parallel replication
• Multiple source replication
• Multiple master replication


Advanced MySQL
Replication for the
masses


Once Upon A Time,
In The Life Of A
Database
Consultant ...

The story of a steel
foundry

• Used MySQL databases to store
production monitoring data
• Inserted a zillion records per second.
• Slaves often lagged behind


transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction

BINARY LOG

ns act act tio n
tra ansio ns sac ntio
ac ion ion n
ac
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa

n
n
tio
ct
tra

tra
REPLICATION
transaction
transaction

MySQL
DBMS


The story of a shoe
maker

• Had a successful business, spread to a
dozen stores.
• Needed to aggregate the data from the
stores in his headquarters database.


MySQL
MySQL
DBMS MySQL
DBMS
DBMS

store
store store

MySQL
DBMS

headquarters


The story of a widgets
seller
• Had a successful business, designed for one
server.
• Products were created in several sites.
• Needed to allow insertions from more
than one master at the time


MySQL MySQL
DBMS DBMS

master master

MySQL MySQL
DBMS DBMS

master master

All these stories tell us:

Nice dream, but MySQL
can’t do it

Enter Tungsten Replicator


Tungsten Replicator 2.0

• What is it?


http://code.google.com/p/tungsten-replicator

Open Source
100% GPL v2


What can it do?

• Easy failover
• Multiple masters
• Multiple sources to a single slave
• Parallel replication
• Replicate to Oracle and PostgreSQL database


MySQL to foreign services

MySQL setup
to run as
MySQL master

Master DB
Data is applied to
PostgreSQL or
Oracle

Bin
Logs
(Binlogs
enabled) Slave DB

© Continuent 2010


From the beginning ...


Fail-over (1)


Fail-over (2)


Fail-over (3)


Fail-over (4)


Fail-over (5)


Fail-over (6)


Fail-over (7)


Fail-over (8)


Failover

• DEMO


master/slave with an attitude


The steel foundry dream
or parallel replication
From here ...


The steel foundry dream
or parallel replication
To here.


Parallel replication facts

• Sharded by database
• Good choice for slave lag problems
• Bad choice for single database projects


Testing parallel
replication


db0
preparation (1)
db1 sysbench

db2 sysbench

db3 sysbench

db4 sysbench

db5 sysbench

db6 sysbench

db7 sysbench

db8 sysbench

db9 sysbench

sysbench


preparation (2)
db0

db1

db2

db3

db4

db5

db6

db7

db8

db9


before the test (1)
db0

db1

db2

db3

db4

db5

db6

db7

db8

db9


before the test (2)
SQL thread

RELAY logs

IO thread

binary logs

MySQL slave

RELAY logs

Tungsten slave direct:
alpha
(slave)

replicator alpha


starting the test
SQL thread

RELAY logs

IO thread

binary logs

MySQL slave

RELAY logs

Tungsten slave direct:
alpha
(slave)

replicator alpha


MySQL native
replication

slave catch up in 00:02:30


Tungsten parallel
replication

slave catch up in 00:01:20


The widget seller
dream, or multi masters

• Tungsten Replicator recipe: use more
services


Bi-directional replication


Bi-directional replication with slaves


True multiple master

We’ll see that in a moment.
But ﬁrst


The shoe maker dream,
or multiple sources

• Tungsten Replicator recipe is still valid: use
more services


Multiple source replication


Multiple masters replication: 3 nodes


Multiple masters replication: 4 nodes


Updating 4 masters : 1 ﬂow


Updating 4 masters : 2 ﬂows


Tungsten in practice

• installation


Installation:
the long way
• Get the binaries
• Expand the tarball
• Check the requirements in the manual
• Run ./conﬁgure
• Run ./conﬁgure-service


Installation:
the quick way

• Get the tarball
• Get the tungsten deployer from Google
Code
• Sit back and enjoy!


Tools

• replicator
• trepctl
• thl


replicator

• It’s the service provider
• You launch it once when you start
• You may restart it when you change conﬁg


trepctl

• Tungsten Replicator ConTroLler
• It’s the driving seat for your replication
• You can start, update, and stop services
• You can get speciﬁc info


thl

• Transaction History List
• Gives you access to the Tungsten relay logs


DA
AG
EN More info on replication


Books


High Performance MySQL


MySQL High Availability


Web Operations


Cloud Application Architectures


Talks
• Monday, 1:30pm “Learn how to cure replication
deprivation with Tungsten”

• Tuesday 10:50am “Replication Update”

• Tuesday 2:00pm “Diagnosing replication failures”

• Tuesday 3:05 “Advanced replication monitoring”

• Thursday 10:50am “Replication for Availability &
Durability with MySQL and Amazon RDS”

• Thursday 11:55am “Build your own PaaS for MySQL
with Tungsten Enterprise”

• Thursday 2:00pm “Error detection and correction with
MySQL Replication”

• Thursday 2:50pm “Performance comparisons and
trade-offs for various MySQL replication schemes”

ADVERTISING


WE ARE HIRING!

QA and support
engineers
http://www.continuent.com/about/careers


http://opendbcamp.org

Open Database Camp

Sardinia
Technology Park

6-7-8 May 2011


Contact Information

Worldwide
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel (866) 998-3642
Fax (408) 668-1009
e-mail: sales@continuent.com

Continuent Web Site:
http://www.continuent.com
Tungsten Project
http://code.google.com/p/tungsten-replicator

© Continuent 2011


Advanced mysql replication techniques

More Related Content

Viewers also liked (20)

Similar to Advanced mysql replication techniques (20)

More from Giuseppe Maxia (20)

Recently uploaded (20)

Advanced mysql replication techniques