Replication Tips & Trick for SMUG

Replication Tips & Tricks + some things about GTIDs
Mats Kindahl
mats.kindahl@oracle.com

1Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12

Program Agenda

 Replication setup and status checking
 Slave fail-over using GTID
 Binary log analysis
 Crash-safe slaves
 Multi-source replication

2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

About the Presentation

This presentation will introduce you to some replication
features and also briefly show tips and tricks on how to
work with replication. The focus is on short ideas and
each item does not go deeper into details.


What is Replication?

● MySQL Master Server
– Changes data
– Sends changes to slave
● MySQL Slave Server
– Receives changes from master
– Applies received changes to
database


Replication Topologies

Dual Master
Circular

Tree
Simple


Why Replication?
Scaling out

Writes

Reads


Why Replication?
Redundancy

Slave
Crash Promotion

New Master


Replication Architecture
Master Slave


Reading Slave Status
Understanding the fields of SHOW SLAVE STATUS

● I/O Thread Status ● SQL Thread Status
– Slave_IO_Running – Slave_SQL_Running
– Last_IO_Errno – Last_SQL_Errno
– Last_IO_Error – Last_SQL_Error
– Last_IO_Error_Timestamp – Last_SQL_Error_Timestamp
● Master being replicated from
– Master_Host
– Master_Port


Reading Slave Status
Understanding the fields of SHOW SLAVE STATUS

● Next event to execute – in master log coordinates
– Relay_Master_Log_File + Exec_Master_Log_Pos
● Next event to execute – in relay log coordinates
Relay_Log_File + Relay_Log_Pos
● Next event to read from master
Master_Log_File + Read_Master_Log_Pos


Reading the Binary Log
Decoding the binary log

● Use mysqlbinlog
--hexdump Show hex dump of event as comment
--start-position Start dumping at a position
--stop-position Stop dumping after this position

# at 275 175 =373
16 10
#120927 23:11:58 server id 3 end_log_pos 373
# Position Timestamp Type Master ID Size Master Pos Flags
# 113 1e c1 64 50 02 03 00 00 00 62 00 00 00 75 01 00 00 00 00

Little-endian
Query

Decoding the binary log

# at 275
#120927 23:11:58 server id 3 end_log_pos 373
# Position Timestamp Type Master ID Size Master Pos Flags
# 113 1e c1 64 50 02 03 00 00 00 62 00 00 00 75 01 00 00 00 00
# 126 c5 03 00 00 00 00 00 00 04 00 00 1a 00 00 00 00 |................|
# 136 00 00 01 00 00 00 00 00 00 00 00 06 03 73 74 64 |.............std|
# 146 04 21 00 21 00 08 00 74 65 73 74 00 69 6e 73 65 |.......test.inse|
# 156 72 74 20 69 6e 74 6f 20 74 32 20 76 61 6c 75 65 |rt.into.t2.value|
# 166 73 20 28 31 2c 27 74 65 73 74 69 6e 67 27 29 |s..1..testing..|
# Query thread_id=965 exec_time=0 error_code=0
SET TIMESTAMP=1348780318/*!*/;
insert into t2 values (1,'testing')
/*!*/;


Decode row events

● Use mysqlbinlog
--verbose Decode row events into pseudo-SQL
# at 849
# at 893
#120928 15:19:24 server id 3 end_log_pos 893 Table_map: `test`.`t2` mapped to number 48
#120928 15:19:24 server id 3 end_log_pos 941 Write_rows: table id 48 flags: STMT_END_F

BINLOG '
3KNlUBMDAAAALAAAAH0DAAAAADAAAAAAAAEABHRlc3QAAnQyAAIDDwIoAAM=
3KNlUBcDAAAAMAAAAK0DAAAAADAAAAAAAAEAAv/8AwAAAA1yb3dzIGFyZSBjb29s
'/*!*/;


Decode row events

● Use mysqlbinlog
--verbose Decode row events into pseudo-SQL
# at 849
# at 893
#120928 15:19:24 server id 3 end_log_pos 893 Table_map: `test`.`t2` mapped to number 48
#120928 15:19:24 server id 3 end_log_pos 941 Write_rows: table id 48 flags: STMT_END_F

BINLOG '
3KNlUBMDAAAALAAAAH0DAAAAADAAAAAAAAEABHRlc3QAAnQyAAIDDwIoAAM=
3KNlUBcDAAAAMAAAAK0DAAAAADAAAAAAAAEAAv/8AwAAAA1yb3dzIGFyZSBjb29s
'/*!*/;
### INSERT INTO test.t2
### SET
### @1=3
### @2='rows are cool'


Global Transaction Identifiers (GTIDs)
Handling positions in MySQL 5.5

● Binary log positions manually handled
● Each server has it's own position
● Failing over slaves to new masters difficult
– What is the position to fail over to?

CHANGE MASTER TO
MASTER_HOST='master1.example.com', MASTER_PORT=3306,
MASTER_USER='repl_user', MASTER_PASSWORD='xyzzy',
MASTER_LOG_FILE='master-bin.00001', MASTER_LOG_POS=22145;
Different for
different servers

Enabling Global Transaction IDs

● MySQL 5.6 has Global Transaction ID
– Positions independent of server
● GTID handshake
– Done on connection with master
– Negotiate position automatically

CHANGE MASTER TO
MASTER_AUTO_POSITION=1;


Enabling Global Transaction IDs

● Turn on GTIDs [mysqld]
…
● Turn off dangerous statements gtid-mode=on
– CREATE TABLE...SELECT enforcegtidconsistency=on
log-bin=master-bin
– CREATE TEMPORARY TABLE log-slave-updates
● Inside transactions

CHANGE MASTER TO
MASTER_AUTO_POSITION=1;


Global Transaction ID Concepts

● Global transaction ID
– A server UUID + Transaction Number
a61678ba488942799e5845ba840af334:1
– Preserved when replicated
● Global Transaction ID Set
– Sequence of Server UUID + Transaction Number Range
a61678ba488942799e5845ba840af334:15,
f88e2a43a13e11e298de0021cc6850ca:122564


Server Variables

● @@GTID_EXECUTED
– GTID set of all transactions logged in the binary log
● @@GTID_PURGED
– GTID set of all transaction purged from binary log
– Cannot be sent to slaves


Cloning a Slave
Be
My fo
Creating a slave image SQ re
L
5.
1. Stop the slave 6
2. Write down where slave has stopped
– SHOW SLAVE STATUS
3. Backup slave
– mysqldump – slow but safe
dumpslave Generate CHANGE MASTER
– Physical file copy – fast but you need to shut down the server


Cloning a Slave
Be
My fo
Setting up the new slave SQ re
L
5.
4. Restore backup on new slave 6
5. Direct the slave to the master
– CHANGE MASTER
– Master_Host + Master_Port
6. Start slave
– START SLAVE


Cloning a Slave
My
SQ
or L
5
1. Stop the slave lat .6
er
2. Backup slave
mysqldump host=slave1.example.com user=root
dumpslave setgtidpurged >image.sql
3. Start the slave
Tell slave that transactions
4. Restore image on new slave
in image cannot be replicated
mysql host=slave2.example.com user=root <image.sql
5. Start Slave


Point in time recovery

● Use mysqlbinlog
--to-datetime Stop reading when reaching date
● Example
mysqlbinlog --read-from-remote-server
e
--host=master -uroot --position=192 ar !
--to-datetime=”2009-04-11 12:36:56” ps ing
am reas
st
master-bin.00002[2-4] | e nc
mysql -uroot --host=slave tim lly i
: a
ing ic
r n ot on
W a on
m
not


Using Relay Servers

● Relieve master server by
creating relay slaves
● Just keep binary logs
● Do not store data in tables
– Use BLACKHOLE engine
– --log-slave-updates

SET STORAGE_ENGINE = BLACKHOLE
ALTER TABLE table ENGINE = BLACKHOLE


Using Relay Servers

replicate-wild-do-table=*.%_east replicate-wild-do-table=*.%_west

● Filtering replication stream
● Filtering done on relay servers
● replicatewild*table
– Work with cross-database
queries


Write Your Own Library
Write your own utility library

● Create a utility library to easier work with replication
● Common functions to manage replication:
– Start/stop slave
– Change master
– Fetch master position
– Fetch slave execute/read position
– Fetch master host information


Basic functions using Bourne shell

# Usage: mysql_exec socket sql ... # Usage: stop_slave socket [ thread ]
mysql_exec () { stop_slave () {
    sock=$1    mysql_exec $1 STOP SLAVE $2
    shift 1 }
    mysql uroot vertical batch
        skipcolumnnames          # Usage: start_slave socket [ thread ]
        socket="$sock"             start_slave () {
        exec "$*"    mysql_exec $1 START SLAVE $2
} }



# Usage: change_master socket [ host [ port [ file [ pos ] ] ] ]
change_master () {
    host=${2:+MASTER_HOST='$2'}
    port=${3:+,MASTER_PORT=$3}
    file=${4:+,MASTER_LOG_FILE='$4'}
    pos=${5:+,MASTER_LOG_POS=$5}
    mysql_exec $1 CHANGE MASTER TO $host $port $file $pos
}

# Usage: fetch_master_pos socket
fetch_master_pos () {
   mysql_exec $1 SHOW MASTER STATUS |
   grep '<File|<Pos'             |
   cut f2 d:
}



# Usage: fetch_slave_exec_pos socket
fetch_slave_exec_pos () {
   mysql_exec $1 SHOW SLAVE STATUS |
   grep '<Relay_Master_Log_File|<Exec_Master_Log_Pos' |
   cut f2 d:
}

# Usage: fetch_slave_read_pos socket
fetch_slave_read_pos () {
   grep '<Master_Log_File|<Read_Master_Log_Pos' |
   cut f2 d:
}


master1.example.com master2.example.com
Round-Robin Replication

● Slave can only replicate from single
master at a time
● Use time sharing to replicate from
several masters
Stop

Switcher
Start Save

slave1.example.com
Restore


A version using Bourne Shell

# Usage: fetch_host_and_pos socket
fetch_host_and_pos () {
grep '<Master_(Host|Port|Log_File)|<Read_Master_Log_Pos' |
cut -f2 -d:
}

# Usage: stop_and_save socket
stop_and_save () {
sock="/var/run/mysqld/$1.sock"
stop_slave $socket
fetch_host_and_pos $sock >$1.savepos
}


A version using Bourne Shell

restore_and_start () { cnt=1
socket="/var/run/mysqld/$1.sock" while true
cat $1.savepos | { do
read host stop_and_save mysqld.$cnt
read port cnt=`expr $cnt % 5 + 1`
read file restore_and_start mysqld.$cnt
read pos sleep 60
change_master $socket done
$host $port $file $pos
start_slave $socket
}
}


Crash-safe Slaves
Keeping replication information in sync with data
FILE TABLE
● Traditional Replication
– Position stored in file
– Update after transaction
– Crash can lose update
● Transactional Replication ● Repository location FILE or TABLE
– Positions stored in table master_info_repository
– Update part of transaction relay_log_info_repository
– Crash-safe


Crash-safe Slaves
Steps to set up crash-safe slaves

● Direct server to use table-based repository
– Defaults to FILE
master_info_repository=TABLE
relay_log_info_repository=TABLE
● Ensure that a transactional engine is used
– InnoDB default since 5.6.6
– Before 5.6.6: set the storage engine
ALTER TABLE slave_master_info ENGINE = InnoDB
ALTER TABLE slave_relay_log_info ENGINE = InnoDB


A version using pure SQL

CREATE TABLE my_masters ( CREATE TABLE current_master (
idx INT PRIMARY KEY, idx INT
host CHAR(50) NOT NULL, ) ENGINE=InnoDB
port INT NOT NULL DEFAULT 3306,
log_file CHAR(50),
log_pos LONG,
UNIQUE INDEX (host,port,user)
) ENGINE=InnoDB

CREATE PROCEDURE save_position()
BEGIN
DECLARE l_idx INT UNSIGNED;

UPDATE my_masters AS mi, mysql.slave_relay_log_info AS rli
SET mi.log_pos = rli.master_log_pos, mi.log_file = rli.master_log_name
WHERE idx = (SELECT idx FROM current_master);
END


In pure SQL

CREATE PROCEDURE fetch_next_master(OUT p_host CHAR(50), OUT p_port INT UNSIGNED,
OUT p_file CHAR(50), OUT p_pos BIGINT)
BEGIN
DECLARE l_next_idx INT DEFAULT 1;

SELECT idx INTO l_next_idx FROM my_masters Select next index,
WHERE idx > (SELECT idx FROM current_master) if there is one
ORDER BY idx LIMIT 1;

SELECT idx INTO l_next_idx FROM my_masters Select the first index, if
WHERE idx >= l_next_idx ORDER BY idx LIMIT 1;
there were no next index
UPDATE current_master SET idx = l_next_idx;

SELECT host, port, log_file, log_pos Fetch master info using
INTO p_host, p_port, p_file, p_pos
FROM my_masters WHERE idx = l_next_idx; the index
END

In pure SQL

CREATE EVENT multi_source ON SCHEDULE EVERY 10 SECOND DO
BEGIN
DECLARE l_host CHAR(50);
DECLARE l_port INT UNSIGNED;
DECLARE l_file CHAR(50);
DECLARE l_pos BIGINT;

SET SQL_LOG_BIN = 0;

STOP SLAVE;
START TRANSACTION;
CALL save_current_position();
CALL fetch_next_master(l_host, l_port, l_file, l_pos);
CALL change_master(l_host, l_port, 'repl_user', 'xyzzy', l_file, l_pos);
COMMIT;
START SLAVE;
END


Different Tables on Master and Slave

● Upgrading
– Example: Upgrading a topology without downtime
● Saving space
– Not keeping big columns on slave
● Hiding data
– Keeping data away from users
– Note: still available in binary log and relay log – just not applied to table
● Debugging and Auditing
– Adding data for auditing


More columns on slave

CREATE TABLE employee (
name CHAR(40),
Master CREATE TABLE employee (
name CHAR(40),
Slave
email CHAR(40) email CHAR(40),
changed TIMESTAMP
DEFAULT CURRENT_TIMESTAMP
ON UPDATE CURRENT_TIMESTAMP
) )

● Same initial columns
● Statement-based or row-based replication


More columns on master

CREATE TABLE employee ( CREATE TABLE employee (
name CHAR(40), name CHAR(40),
email CHAR(40), email CHAR(40),
password CHAR(40) password CHAR(40)
) Master ) Slave

● Same initial columns
● Row-based replication only


Different types of columns on master and slave

emp_id INT, emp_id TINYINT,
email CHAR(40) email CHAR(40)
) Master ) Slave

● Table definitions are identical
… except that some column type are different


Slave type conversions

) Master ) Slave

● Consider this statement: St
at
em
INSERT INTO employee VALUES Re ent
Works fine (1, 'Bob', 'bob@example.com'), pl -b
ica a
(500, 'Alice', 'alice@example.com') tio sed
Fails silently!
n


Different Tables on Master and Slave Before
MySQL 5.5.3

emp_id TINYINT, emp_id INT,
) Master ) Slave

● Consider this statement: Ro
w
Re -ba
pl
ica sed
INSERT INTO employee VALUES
(1, 'Bob', 'bob@example.com'), tio
Throws an error!
(100, 'Alice', 'alice@example.com') n


Different Tables on Master and Slave MySQL 5.5.3
and later

● @@GLOBAL.SLAVE_TYPE_CONVERSIONS
– Type: SET('ALL_LOSSY', 'ALL_NON_LOSSY')
– Require slave restart

● Lossy conversions ● Non-lossy conversions
– INT → TINYINT – TINYINT → INT
– CHAR(64) → VARCHAR(32) – VARCHAR(32) → CHAR(64)
– DOUBLE → FLOAT – FLOAT → DOUBLE


and later

● Conversions within same domain possible
– INT, TINYINT, SMALLINT, MEDIUMINT, BIGINT
– CHAR(n), VARCHAR(n), TINYTEXT, TEXT, …
– DECIMAL(n,m), DOUBLE, FLOAT
● ALL_NON_LOSSY in SLAVE_TYPE_CONVERSIONS
– Conversion to larger domains allowed
● ALL_LOSSY in SLAVE_TYPE_CONVERSIONS
– Conversion to smaller domains allowed (truncation/rounding may occur)


and later

emp_id TINYINT, emp_id INT,
name VARCHAR(40), name VARCHAR(40),
email VARCHAR(40) email VARCHAR(40)
) Master ) Slave

w
Re -ba
pl
ica sed
Works fine!

SLAVE_TYPE_CONVERSIONS = 'ALL_NON_LOSSY'


and later

name VARCHAR(40), name VARCHAR(40),
email VARCHAR(40) email VARCHAR(40)
) Master ) Slave

w
Re -ba
pl
ica sed
Throws an error!

SLAVE_TYPE_CONVERSIONS = 'ALL_NON_LOSSY'


Master and Slave out of sync

● Slave stopped with a strange error
ERROR 1062 (23000): Duplicate entry '3' for key 'PRIMARY'
● Compare two databases using mysqldbcompare
mysqldbcompare
--server1=root@master.example.com
--server2=root@slave1.example.com
employee


Comparing two databases and finding the differences

$ mysqldbcompare --quiet --server1=root@localhost:3309 --server2=root@localhost:3310 world
# Checking databases world on server1 and world on server2
#

#
# Data differences found among rows:
--- world.City
+++ world.City
@@ -1,5 +1,5 @@
-+-------+------------+--------------+-----------+-------------+
-| ID | Name | CountryCode | District | Population |
-+-------+------------+--------------+-----------+-------------+
-| 3048 | Stockholm | SWE | Lisboa | 750348 |
-+-------+------------+--------------+-----------+-------------+
++-------+-----------+--------------+-----------+-------------+
+| ID | Name | CountryCode | District | Population |
++-------+-----------+--------------+-----------+-------------+
+| 3048 | Helsinki | SWE | Lisboa | 750348 |
++-------+-----------+--------------+-----------+-------------+


Synchronizing master and slave databases

$ mysqldbcompare --quiet --difftype=sql --changes-for=server2
> --server1=root@master.example.com --server2=root@slave1.example.com
> world
#

#
# Transformation for --changes-for=server2:
#

UPDATE world.City SET Name = 'Stockholm' WHERE ID = '3048';


Synchronizing master and slave databases

$ mysqldbcompare --quiet --difftype=sql --changes-for=server1
> --server1=root@master.example.com --server2=root@slave1.example.com
> world
#

#
# Transformation for --changes-for=server1:
#

UPDATE world.City SET Name = 'Helsinki' WHERE ID = '3048';


Replication Event Checksums
Master Slave

Event
Checksums


Replication Event Checksums
Configurations and options

● Controlling checksum generation (default is CRC32 since 5.6.6)
SET GLOBAL BINLOG_CHECKSUM = CRC32
● Enable verification on events read by dump thread
SET GLOBAL MASTER_VERIFY_CHECKSUM = ON
● Enable verification on events read by SQL thread
SET GLOBAL SLAVE_SQL_VERIFY_CHECKSUM = ON


The preceding is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole
discretion of Oracle.


Replication Tips & Trick for SMUG

More Related Content

What's hot (20)

Similar to Replication Tips & Trick for SMUG (20)

More from Mats Kindahl (10)

Recently uploaded (20)

Replication Tips & Trick for SMUG