Webinar slides: Replication Topology Changes for MySQL and MariaDB
1 like4,365 views
This document discusses replication topology changes for MySQL and MariaDB databases. It covers making changes using GTID or regular replication, the failover process, and tools like MaxScale and ProxySQL that can help automate query rerouting during a failover. Specific topics covered include reslaving nodes, setting up master-master replication, and performing both offline and online failovers.
Webinar slides: Replication Topology Changes for MySQL and MariaDB
- 1. 1 Your host & some logistics I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.
- 2. Copyright 2015 Severalnines AB Replication Topology Changes for MySQL and MariaDB September 29, 2015 Krzysztof Książek Severalnines [email protected] 2
- 3. Copyright 2015 Severalnines AB ! We want to help all non-DBA people who look after MySQL infrastructures ! Share tips and good practices ! Watch the replays of the previous webinars on our Slideshare page ! http://www.slideshare.net/Severalnines/videos 3 “Become a MySQL DBA” series
- 4. Copyright 2015 Severalnines AB ! MySQL Replication topology changes ! using GTID ! using regular replication ! Failover process ! Using MaxScale for automatic re-routing of queries ! Other external tools useful when dealing with failover 4 Agenda
- 5. Copyright 2015 Severalnines AB MySQL Replication topology changes 5
- 6. Copyright 2015 Severalnines AB ! Maintenances ! Adding an index on a large table ! Delete data and then rebuild a large table to reclaim disk space ! Reslaving nodes for a planned failover ! Reslaving nodes after master crash 6 Why do we make changes in a replication chain?
- 7. Copyright 2015 Severalnines AB ! GTID allows for a flexible topology changes ! CHANGE MASTER TO … MASTER_AUTO_POSITION=1; ! You can slave off every node in the replication chain ! Be aware of errant transactions, though ! http://www.severalnines.com/blog/mysql-replication-and- gtid-based-failover-deep-dive-errant-transactions 7 Replication topology changes - GTID
- 8. Copyright 2015 Severalnines AB ! Far less flexible than GTID ! Slave knows only a binlog position of itself and it’s master ! You can use binary and relay logs to identify position of a given transaction in the replication chain but it’s a complex and error prone process ! To avoid it, you have to stop writes on a involved subset of hosts 8 Replication topology changes - no GTID
- 9. Copyright 2015 Severalnines AB Replication topology changes - no GTID 9
- 10. Copyright 2015 Severalnines AB ! First step requires to slave DB3 and DB4 off DB2 ! You need to enable log- slave-updates on DB2 ! Stop DB2, DB3 and DB4 at the same position using START SLAVE UNTIL … ! Start it until a beginning of binary log two files later 10 Replication topology changes - no GTID
- 11. Copyright 2015 Severalnines AB ! Locate your first position in binlog using mysqlbinlog tool ! START SLAVE UNTIL master_log_file='mysql-bin.000121', master_log_pos=4; ! Run FLUSH LOGS on the master two times to rotate logs - slaves should end up stopped at the same position ! Check the position on DB2 (SHOW MASTER STATUS) ! Use it to slave DB3 and DB4 off DB2 11 Replication topology changes - no GTID
- 12. Copyright 2015 Severalnines AB ! We should finally reach the desired topology ! Next step - setup master - master replication between DB1 and DB2 ! Confirm that DB2 is not taking writes other than the replication ! Use server_id in binary logs for that - you should see only DB1 ! If it does writes, you’re in troubles - investigate and eliminate the culprit 12 Replication topology changes - no GTID
- 13. Copyright 2015 Severalnines AB ! Once confirmed writes are not hitting DB2, execute CHANGE MASTER TO … on DB1 pointing it to the DB2 and using any recent coordinates ! Monitor Exec_Master_Log_Pos in the SHOW SLAVE STATUS on DB1, it should be stable. If it does increase, something is still writing to DB2. ! Once all is set up, you are ready for a failover 13 Replication topology changes - no GTID
- 14. Copyright 2015 Severalnines AB Failover process 14
- 15. Copyright 2015 Severalnines AB ! Exact failover process may depend strongly on the infrastructure that you use ! In short - it’s a process of moving writes from one database node to another ! It may or may not be graceful, depending on your application and tools that you use 15 Failover Process
- 16. Copyright 2015 Severalnines AB ! Requires downtime period for a master switch ! Not ideal solution but easiest to implement and it will work with every application that can be stopped ! We are assuming the topology we reached in our previous slides, with failover about to happen from DB1 to DB2 16 Offline failover
- 17. Copyright 2015 Severalnines AB ! As a first step, you need to stop all activity in your application - you don’t want to have any writes hitting db ! You can monitor it using SHOW MASTER STATUS on the master ! You can monitor binary logs to locate and identify writes that hit the database ! Com_* counters can also be monitored 17 Offline failover
- 18. Copyright 2015 Severalnines AB ! Next step - you need to repoint your application to the new master, DB2 in our example ! Depending on your setup you can: ! Change DNS entries ! Set read_only values accordingly on MySQL hosts ! Use your proxy’s settings to redirect connections from one host to the other 18 Offline failover
- 19. Copyright 2015 Severalnines AB ! Once you pointed writes to a new master, you may want to perform some tests before going back live ! All the time, ensure that any writes are happening only on the correct host - new master, DB2 in our case ! Do not use any ‘hardcoded’ IP’s in your scripts - this may become a serious problem when you will be performing a failover 19 Offline failover
- 20. Copyright 2015 Severalnines AB ! Moving writes without downtime period - planned failover ! Stop the writes on the master, failover, move the writes to the new master - sounds familiar? ! Main problem is - how to do it gracefuly? ! FTWRL + read_only = 1? May take too long ! read_only = 1? Will kill current connections ! Drop the user + wait few seconds + read_only = 1? Still not ideal but may be acceptable 20 Online failover
- 21. Copyright 2015 Severalnines AB ! Proxies may help significantly in this regard - single point of config change, immediate result ! Caching reads and writes can also be useful ! Application will have to be able to handle errors and rollbacks as transition may but also may not be graceful ! Some of the tools can help you to make it graceful, without application seeing any error at all 21 Online failover
- 22. Copyright 2015 Severalnines AB Using MaxScale to reroute queries 22
- 23. Copyright 2015 Severalnines AB ! MaxScale can implement read-write split ! MaxScale can be used with MySQL replication backend ! It monitors health of the backend nodes ! It detects replication topology changes ! Makes easier to handle any changes like failover 23 Using MaxScale to reroute queries
- 24. Copyright 2015 Severalnines AB ! For reads, most of the failures are being handled gracefully ! For writes, not yet - errors will be thrown and transactions rolled back, application has to handle them ! But it should be handling them anyway, right? ! Experimental support for handling failovers (along with reslaving) in the latest MaxScale ! No easy way of performing a planned failover, though 24 Using MaxScale to reroute queries
- 25. Confidential Copyright 2015 Severalnines AB External tools which help handling failover process
- 26. Copyright 2015 Severalnines AB ! Handles the failover process for you ! Both automatic failover and manual one ! Can be used as a standalone solution or a part of a set of scripts or tools ! Integrates nicely with other tools in a manual failover process ! Can work with GTID and regular replication 26 MHA
- 27. Copyright 2015 Severalnines AB ! Part of MySQL Utilities from Oracle ! Allows user to perform a manual failover ! Works only with GTID replication ! Checks for errant transactions ! Can be easily combined with other scripts through --exec- before and --exec-after 27 mysqlrpladmin
- 28. Copyright 2015 Severalnines AB ! New addition to the ‘database-aware’ proxy family ! Works as a gateway for all connections to databases ! Performs read/write split based on user-defined set of rules ! Gives you ability to perform a sub second graceful failover in case of a master-master replication. ! Works great with external scripts for handling topology changes 28 ProxySQL
- 29. Copyright 2015 Severalnines AB ! Idea is simple - all queries go through proxy to backend nodes ! You can use failover script to switch a backend node into a ‘failover’ mode ! Proxy will not forward more connections to it, it will “buffer” new connections ! Your failover script can wait till the current transactions finish or timeout will be reached and transactions can be killed ! In a next step you can reconfigure the replication topology as needed 29 ProxySQL
- 30. Copyright 2015 Severalnines AB ! Once there are no transactions running and new topology has been set up, proxy can be reconfigured to move writes to another backend ! Depends on how long are your transactions and how long timeout is set, failover can take even a fraction of a second ! Application will see temporary slowdown but should not see any errors ! Additional scripting is required - ProxySQL doesn’t monitor replication topology on it’s own 30 ProxySQL
- 31. Copyright 2015 Severalnines AB ! More blogs in “Become a MySQL DBA” series: ! http://www.severalnines.com/blog/become-mysql- dba-blog-series-query-tuning-process ! http://www.severalnines.com/blog/become-mysql- dba-blog-series-configuration-tuning-performance ! Contact: [email protected] 31 Thank You!