2019 BGOUG Autumn MySQL Clone
Download as PPTX, PDF4 likes829 views
This document introduces MySQL Clone, a new plugin that provides a simple way to create consistent replicas of a MySQL database instance. It allows cloning data from a "donor" server to a new "recipient" server. The clone operation produces a replica with matching binary log positions and transactional consistency. It is faster than traditional replication and supports resuming interrupted clones. MySQL Clone can be used to provision new replication nodes or re-provision existing nodes.
2019 BGOUG Autumn MySQL Clone
- 1. Georgi “Joro” Kodinov Software Development Director MySQL Server Team Nov, 2019 MySQL Clone A better way to migrate databases 1 © 2019 Oracle
- 2. Safe harbor statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. 2 © 2019 Oracle
- 3. What is MySQL Clone ? A simple and efficient way to create a MySQL instance replica © 2019 Oracle3
- 4. WhyYet Another Method ? • It’s easy to use: a single command • It’s a data snapshot with consistent GTID and binary log position • Data can be cloned locally or from remote • It’s resumable ! © 2019 Oracle4
- 5. It’s FASTER Too ! 5 https://mydbops.wordpress.com/2019/11/14/ mysql-clone-plugin-speed-test/ 2 servers CPU: 2 Cores RAM: 4 GB DISK: 200 GB SSD approx. 122.40 GB data from sysbench
- 6. Clone: Use Cases © 2019 Oracle6
- 7. Provision a new replication node © 2019 Oracle7 Master Slave Slave 2
- 8. Re-provision a returning node © 2019 Oracle8 BinlogMaster SlaveData ??
- 9. Clone: Operation © 2019 Oracle9
- 10. Terminology © 2019 Oracle10 Donor RecipientData
- 11. Prerequisites on the Donor • Install the plugin: INSTALL PLUGIN clone SONAME 'mysql_clone.so'; • Grant BACKUP_ADMIN to a user account CREATE USER foo …; GRANT BACKUP_ADMIN TO foo; © 2019 Oracle11
- 12. Prerequisites on the Recipient • Install the plugin: INSTALL PLUGIN clone SONAME 'mysql_clone.so'; • Grant CLONE_ADMIN to a user account CREATE USER bar …; GRANT CLONE_ADMIN TO bar; • Allow the donor host SET GLOBAL clone_valid_donor_list='donor_computer:3306'; © 2019 Oracle12
- 13. On the recipient, as “bar” CLONE INSTANCE FROM foo@donor_computer:3306 IDENTIFIED BY 'foo_pwd‘ DATA DIRECTORY = 'clone_dir' REQUIRE SSL © 2019 Oracle13
- 14. Clone: What Has Just Happened ? © 2019 Oracle14
- 15. Clone Stages © 2019 Oracle15 Init Data Files File Copy Modified Page IDs Page Copy Redo Logs Redo Copy Done
- 16. Clone: Network Failure Recovery © 2019 Oracle16 Donor RecipientData Less than 5 min
- 17. Clone Monitoring: performance_schema.clone_status mysql> SELECT STATE FROM performance_schema.clone_status; +-----------+ | STATE | +-----------+ | Completed | +-----------+ © 2019 Oracle17
- 18. Clone Monitoring: performance_schema.clone_progress mysql> SELECT STAGE, STATE, END_TIME FROM performance_schema.clone_progress; +-----------+-----------+----------------------------+ | stage | state | end_time | +-----------+-----------+----------------------------+ | DROP DATA | Completed | 2019-10-27 14:45:43.141261 | | FILE COPY | Completed | 2019-10-27 14:45:44.457572 | | PAGE COPY | Completed | 2019-10-27 14:45:44.577330 | | REDO COPY | Completed | 2019-10-27 14:45:44.679570 | | FILE SYNC | Completed | 2019-10-27 14:45:44.918547 | | RESTART | Completed | 2019-10-27 14:45:48.583565 | | RECOVERY | Completed | 2019-10-27 14:45:49.626595 | +-----------+-----------+----------------------------+ © 2019 Oracle18
- 19. Performance And Resource Use Control Variable Type Meaning clone_max_concurrency Int (Maximum) number of cloning threads clone_autotune_concurrency Bool Enable dynamic spawning of cloning threads clone_max_network_bandwidth Int, Mbps (recipient only) limit network usage clone_max_data_bandwidth Int, Mbps (recipient only) limit clone transfer rate © 2019 Oracle19
- 20. Clone Limitations • Does not clone configuration values • Does not clone binary logs • Supports only InnoDB • Blocks all concurrent DDLs on the Donor • Will FULLY override recipient data • Cloning can be done only to the SAME server version • Doesn’t work over the X protocol © 2019 Oracle20
- 21. Further Reading • Reference Manual • https://dev.mysql.com/doc/refman/8.0/en/clone-plugin.html • Blogs • https://mysqlserverteam.com/clone-create-mysql-instance-replica/ • https://mysqlhighavailability.com/mysql-innodb-cluster-automatic-node-provisioning • https://mysqlhighavailability.com/a-breakthrough-in-usability-automatic-node-provisioning • https://mysqlhighavailability.com/automatic-provisioning-in-group-replication • InnoDB cluster video • https://lefred.be/content/migrate-from-a-single-mysql-instance-to-mysql-innodb-cluster- using-clone-plugin/ © 2019 Oracle21
- 22. Thank you ! Georgi Kodinov, Software Development Director [email protected] © 2019 Oracle22
Editor's Notes
- #9: * The node is far behind and binary log sync would take long time. * Binary logs are not available to sync
- #11: Recipient: Server instance, where the clone statement is executed. Donor: The source server instance from which the data is going to be cloned. The Donor collects all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in Recipient. After successful execution of a CLONE SQL statement the Recipient server is restarted automatically. The restart involves recovery. Once restarted, recipient is a clone of donor server!
- #16: Donor dynamic snapshot is transferred in three stages. File Copy Page Copy Redo Copy
- #17: Recipient: Server instance, where the clone statement is executed. Donor: The source server instance from which the data is going to be cloned. The Donor collects all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in Recipient. After successful execution of a CLONE SQL statement the Recipient server is restarted automatically. The restart involves recovery. Once restarted, recipient is a clone of donor server!
- #18: One or zero rows ID A unique cloning operation identifier in the current MySQL server instance. PID Process list ID of the session executing the cloning operation. STATE Current state of the cloning operation. Values include Not Started, In Progress, Completed, and Failed. BEGIN_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning operation started. END_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning operation finished. Reports NULL if the operation has not ended. SOURCE The donor MySQL server address in 'HOST:PORT' format. The column displays 'LOCAL INSTANCE' for a local cloning operation. DESTINATION The directory being cloned to. ERROR_NO The error number reported for a failed cloning operation. ERROR_MESSAGE The error message string for a failed cloning operation. BINLOG_FILE The name of the binary log file up to which data is cloned. BINLOG_POSITION The binary log file offset up to which data is cloned. GTID_EXECUTED The GTID value for the last cloned transaction.
- #19: ID A unique cloning operation identifier in the current MySQL server instance. STAGE The name of the current cloning stage. Stages include DROP DATA, FILE COPY, PAGE_COPY, REDO_COPY, FILE_SYNC, RESTART, and RECOVERY. STATE The current state of the cloning stage. States include Not Started, In Progress, and Completed. BEGIN_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning stage started. Reports NULL if the stage has not started. END_TIME A timestamp in 'YYYY-MM-DD hh:mm:ss[.fraction]' format that shows when the cloning stage finished. Reports NULL if the stage has not ended. THREADS The number of concurrent threads used in the stage. ESTIMATE The estimated amount of data for the current stage, in bytes. DATA The amount of data transferred in current state, in bytes. NETWORK The amount of network data transferred in the current state, in bytes. DATA_SPEED The current actual speed of data transfer, in bytes per second. This value may differ from the requested maximum data transfer rate defined by clone_max_data_bandwidth. NETWORK_SPEED The current speed of network transfer in bytes per second.
- #20: Clone plugin uses multiple connections from recipient to donor server. - clone_max_concurrency limits maximum connections Clone operation starts with 1 connection thread and threads are increased in steps. Auto tuning checks and stop spawning new connections if no significant performance improvement is noticed. - clone_autotune_concurrency If cpu, i/o, Network resource is saturated then increasing connections/threads could degrade performance and could also impact donor performance. Auto tuning helps here.