Migrating with Debezium

Editor's Notes

• #3: Senior Site Reliability Engineer in the Public Cloud Practice Background in Software & Systems Engineering, System & Database Administration Using PostgreSQL since version 7.4 Contributed to PostgreSQL, Terraform & YAWL Been involved with flossUK, formerly UKUUG for many years first reviewing books and then as a speaker Work exclusively on public clouds, hence the collection of Public Cloud certifications
• #4: Image Source: Author’s personal collection “As you can see from my background, I like to think that I know Data”
• #5: Image Source: https://wall.alphacoders.com/big.php?i=789452 Identify what requires migration Database schemas Client applications Analyse for PostgreSQL compatibility In each schema we need to convert Storage Objects Tables, views, data types, etc. Code Objects Packages, procedures & functions Most projects are qualified out at this stage
• #6: Image Source: https://www.coventrytelegraph.net/news/uk-world-news/big-ben-scaffolding-14826151 Determine what needs to be migrated Whole schemas may no longer be required Some tables could be archived out of the database Extract tables to a data interchange format and store in Google Cloud Storage or S3 Some packages/procedures/functions could be removed Rewrite stored procedures and functions to PL/PgSQL or equivalent Test Driven Compatibility Development Existing bugs may be identified Preserving them may be necessary, and hard Fixes should be retrofitted so that switchover day surprises don’t occur
• #7: Image Source: https://www.amazon.com/Battlestar-Galactica-Fleet-Poster-2ftx3ft/dp/B005E9JWTW Migration of test database will commence with the refactoring Final migration of the production database occurs after: All refactorings are complete The migration has been performed multiple times The applications have been tested against a pre-migrated production database
• #8: Two primary approaches Extract,Transform & Load (also called Dump & Restore) Change Data Capture & Replay Picking on MySQL but MongoDB and PostgreSQL itself can also be sources Oracle & SQL Server are in development
• #9: Simplest strategy Perceived as low risk for data loss Less “moving parts” Easy to test repeatedly and “prove” migration Downtime is function of database size Downtime can be minimised Move historic data ahead of time Opportunity to clear out unused data Consider introducing partitions Over provision resources Gives higher throughput during data load Downscale once operational
• #10: Usually a complicated option that requires connected two unrelated technologies The same ETL as the CSV approach needs to be performed, but continually and reliably Failures often require a complete restart of the process to ensure no data loss Numerous commercial offerings but few open source Enter Kafka Append only log journaling system with messaging semantics Entire history can be maintained if you’ve the disk space Consumers maintain their position allowing for data replay in the event of failure
• #11: Kafka Connect is an extensible framework for producers and consumers Debezium is a Kafka Connect source Configured to read from MySQL and write to Kafka Each table becomes it’s own topic with a dedicated topic for schema changes The data and schema can be recorded in Kafka in Avro format using open source tools from Confluent Kafka Connect JDBC is both a Kafka Connect source and sink, we’ll use it to sink data to PostgreSQL We need to create one connector for each table topic to write to PostgreSQL Able to read the schema data and create a corresponding table to host the incoming data
• #12: All of this software is open source however configuring and building the Confluent tools is involved Working demonstration using Vagrant is available in GitHub (github.com/gh-mlfowler/debezium-demo) We’ll walk through the interesting pieces of configuration to give an idea of what is involved and how easy it is to get data moving
• #13: The MySQL installation is the basic default available in Ubuntu A database and user is created, all using the ‘sakila’ identifier The Sakila (https://dev.mysql.com/doc/sakila/en/) sample database is loaded Binary logging is not enabled so there is no change data capture available for this data
• #14: Now the data is loaded we enable binary logging We need a unique server-id to identify this server, we’ll need another one later MySQL is restarted and any new changes are now recorded in the binary log
• #15: Again, vanilla PostgreSQL as found in Ubuntu Just a database and a user which owns the database No schema is loaded, we just leave the empty database to be populated by JDBC Connect later
• #16: Each Kafka connector is defined by a JSON document passed to a web service Items highlighted in red are what we likely need to change for each new connector We need a unique database.server.id so Debezium can identify itself to the MySQL instance A whitelist allows us to pick only the database or databases of interest
• #17: A POST with the JSON document as the body will create the connector and we can remove it later with a DELETE Once the connector is created, it will interrogate MySQL to extract the schema This requires a brief database wide lock but the timing is short and logged - check this for your planning! All data older than the current binary log is read and transferred to Kafka A topic is created per table A topic recording schema changes is created A second history table is created and is really for internal use
• #18: We can verify this by browsing our Kafka box and listing the topics The topic names follow the pattern “server-name.database.table” Server name is “mysql” in this example. This topic has the schema changes
• #19: At this stage all existing data is resting in topics in Kafka New changes to MySQL will be appended to Kafka shortly after they occur DDL changes to the server name topic Data changes to the relevant table topic Although PostgreSQL is running, no data has been sent there yet
• #20: Configuring the Kafka Connect JDBC sink is similar to Debezium Create one for each table we wish to sink or read multiple in a single connector In this example we’ll consume only the actor table If the table name isn’t specified the topic name will be used which PostgreSQL will reject
• #21: Loading the JDBC connector is identical, just a POST to the Kafka Connect web service
• #22: All finished, end to end replication of data in near real time from MySQL to PostgreSQL We can create many replicas simply by reading from the topics One off read from the source database as long as Kafka is large enough to store all the data
• #23: Image Source: https://eguidemagazine.com/theres-no-such-thing-as-a-free-lunch/ With the simplicity of set-up we lose some things Constraints are not copied across These can be added at the time of the final switchover however you’ll need to time this to measure the impact Data types are converted to the lowest common denominator You can pre-create the schema to hand nominate the conversion Does not deal with stored procedures There are some tools for this, but this isn’t the tool for it