PolarDB:Release notes for PolarDB-X CDC nodes

Category

Description

New feature

Supports filtering delete events generated by archived tables when CDC nodes create binary logs.

Category

Description

New feature

The multi-stream binlogs feature of CDC nodes allows each data node to process one stream of binary logs.

Feature optimization

The memory management is optimized when the system performs dump operations on binary logs.

The performance of CDC nodes in transmitting binary logs to downstream systems is improved.

The SQL queries sent from CDC nodes to global meta service (GMS) are optimized to reduce the load on GMS.

Category

Description

New feature

One binary log stream per data node is supported.

Category

Description

New feature and feature optimization

CHECK constraints are supported. For more information, see CHECK constraints.

The allocation policy of CDC off-heap direct memory is optimized to avoid out-of-memory (OOM) issues caused by insufficient direct memory allocation.

The forced restart mechanism of CDC is optimized to support forced restart when the CDC worker process is blocked.

Category

Description

Feature optimization

The default size of the binary log file is updated to 1 GB. When the binary log file number exceeds 999999, the binary log file can be automatically rotated.

The download of the transparent consumption file for the binary logs is optimized.

Category

Description

Feature optimization

The replication logic of the primary and secondary nodes is optimized.

The heartbeat timeout detection logic of the CDC high-availability module is optimized to avoid unnecessary process restarts.

Fixed issue

The following issue is fixed: The update of an earlier version of CDC is interrupted due to the lack of the RuntimeVersion field in the system table.

The following issue is fixed: The WITH PARSER NGRAM attribute is not included in the results of SHOW CREATE TABLE statement.

Category

Description

Fixes

The following issue is fixed: CDC links are interrupted due to compatibility differences when REAL and DOUBLE data types in CDC metadata are processed.

Category

Description

Fixes

The following issue is fixed: When a single SQL statement is executed to update multiple tables, consecutive TableMapEvents may occur, resulting in disordered trace IDs.

Category

Description

Fixes

The following issue is fixed: The CDC process cannot be started because a single compute node is unavailable.

The following issue is fixed: Dump operations fail on binary logs because the offset values in the memory are inconsistent with those in binary logs.

Category

Description

Fixed issue

The following issue is fixed: Information returned from the SHOW FULL BINARY LOGS statement is incomplete.

The following issue is fixed: A binary log file is lost due to concurrent uploads of the file during the CDC topology reconstruction process.

The following issues are fixed: The ALTER statements that contain implicit table groups fail to be parsed by the SQL parser or fail to be correctly processed.

The following issue is fixed: The system automatically removes local files on the master node during the primary-secondary synchronization process using a CDC dumper. As a result, the replication links cannot be automatically restored.

Category

Description

New feature

Sync points are periodically created on CDC nodes.

Fixed issue

The following issue is fixed: The binary log cursor fails to advance through binary logs as expected, which leads to hindered binary log consumption.

Category

Description

New feature and feature optimization

Sync Point is periodically created on CDC nodes.

The full data verification capability used to evaluate the upgrade of a PolarDB-X 1.0 instance to a PolarDB-X 2.0 instance is optimized.

Fixed issue

The following issue is fixed: Global binary logs fail to be generated due to metadata changes.

The following issue is fixed: The binary log dump service cannot be provided because the destination binary log file is empty.

Category

Description

New feature and feature optimization

Functions can be replicated between master and replica nodes using DDL operations.

Sequences can be replicated between master and replica nodes using DDL operations.

Views can be replicated between master and replica nodes using DDL operations.

Procedures can be replicated between master and replica nodes using DDL operations.

Accounts, roles, and permissions can be replicated between master and replica nodes using DDL operations.

ALTER TABLEGROUP operations can be replicated between master and replica nodes using DDL operations.

ALTER INDEX operations can be replicated between master and replica nodes using DDL operations.

Consistent and aligned replication can be ensured when multi-stream binlogs are used during master-replica replication using DDL operations.

Replication links can be created by timestamp in master-replica replication.

Consistency verification of upstream and downstream data using SQL statements is supported in master-replica replication.

The secondary instance role is added. An instance to which the role is assigned can be read-only at the instance level.

WHERE clauses in DML statements can be fully replicated from the primary instance to the secondary instance.

Multiple binary log streams can be associated with specific account names. When you use the associated account to execute binary log-related SQL statements, you do not need to add the with option.

Server_id-based two-way replication between the primary instance and the secondary instance is optimized. Filter rules can be specified using the CLI and anomalies with server_id can be automatically detected and reported.

Fixed issue

The following issue is fixed: The column order of metadata is incorrect after multiple ALTER ADD COLUMN or ALTER DROP COLUMN statements are executed, which leads to interrupted binary log links.

The following issue is fixed: Metadata indexes are lost after multiple ALTER ADD INDEX, ALTER DROP INDEX, and ALTER RENAME INDEX statements are executed, which causes incorrect DDL statements to be recorded in binary logs.

The following issue is fixed: DDL is filtered when the ALTER DDL SQL statement contains the 'user' keyword due to a configuration defect in the meta_build_physical_ddl_sql_blacklist_regex parameter.

The following issue is fixed: DROP INDEX DDL statements cannot be output to binary logs.

Category

Description

New feature

Binary log-related statements such as show full binary logs and show full master status are added.

Fixed issue

The following issue is fixed: SQL flashback exact match fails.

Category

Description

New feature

The show full master status statement is added.

Feature optimization

The MySQL binary log dump process can use parameters such as checksum and heartbeat to optimize data integrity and replication monitoring.

Category

Description

New feature

SQL flashback is supported and can roll back a transaction.

Fixed issue

The method used to record transaction sequence object (TSO) heartbeat entries in binary logs is modified. This eliminates the associated downstream crash issue.

Category

Description

Feature optimization

The gRPC backpressure sleep duration is optimized. This increases the binary log dump speed to 130 MB/s.

Fixed issue

The following issue is fixed: The SQL flashback process fails to complete due to the absence of DML or DDL operations within the specified time range.

Category

Description

Feature optimization

The query efficiency of the SHOW BINARY LOGS statement in scenarios that involve large numbers of binary logs is optimized. The slow query issue is also fixed.

Fixed issue

The following issue is fixed: A buffer overflow occurs when you perform data shaping on JSON data.

The following issue is fixed: During the upgrade of the change data capture (CDC) feature, the metaDataSource scan thread exits due to high-availability (HA) switchover in the metadatabase.

The following issue is fixed: Deadlocks occur when binlog_dumper_info is updated.

Category

Description

New feature and feature optimization

The FLUSH LOGS statement is supported.

The data dump efficiency of the transparent consumption feature is optimized.

CDC nodes do not need to obtain the connection string of the data node from the is_vip parameter.

CDC nodes do not need to obtain the connection string of the metadata database from the control module.

Fixed issue

The following issue is fixed: Errors occur if the table creation statement and parsing syntax contain the BIGINT SIGNED column definition.

The following issue is fixed: Holes exist in the binary log list when the syntax for generating transaction sequence objects (TSOs) is restored.

The following issue is fixed: After the read-only instance is scaled, task dump cannot locate the replication start point on the data node of the primary instance.

Category

Description

New feature

Transaction objects can be saved to persistent medium to reduce memory usage.

Fixed issue

The following issue is fixed: The number of TSOs to be restored based on previous TSOs must be greater than or equal to the number of TSOs of the transactions that are most recently tagged with scaling.

The following issue is fixed: When data is synchronized from the primary instance to a secondary instance, the gRPC thread cannot exit because local files generated by Dumper Master are deleted.

Category

Description

Fixed issue

The following issue is fixed: When you consume binary logs from Object Storage Service (OSS), a conflict in the timing of sending heartbeat packets may occur and lead to a link failure.

The following issue is fixed: Errors in data type conversion may occur when you perform reformatting on TableMapEvent.

The following issue is fixed: The Sql_Mode value recorded in a DDL event may differ from the original Sql_Mode value.

The following issue is fixed: Inconsistencies in metadata exist between logical and physical tables when NCHAR and NVARCHAR data types are used.

The following issue is fixed: When a DDL SQL statement contains line breaks, converting the statement into a single-line comment may not work as expected.

Category

Description

New feature and feature optimization

The monitoring and statistics on the CPU utilization of worker processes are added.

Fine-grained monitoring of EPS (INSERT, UPDATE, and DELETE) is supported.

The upload speed of binary log files to OSS is improved to 700 MB/s.

The CDC O&M statement (START MASTER, STOP MASTER, RESTART MASTER, and RESET MASTER) are added.

Fixed issue

The following issue is fixed: Blocks occur after a long period when offline binary logs are transparently consumed.

The following issue is fixed: The dump thread cannot exit when binary logs are deleted during data reads.

The following issue is fixed: The RESET MASTER statement fails when a large amount of metadata is involved in DDL physical duplication.

The following issue is fixed: The sql_mode value in the DDL event is inconsistent with the original sql_mode value.

The following issue is fixed: When an SQL statement contains both add primary key and drop primary key, an error occurs in the MySQL database.

Category

Description

Fixed issue

The following issue is fixed: The column order is incorrect after an SQL statement is executed in the SchemaRepository.

The following issue is fixed: A null pointer exception (NPE) is triggered when an empty string is reshaped.

Category

Description

Fixed issue

The following issue is fixed: Data is truncated because the length of metadata information is greater than the length of the data of the mediumtext data type.

Category

Description

Fixed issue

The following issue is fixed: Data fails to be parsed because the character sets of MySQL are not converted to the character sets of Java.

Category

Description

Feature optimization

The recovery time objective (RTO) for binary log synchronization is reduced to 15 seconds.

The speed for restoring CDC metadata when hundreds of thousands of tables are involved is optimized.

Fixed issue

The following issue is fixed: An error may occur in the calculation of the binary log file position when binary logs are directly consumed.

The following issue is fixed: Compatibility errors occur when metadata consistency is checked in scenarios in which text(0) is used.

Category

Description

Fixed issue

The following issue is fixed: The CDC link is interrupted because the primary keys are continuously created and deleted.

The following issue is fixed: CDC closes the connection because a consumer program does not send the COM_REGISTER_SLAVE command to CDC before the program dumps the binary log files.

The following issue is fixed: The data link is interrupted when the hash value is Integer.MIN_VALUE in a multi-stream routing scenario.

Category

Description

New feature and feature optimization

The multi-level merging feature is supported. Data nodes can linearly increase without significant performance degradation.

Transparent consumption is supported. The downstream consumer program can use the binary log dump protocol to directly consume binary log files that are archived in OSS.

Multi-stream binary logs are supported. You can customize the number of streams. Each stream of binary logs is independent.

Scheduled snapshots are supported for the full metadata. This helps resolve the excessive expansion of labeling records for historical DDL operations.

Binary logs can be concurrently downloaded from OSS, and the amount of time required to restore the PolarDB-X instance is decreased.

Fixed issue

The following issue is fixed: Compatibility errors and incorrect results may be returned when the event data in binary logs is reshaped. This improves the stability of CDC links during DDL operations.

The following issue is fixed: A timeout may occur when the downstream consumer program dumps binary logs after a CDC cluster is restarted.

The following issue is fixed: The estimated latency is incorrect when the Dumper process is started for the first time.