[FLINK-36520] CDC Supports MySQL 8.x and above by yuxiqian · Pull Request #3666 · apache/flink-cdc

yuxiqian · 2024-10-25T01:10:50Z

Currently, trying to use MySQL CDC with MySQL 8.4 would cause the following exception:

Caused by: org.apache.flink.util.FlinkRuntimeException: Read split MySqlSnapshotSplit{tableId=customer.customers, splitId='customer.customers:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} error due to org.apache.flink.util.FlinkRuntimeException: Cannot read the binlog filename and position via 'SHOW MASTER STATUS'. Make sure your server is correctly configured.

The reason is MySQL has changed some questionable words like MASTER and SLAVE to more appropriate ones, so some commands are no longer compatible. Legacy MySqlSource (based on Debezium) wasn't fixed since it's not possible to upgrade Debezium dependency for now, and it has been deprecated for a long time anyway.

Marked this PR as a draft, needs more discussion and testing over this.

leonardBang · 2025-02-18T02:06:13Z

Could you investigate binlog compress feature would been supported via this PR or not? It requires MySQL 8.0.20 above version IIRC.

morozov · 2025-06-05T20:40:06Z

Correct me if I'm wrong, but it looks like using the proper SQL command for showing binary logs is the only major change here. The rest is about refactoring the test suite.

@leonardBang is the support for binlog compression a blocker for these changes to get merged?

[...] needs more discussion and testing over this.

@yuxiqian do you have any specific discussion topics in mind? As for testing, I believe having the test suite pass would be a great start, although it's not clear why the CI didn't run the tests on this PR.

morozov

While this PR is being reviewed, I tried to test it and got a few questions.

morozov · 2025-06-05T17:07:22Z

...ors/flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/MySqlConnection.java

        this.mysqlFieldReader = fieldReader;
        this.urlPattern = connectionConfig.getUrlPattern();
+
+        probeShowBinaryLogStatement();


This method both returns the result of the probe and stores it in a property, which isn't nice from the CQRS standpoint (a method should either return a value or produce side effects, but not both).

Would it make sense to split it into private String probeShowBinaryLogStatement() and public String getShowBinaryLogStatement()? The constructor would call probe() and store the result in a property, and get() would return the property. In this case, the field can be made final, and the if (showBinaryLogStatement != null) condition could be removed.

Alternatively, if we want showBinaryLogStatement to be lazily initialized, the call to probeShowBinaryLogStatement() should be removed from the constructor. The first call will initialize it.

morozov · 2025-06-05T17:10:39Z

...ors/flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/MySqlConnection.java

+                        MYSQL_NEW_SHOW_BINARY_LOG_STATEMENT,
+                        MYSQL_CLASSIC_SHOW_BINARY_LOG_STATEMENT,
+                        e);
+                query(MYSQL_CLASSIC_SHOW_BINARY_LOG_STATEMENT, rs -> {});


If we're going to fall back to this statement anyways, why probe it? The corresponding code in Debezium looks simpler.

morozov · 2025-06-05T19:10:47Z

...ors/flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/MySqlConnection.java

+                        "Probing with {} failed, try {}. Caused by: {}",
+                        MYSQL_NEW_SHOW_BINARY_LOG_STATEMENT,
+                        MYSQL_CLASSIC_SHOW_BINARY_LOG_STATEMENT,
+                        e);


Would it be sufficient to just log the exception message? It isn't an error, so someone looking at the logs may see a stack trace, think that there is some problem, but in fact there is none.

yuxiqian · 2025-06-06T01:47:02Z

Thanks @morozov's kindly review!

@leonardBang is the support for binlog compression a blocker for these changes to get merged?

I did some experiment earlier, and noticed that binlog compression would break current binlog backfilling code. However, considering this feature was introduced in MySQL 8.0.20, perhaps it could be handled independently.

@yuxiqian do you have any specific discussion topics in mind? As for testing, I believe having the test suite pass would be a great start, although it's not clear why the CI didn't run the tests on this PR.

Seems there are a lot of merging conflicts blocking the CI. Will resolve them soon.

sjmudd · 2025-06-20T16:22:13Z

.github/workflows/flink_cdc.yml

      matrix:
        java-version: [ '8' ]
-        module: [ 'core', 'pipeline_connectors', 'mysql', 'postgres', 'oracle', 'mongodb6', 'mongodb7', 'sqlserver', 'tidb', 'oceanbase', 'db2', 'vitess' ]
+        module: [ 'core', 'pipeline_connectors', 'mysql57', 'mysql80', 'mysql84', 'mysql90', 'postgres', 'oracle', 'mongodb6', 'mongodb7', 'sqlserver', 'tidb', 'oceanbase', 'db2', 'vitess' ]


Do not use 'mysql90" as a label. Oracle has horrible versioning and basically you probably need to do one of the following: label as "mysql9x" as 9.X.0 changes every 3 months until it's expected to be 9.7 LTS in April 2026 and then numbers become normal again. e.g. expected to be 9.7.0, 9.7.1, ...

Basically 9.0 should not be used 9.3.0 is current innovation release and it'll be 9.4.0 on ~15th July. So until 9.X is confirmed to be LTS either don't mention it or label it 'mysql9x'.

Reference: https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions/

Clearly do what makes sense but 'mysql90' will never be a useful label.

JeffHu88 · 2025-07-01T03:30:16Z

Hi, could you please let me know when this issue will go live? Thanks!

j4k4 · 2025-08-29T10:25:41Z

A release of this would be very helpful as MySQL 8.0 is nearing it's end of life.
E.g. AWS stops supporting version 8.0 soon.

shengjk · 2025-09-16T06:25:05Z

Hi, I would like to ask why we do not use Debezium directly, but implement it ourselves? Debezium already supports MySQL 8 and above, and Debezium updates much faster than we do.

yuxiqian · 2025-09-16T07:03:59Z

Hi, I would like to ask why we do not use Debezium directly, but implement it ourselves? Debezium already supports MySQL 8 and above, and Debezium updates much faster than we do.

Thanks for @shengjk's nice suggestion! For a long time we've been stuck at Debezium 1.9.x and monkey-patching Debezium, mainly for keeping compatibility with Java 8.

FLINK-36605 also suggests a similar idea, but it is more likely to happen after bumping to Flink 2.x and dropping Java 8 support.

zhangshenghang · 2025-10-09T07:46:13Z

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

yuxiqian · 2025-10-09T09:06:00Z

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

Thanks @zhangshenghang for tracing this issue!

IIRC, one concern from maintainers is we can't provide decent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.

The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.

Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.

zhangshenghang · 2025-10-09T09:10:08Z

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

Thanks @zhangshenghang for tracing this issue!

IIRC, one concern from maintainers is we can't provide descent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.

The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.

Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.

@yuxiqian Thank you for your reply. Yes, upgrading to JDK 11 would be a significant change. If there could be a temporary transitional solution for 1.8, that would also be a good option.

Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com> # Conflicts: # .github/workflows/flink_cdc_base.yml # .github/workflows/flink_cdc_ci.yml # flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java # flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlMetadataAccessorITCase.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/io/debezium/connector/mysql/util/ErrorMessageUtilsTest.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/LegacyMySqlTestBase.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/debezium/reader/BinlogSplitReaderTest.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/table/MySqlCompatibilityITCase.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/table/MySqlConnectorITCase.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/table/MySqlTimezoneITCase.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/testutils/MySqlVersion.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/polardbx/PolardbxSourceTestBase.java # Conflicts: # .github/workflows/flink_cdc_base.yml # .github/workflows/flink_cdc_ci.yml # .github/workflows/flink_cdc_ci_nightly.yml # flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java # flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/debezium/reader/BinlogSplitReaderTest.java

zhangshenghang · 2025-10-09T10:45:56Z

@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?

Thanks @zhangshenghang for tracing this issue!
IIRC, one concern from maintainers is we can't provide descent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.
The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.
Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.

@yuxiqian Thank you for your reply. Yes, upgrading to JDK 11 would be a significant change. If there could be a temporary transitional solution for 1.8, that would also be a good option.

@yuxiqian Hi，I saw a discussion here about upgrading Debezium to version 2.7.x: https://issues.apache.org/jira/browse/FLINK-36605.
However, it seems that version 2.7.x also does not support MySQL 8.4.
MySQL 8.4 supported starting from version 3.2

yuxiqian · 2025-10-09T10:56:56Z

Thanks @zhangshenghang for confirming this! Debezium 3.2 requiring Java 17 seems not very likely, since Flink 2.x still supports Java 11 (deprecated though). I'll fix CI test cases first.

Romeoy · 2026-01-12T03:21:36Z

Hi @yuxiqian , much appreciate! Is there any update for the plan to support mysql 8.4? As mysql 8.0 will not been matained soon for many cloud service providers, I believe many companies will met this issue as well. By the way, I saw another PR to resolve same issue #4196 .

sridhard · 2026-01-13T15:39:47Z

Hi,
When this will be released?

github-actions bot added build mysql-cdc-connector base e2e-tests mysql-pipeline-connector labels Oct 25, 2024

yuxiqian force-pushed the FLINK-36590 branch from d957e4f to b57880b Compare October 25, 2024 01:15

yuxiqian marked this pull request as ready for review October 29, 2024 11:21

leonardBang requested a review from ruanhang1993 November 8, 2024 08:22

yuxiqian force-pushed the FLINK-36590 branch from 3087c70 to 17403e2 Compare November 26, 2024 07:19

morozov reviewed Jun 5, 2025

View reviewed changes

yuxiqian marked this pull request as draft June 6, 2025 01:47

sjmudd reviewed Jun 20, 2025

View reviewed changes

melin mentioned this pull request Sep 9, 2025

support mysql 8.5 risingwavelabs/risingwave#23115

Closed

yuxiqian force-pushed the FLINK-36590 branch from 17403e2 to d9978de Compare October 9, 2025 10:29

github-actions bot added the debezium label Oct 9, 2025

Conversation

yuxiqian commented Oct 25, 2024

Uh oh!

leonardBang commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

morozov commented Jun 5, 2025

Uh oh!

morozov left a comment

Choose a reason for hiding this comment

Uh oh!

morozov Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morozov Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

morozov Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

yuxiqian commented Jun 6, 2025

Uh oh!

sjmudd Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

JeffHu88 commented Jul 1, 2025

Uh oh!

j4k4 commented Aug 29, 2025

Uh oh!

shengjk commented Sep 16, 2025

Uh oh!

yuxiqian commented Sep 16, 2025

Uh oh!

zhangshenghang commented Oct 9, 2025

Uh oh!

yuxiqian commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangshenghang commented Oct 9, 2025

Uh oh!

zhangshenghang commented Oct 9, 2025

Uh oh!

yuxiqian commented Oct 9, 2025

Uh oh!

Romeoy commented Jan 12, 2026

Uh oh!

sridhard commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

leonardBang commented Feb 18, 2025 •

edited

Loading

morozov Jun 5, 2025 •

edited

Loading

yuxiqian commented Oct 9, 2025 •

edited

Loading