为什么换到了3.4.0了 这个全量的切片逻辑还是修复不好 总是内存溢出

[617450941]

某种情况下 splitEnd直接是Null 导致全部数据查出来 内存溢出 试了下flink 3.0 3.1 3.2 都有这个问题 原本以为3.4能解决 实测效果是好点 比较少出现这个情况 但是还是出现了这种问题

[617450941]

拉的最新的master分支代码实测的 这个bug真的问题太大了 稍微表数据量大点 都没法用

[yuxiqian]

Hi Junwen,

I think a split with splitEnd=null is intended (called "unbounded split"), because we can't determine a certain upper bound of snapshot data. As incremental data during the snapshot phase might increase the size of that split, reading that split at last would easily cause OOM.

Maybe enabling scan.incremental.snapshot.unbounded-chunk-first.enabled option to read unbounded splits first would help? See #3856 for more details.

[617450941]

scan.incremental.snapshot.unbounded-chunk-first.enabled = true 无法解决问题 问题依然存在

[beryllw]

It looks like "scan.incremental.snapshot.unbounded-chunk-first.enabled=true" doesn't work, the unbounded splits are not assign first.

[617450941]

为什么我有的表不会 有的表数据量没那么大 反而也出现这种问题

[617450941]

scan.incremental.snapshot.unbounded-chunk-first.enabled = true 无法解决问题 问题依然存在

[617450941]

是不是我拉取的这张表的id 规则不一样导致触发这个bug呢?

[beryllw]

Yes, this PR might be useful.
https://github.com/apache/flink-cdc/pull/3954/files#diff-481e1aa07355ee1a0d07a42e5b92b28b377ad224b279f48fa77144b5d9f4f1b4