FlinkCDC 有没有类似Maxwell bootstrap全量同步表的功能 #1102
Unanswered
guojiel-mega
asked this question in
Q&A
Replies: 3 comments
-
|
help! |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
sure, please refer https://ververica.github.io/flink-cdc-connectors/release-2.2/content/connectors/mysql-cdc.html |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
请问 maxwell可能丢失数据的问题你们是怎么解决的呢？期待回复 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
如题，最近在搭建离线数仓，使用Maxwell接入正式服的MySQL进行业务表的增量同步，由于该MySQL是阿里云做了一层代理的，host是192.168.x.x，端口号改为了32306，结果maxwell读取binlog正常，但是使用bootstrap时会报错，报错信息如下：
`[root@bigdata101 scripts]# ./mysql_to_kafka_inc_init.sh auth_user
connecting to jdbc:mysql://192.168.2.170:32306/maxwell?allowPublicKeyRetrieval=true&connectTimeout=5000&zeroDateTimeBehavior=convertToNull
71125 [WARN] BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@fd9e3cd -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.27-bin.jar:?]
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) [c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) [mchange-commons-java-0.2.19.jar:0.2.19]
Caused by: java.net.UnknownHostException: null
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308) ~[mysql-connector-java-5.1.27-bin.jar:?]
... 20 more
71125 [WARN] BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3789aa0c -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.27-bin.jar:?]
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) [c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) [mchange-commons-java-0.2.19.jar:0.2.19]
Caused by: java.net.UnknownHostException: null: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_212]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_212]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308) ~[mysql-connector-java-5.1.27-bin.jar:?]
... 20 more
71125 [WARN] BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@2a1e8035 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.27-bin.jar:?]
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) [c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) [mchange-commons-java-0.2.19.jar:0.2.19]
Caused by: java.net.UnknownHostException: null
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308) ~[mysql-connector-java-5.1.27-bin.jar:?]
... 20 more
71151 [WARN] BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@7e07db1f is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
71152 [WARN] BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@7e07db1f is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
71152 [WARN] BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@7e07db1f is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
71156 [ERROR] MaxwellBootstrapUtility: failed to connect to mysql server @ jdbc:mysql://192.168.2.170:32306/maxwell?allowPublicKeyRetrieval=true&connectTimeout=5000&zeroDateTimeBehavior=convertToNull
71156 [ERROR] MaxwellBootstrapUtility: Connections could not be acquired from the underlying database!
java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at com.zendesk.maxwell.util.C3P0ConnectionPool.getConnection(C3P0ConnectionPool.java:18)
at com.zendesk.maxwell.bootstrap.MaxwellBootstrapUtility.run(MaxwellBootstrapUtility.java:40)
at com.zendesk.maxwell.bootstrap.MaxwellBootstrapUtility.main(MaxwellBootstrapUtility.java:253)
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 4 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311)
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.net.UnknownHostException: null
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249)
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308)
... 20 more
`
不知如何解决，想看看FlinkCDC有没有比较好的解决方法，感谢提供思路的大佬们
（使用maxwell连接本地搭建的MySQL时bootstrap和获取binlog一切正常）
Beta Was this translation helpful? Give feedback.
All reactions