Flink1.20.1集成Paimon遇到的问题

发布于:2025-07-10 ⋅ 阅读:(18) ⋅ 点赞:(0)

flinkcdc mysql 到paimon

1:Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schema

可以参考这个文章 明确指出了flink-connector-mysql-cdc-3.4.0.jar存在这个包,但是flink-sql-connector-mysql-cdc-3.4.0.jar中没有这个包 可以在maven仓库中看到
在这里插入图片描述

https://developer.aliyun.com/ask/574255?spm=a2c6h.12873639.article-detail.8.6a82fe85SbDdj5

缺少 kafka的 connect-api-3.3.2.jar

Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/connect/data/Schema
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1575)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:508)
at java.io.ObjectStreamClass2.run(ObjectStreamClass.java:482)atjava.security.AccessController.doPrivileged(NativeMethod)atjava.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)atjava.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)atorg.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)atorg.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)atorg.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambda2.run(ObjectStreamClass.java:482) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502) at org.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62) at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambda2.run(ObjectStreamClass.java:482)atjava.security.AccessController.doPrivileged(NativeMethod)atjava.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)atjava.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)atorg.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)atorg.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)atorg.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambdacreateJobVertex24(StreamingJobGraphGenerator.java:1021)atjava.util.concurrent.CompletableFuture24(StreamingJobGraphGenerator.java:1021) at java.util.concurrent.CompletableFuture24(StreamingJobGraphGenerator.java:1021)atjava.util.concurrent.CompletableFutureAsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:624)...1moreCausedby:java.lang.ClassNotFoundException:org.apache.kafka.connect.data.Schemaatjava.net.URLClassLoader.findClass(URLClassLoader.java:381)atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)atsun.misc.LauncherWorker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schema at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.LauncherWorker.run(ThreadPoolExecutor.java:624)...1moreCausedby:java.lang.ClassNotFoundException:org.apache.kafka.connect.data.Schemaatjava.net.URLClassLoader.findClass(URLClassLoader.java:381)atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)atsun.misc.LauncherAppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 45 more

2:Caused by: java.io.InvalidClassException: org.apache.flink.cdc.connectors.shaded.com.fasterxml.jackson.databind.cfg.MapperConfig; incompatible types for field _mapperFeatures

不能同时存在
flink-connector-mysql-cdc-3.4.0.jar
flink-sql-connector-mysql-cdc-3.4.0.jar

3:cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.userFunction of type org.apache.flink.api.common.functions.Function in instance of org.apache.flink.streaming.api.operators.StreamFlatMap

dinky standalone 提mysqlCDC整库同步到paimon
将dinky-app的依赖丢进flink lib中就可以了

025-07-10 10:50:36,319 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: MySQL CDC Source -> PartitionByPrimaryKey -> Shunt -> FlatMapRow -> *anonymous_datastream_source$6*[22] -> Calc[23] -> ConstraintEnforcer[24] -> Map (1/2)#24 (2548d4605e02b73a83d284ff33f2e106_cbc357ccb763df2852fee8c4fc7d55f2_0_24) switched from INITIALIZING to FAILED with failure cause:
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot instantiate user function.
	at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:416) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:869) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:836) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:825) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:825) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:202) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.<init>(RegularOperatorChain.java:60) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:789) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939) [flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist-1.20.1.jar:1.20.1]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.userFunction of type org.apache.flink.api.common.functions.Function in instance of org.apache.flink.streaming.api.operators.StreamFlatMap
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233) ~[?:1.8.0_171]
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2291) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) ~[?:1.8.0_171]
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_171]
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:488) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:472) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:467) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:422) ~[flink-dist-1.20.1.jar:1.20.1]
	at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:400) ~[flink-dist-1.20.1.jar:1.20.1]
	... 16 more

网站公告

今日签到

点亮在社区的每一天
去签到