将 Flink DataStream 写入冰山表：NoSuchMethodError: org.apache.parquet.schema.Types$PrimitiveBuilder.as

如何解决将 Flink DataStream 写入冰山表：NoSuchMethodError: org.apache.parquet.schema.Types$PrimitiveBuilder.as

我尝试将flink数据流写入冰山表，如下： '''

val kafkaStream = new KafkaDataSource(parameter,new PacketSchema).getStream(env)
val dataStream = kafkaStream.flatMap(new NullPacketFilter).map(FilteredPacket.from(_).toRow).javaStream

FlinkSink.forRow(dataStream,FilteredPacket.schema)
  .tableLoader(tableLoader)
  .build

然后我得到错误：

2021-02-18 18:12:12,086 WARN  org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-02-18 18:12:12,424 INFO  org.apache.iceberg.BasemetastoreCatalog [] - Table loaded by catalog: iceberg.flink_test.filtered_packets
2021-02-18 18:12:12,477 WARN  org.apache.flink.runtime.taskmanager.Task [] - Source: tianyi -> Flat Map -> Map -> Map -> IcebergStreamWriter (1/1) (9612408d42df7e69b829367434bbc43d) switched from RUNNING to Failed.
java.lang.NoSuchMethodError: org.apache.parquet.schema.Types$PrimitiveBuilder.as(Lorg/apache/parquet/schema/LogicalTypeAnnotation;)Lorg/apache/parquet/schema/Types$Builder;
   at org.apache.iceberg.parquet.TypetoMessageType.primitive(TypetoMessageType.java:145) ~[tianyi112-1.0-SNAPSHOT.jar:?]
   at org.apache.iceberg.parquet.TypetoMessageType.field(TypetoMessageType.java:88) ~[tianyi112-1.0-SNAPSHOT.jar:?]
   at org.apache.iceberg.parquet.TypetoMessageType.convert(TypetoMessageType.java:65) ~[tianyi112-1.0-SNAPSHOT.jar:?]
   at org.apache.iceberg.parquet.ParquetSchemaUtil.convert(ParquetSchemaUtil.java:43) ~[tianyi112-1.0-SNAPSHOT.jar:?]
   ...

我已经检查了 jar 的内容，它包括所需的 Types$PrimitiveBuilder 类：

2651 Fri Feb 19 08:32:10 CST 2021 org/apache/parquet/schema/Types$PrimitiveBuilder.class
3101 Fri Feb 19 08:32:12 CST 2021 org/apache/flink/hive/shaded/parquet/schema/Types$PrimitiveBuilder.clas

当我查看源代码时，发现idea有错误：

库源与 TypetoMessageType 类的字节码不匹配

但所有其他课程都可以。

我尝试删除我的 Maven 存储库中的 iceberg-parquet.jar 和 parquet-column.jar 并重新导入项目，并尝试禁用 Idea 的 Lombok 插件 - 但没有效果。

版本：CDH 6.3.2 Flink 1.11.2 Iceberg 0.11.0

解决方法

我已经解决了这个问题。原因是jar冲突，特别是parquet-hadoop.jar(hive-exec 2.3.4)和iceberg-parquet.jar(0.11.0)