如何解决Apache Iceberg 使用 Spark 无法在表中写入数据
我是 Apache Iceberg 的新手。我想使用 Apache Iceberg 执行读写操作。我使用的是 Spark 3.0.0。
代码:
System.setProperty("hadoop.home.dir","C:\\hadoop" )
val conf = new SparkConf()
conf.set("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
conf.set("spark.sql.catalog.spark_catalog","org.apache.iceberg.spark.SparkSessionCatalog")
conf.set("spark.sql.catalog.spark_catalog.type","hive")
conf.set("spark.sql.catalog.local","org.apache.iceberg.spark.SparkCatalog")
conf.set("spark.sql.catalog.local.type","hadoop")
conf.set("spark.sql.catalog.local.warehouse","warehouse")
val spark = SparkSession.builder().master("local").config(conf).getorCreate()
spark.sql("CREATE TABLE local.db.table (id bigint,data string) USING iceberg")
spark.sql("INSERT INTO local.db.table VALUES (1,'a'),(2,'b'),(3,'c')")
依赖列表:
compile group: 'org.scala-lang',name: 'scala-library',version: '2.12.0'
compile group: 'org.apache.spark',name: 'spark-core_2.12',version: '3.0.0'
compile group: 'org.apache.spark',name: 'spark-sql_2.12',version: '3.0.0'
compile group: 'org.apache.iceberg',name: 'iceberg-spark',version: '0.11.1'
compile group: 'org.apache.iceberg',name: 'iceberg-spark3-runtime',version: '0.11.1'
compile group: 'org.apache.hive',name: 'hive-metastore',version: '2.0.1'
异常信息:
java.lang.classCastException: org.apache.iceberg.shaded.org.apache.parquet.schema.MessageType cannot be cast to org.apache.parquet.schema.MessageType
at org.apache.iceberg.parquet.ParquetWriter.<init>(ParquetWriter.java:96)
at org.apache.iceberg.parquet.Parquet$WriteBuilder.build(Parquet.java:250)
at org.apache.iceberg.spark.source.SparkAppenderFactory.newAppender(SparkAppenderFactory.java:110)
我是否需要任何其他依赖项来启用 MessageType 转换。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。