如何解决读取镶木地板文件时合并架构失败,因为数据类型int和bigint不兼容
尝试通过架构合并加载镶木地板文件时
df = spark.read.option("mergeSchema","true").parquet('some_path/partition_date')
df.show()
我遇到以下异常:
Py4JJavaError: An error occurred while calling o421.parquet.
: org.apache.spark.SparkException: Failed merging schema:
root
....
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$mergeSchemasInParallel$1.apply(ParquetFileFormat.scala:643)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$mergeSchemasInParallel$1.apply(ParquetFileFormat.scala:639)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.mergeSchemasInParallel(ParquetFileFormat.scala:639)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.inferSchema(ParquetFileFormat.scala:241)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.getorInferFileFormatSchema(DataSource.scala:179)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:643)
at sun.reflect.GeneratedMethodAccessor137.invoke(UnkNown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Failed to merge fields 'some_field_name' and 'some_field_name'. Failed to merge incompatible data types int and bigint
是否有更好的方法来读取具有某些模式演变(包括不兼容类型)的数据?谢谢
解决方法
您可以通过强制模式和/或使用强制转换来分别加载每个不同的文件。然后,你可以做一个联合。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。