将pyspark中的嵌套数据框展平为列

分类：编程问答

如何解决将pyspark中的嵌套数据框展平为列

嗨，我有pyspark中提取的JSON数据，示例如下。

{
    "data": [
        ["row-r9pv-p86t.ifsp","00000000-0000-0000-0838-60C2FFCC43AE",1574264158,null,"{ }","2007","ZOEY","KINGS","F","11"],["row-7v2v~88z5-44se","00000000-0000-0000-C8FC-DDD3F9A72DFF","SUFFOLK","6"],["row-hzc9-4kvv~mbc9","00000000-0000-0000-562E-D9A0792557FC","MONROE","6"]
    ]
}

我正在尝试分解多数组并将每个记录分解为数据帧的单行，但看起来像这样：

df= spark.read.json('data/rows.json',multiLine=True)
temp_df = df.select(explode("data").alias("data"))
temp_df.show(n=3,truncate=False)

结果：

+-----------------------------------------------------------------------------------------------------------------------+
|data                                                                                                                   |
+-----------------------------------------------------------------------------------------------------------------------+
|[row-r9pv-p86t.ifsp,00000000-0000-0000-0838-60C2FFCC43AE,{ },2007,ZOEY,KINGS,F,11] |
|[row-7v2v~88z5-44se,00000000-0000-0000-C8FC-DDD3F9A72DFF,SUFFOLK,6]|
|[row-hzc9-4kvv~mbc9,00000000-0000-0000-562E-D9A0792557FC,MONROE,6] |
+-----------------------------------------------------------------------------------------------------------------------+

temp_df.printSchema()
temp_df.show(5)
temp_df.select(flatten(temp_df.data)).show(n=10)

到目前为止还不错，但是当我尝试使用flatten方法将数据帧的每一行中的数组展平时，它给了我错误提示 cannot resolve 'flatten('data')' due to data type mismatch: The argument should be an array of arrays,but 'data' is of array<string> type.很有道理，但我不确定如何才能展平数组。

我应该编写任何自定义map方法将行数组映射到数据框列吗？

HERE IS THE SCHEMA OF DATAFRAME

解决方法

回答我自己的问题。这样可以帮助任何需要的人。

  1. 从文件读取源数据
df= spark.read.json('data/rows.json',multiLine=True)
temp_df = df.select(explode("data").alias("data"))
temp_df.show(n=3,truncate=False)

结果：

+-----------------------------------------------------------------------------------------------------------------------+
|data                                                                                                                   |
+-----------------------------------------------------------------------------------------------------------------------+
|[row-r9pv-p86t.ifsp,00000000-0000-0000-0838-60C2FFCC43AE,1574264158,{ },2007,ZOEY,KINGS,F,11] |
|[row-7v2v~88z5-44se,00000000-0000-0000-C8FC-DDD3F9A72DFF,SUFFOLK,6]|
|[row-hzc9-4kvv~mbc9,00000000-0000-0000-562E-D9A0792557FC,MONROE,6] |
+-----------------------------------------------------------------------------------------------------------------------+

在上面的数据框中，每个单元格包含一个字符串数组，而我需要的是每个元素都在单独的列中和特定的数据类型。

df = temp_df.withColumn("sid",temp_df["data"].getItem(0).cast(StringType())) \
       .withColumn("id",temp_df["data"].getItem(1).cast(IntegerType())) \
       .withColumn("position",temp_df["data"].getItem(2).cast(IntegerType())) \
       .withColumn("created_at",temp_df["data"].getItem(3).cast(TimestampType())) \
       .withColumn("created_meta",temp_df["data"].getItem(4).cast(StringType())) \
       .withColumn("updated_at",temp_df["data"].getItem(5).cast(TimestampType())) \
       .withColumn("updated_meta",temp_df["data"].getItem(6).cast(StringType())) \
       .withColumn("meta",temp_df["data"].getItem(7).cast(StringType())) \
       .withColumn("Year",(temp_df["data"].getItem(8)).cast(IntegerType())) \
       .withColumn("First Name",temp_df["data"].getItem(9).cast(StringType())) \
       .withColumn("County",temp_df["data"].getItem(10).cast(StringType())) \
       .withColumn("Sex",temp_df["data"].getItem(11).cast(StringType())) \
       .withColumn("Count",temp_df["data"].getItem(12).cast(IntegerType())) \
       .drop("data")
df.show()
df.printSchema()

+------------------+----+--------+----------+------------+----------+------------+----+----+----------+-------+---+-----+
|               sid|  id|position|created_at|created_meta|updated_at|updated_meta|meta|Year|First Name| County|Sex|Count|
+------------------+----+--------+----------+------------+----------+------------+----+----+----------+-------+---+-----+
|row-r9pv-p86t.ifsp|null|       0|      null|        null|      null|        null| { }|2007|      ZOEY|  KINGS|  F|   11|
|row-7v2v~88z5-44se|null|       0|      null|        null|      null|        null| { }|2007|      ZOEY|SUFFOLK|  F|    6|
|row-hzc9-4kvv~mbc9|null|       0|      null|        null|      null|        null| { }|2007|      ZOEY| MONROE|  F|    6|
+------------------+----+--------+----------+------------+----------+------------+----+----+----------+-------+---+-----+

==================== SCHEMA ====================

root
 |-- sid: string (nullable = true)
 |-- id: integer (nullable = true)
 |-- position: integer (nullable = true)
 |-- created_at: timestamp (nullable = true)
 |-- created_meta: string (nullable = true)
 |-- updated_at: timestamp (nullable = true)
 |-- updated_meta: string (nullable = true)
 |-- meta: string (nullable = true)
 |-- Year: integer (nullable = true)
 |-- First Name: string (nullable = true)
 |-- County: string (nullable = true)
 |-- Sex: string (nullable = true)
 |-- Count: integer (nullable = true)
, 
val resDF = temp_df.select(
  'data.getItem(0).alias("c0"),'data.getItem(1).alias("c1"),'data.getItem(2).alias("c2"),'data.getItem(3).alias("c3")
  // ...
)
resDF.show(false)
//  +------------------+------------------------------------+---+----------+
//  |c0                |c1                                  |c2 |c3        |
//  +------------------+------------------------------------+---+----------+
//  |row-r9pv-p86t.ifsp|00000000-0000-0000-0838-60C2FFCC43AE|0  |1574264158|
//  |row-7v2v~88z5-44se|00000000-0000-0000-C8FC-DDD3F9A72DFF|0  |1574264158|
//  |row-hzc9-4kvv~mbc9|00000000-0000-0000-562E-D9A0792557FC|0  |1574264158|
//  +------------------+------------------------------------+---+----------+

V 2（使用WithColumn和concat_ws）：

 val sourceDF = Seq(
    Array("row-r9pv-p86t.ifsp","00000000-0000-0000-0838-60C2FFCC43AE","0","1574264158","","{ }","2007","ZOEY","KINGS","F","11"),Array("row-7v2v~88z5-44se","00000000-0000-0000-C8FC-DDD3F9A72DFF","SUFFOLK","6"),Array("row-hzc9-4kvv~mbc9","00000000-0000-0000-562E-D9A0792557FC","MONROE","6")
  ).toDF("dataColumn")

  sourceDF.show(false)

//  +-------------------------------------------------------------------------------------------------------------------------+
//  |dataColumn                                                                                                               |
//  +-------------------------------------------------------------------------------------------------------------------------+
//  |[row-r9pv-p86t.ifsp,11] |
//  |[row-7v2v~88z5-44se,6]|
//  |[row-hzc9-4kvv~mbc9,6] |
//  +-------------------------------------------------------------------------------------------------------------------------+


  val df1 = sourceDF
    .withColumn("dataString",concat_ws(",",'dataColumn))
    .select('dataString)

  df1.printSchema()

  df1.show(false)
//  root
//  |-- dataString: string (nullable = false)
//
//  +-----------------------------------------------------------------------------------------------------------------------+
//  |dataString                                                                                                             |
//  +-----------------------------------------------------------------------------------------------------------------------+
//  |row-r9pv-p86t.ifsp,11 |
//  |row-7v2v~88z5-44se,6|
//  |row-hzc9-4kvv~mbc9,6 |
//  +-----------------------------------------------------------------------------------------------------------------------+

  val df2 = df1.select(
    split('dataString,").getItem(0).alias("c0"),split('dataString,").getItem(1).alias("c1"),").getItem(2).alias("c2"),").getItem(3).alias("c3"),").getItem(4).alias("c4"),").getItem(5).alias("c5"),").getItem(6).alias("c6"),").getItem(7).alias("c7"),").getItem(8).alias("c8"),").getItem(9).alias("c9"),").getItem(10).alias("c10"),").getItem(11).alias("c11"),").getItem(12).alias("c12")
  )
  df2.printSchema()
//  root
//  |-- c0: string (nullable = true)
//  |-- c1: string (nullable = true)
//  |-- c2: string (nullable = true)
//  |-- c3: string (nullable = true)
//  |-- c4: string (nullable = true)
//  |-- c5: string (nullable = true)
//  |-- c6: string (nullable = true)
//  |-- c7: string (nullable = true)
//  |-- c8: string (nullable = true)
//  |-- c9: string (nullable = true)
//  |-- c10: string (nullable = true)
//  |-- c11: string (nullable = true)
//  |-- c12: string (nullable = true)

  df2.show(false)
//  +------------------+------------------------------------+---+----------+---+----------+---+---+----+----+-------+---+---+
//  |c0                |c1                                  |c2 |c3        |c4 |c5        |c6 |c7 |c8  |c9  |c10    |c11|c12|
//  +------------------+------------------------------------+---+----------+---+----------+---+---+----+----+-------+---+---+
//  |row-r9pv-p86t.ifsp|00000000-0000-0000-0838-60C2FFCC43AE|0  |1574264158|   |1574264158|   |{ }|2007|ZOEY|KINGS  |F  |11 |
//  |row-7v2v~88z5-44se|00000000-0000-0000-C8FC-DDD3F9A72DFF|0  |1574264158|   |1574264158|   |{ }|2007|ZOEY|SUFFOLK|F  |6  |
//  |row-hzc9-4kvv~mbc9|00000000-0000-0000-562E-D9A0792557FC|0  |1574264158|   |1574264158|   |{ }|2007|ZOEY|MONROE |F  |6  |
//  +------------------+------------------------------------+---+----------+---+----------+---+---+----+----+-------+---+---+

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
VM VirtualBox中安装centos时报错：设置基础软件仓库时出错
参考1 参考2 解决方案 # 点击安装源 协议选择 http:// 路径填写 mirrors.aliyun.com/centos/8.3.2011/BaseOS/x86_64/os URL类型 软件库URL 其他路径 # 版本 7 mirrors.aliyun.com/centos/7/os/x86
kafka常见问题
报错1 [root@slave1 data_mocker]# kafka-console-consumer.sh --bootstrap-server slave1:9092 --topic topic_db [2023-12-19 18:31:12,770] WARN [Consumer clie
hive常见问题
错误1 # 重写数据 hive (edu)&gt; insert overwrite table dwd_trade_cart_add_inc &gt; select data.id, &gt; data.user_id, &gt; data.course_id, &gt; date_format(
装载数据时报错：Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
错误1 hive (edu)&gt; insert into huanhuan values(1,&#39;haoge&#39;); Query ID = root_20240110071417_fe1517ad-3607-41f4-bdcf-d00b98ac443e Total jobs = 1
flume常见问题
报错1：执行到如下就不执行了，没有显示Successfully registered new MBean. [root@slave1 bin]# /usr/local/software/flume-1.9.0/bin/flume-ng agent -n a1 -c /usr/local/softwa
hadoop集群常见错误
虚拟及没有启动任何服务器查看jps会显示jps，如果没有显示任何东西 [root@slave2 ~]# jps 9647 Jps 解决方案 # 进入/tmp查看 [root@slave1 dfs]# cd /tmp [root@slave1 tmp]# ll 总用量 48 drwxr-xr-x. 2
hive常见错误
报错1 hive&gt; show databases; OK Failed with exception java.io.IOException:java.lang.RuntimeException: Error in configuring object Time taken: 0.474 se
Linux常见错误
报错1 [root@localhost ~]# vim -bash: vim: 未找到命令 安装vim yum -y install vim* # 查看是否安装成功 [root@hadoop01 hadoop]# rpm -qa |grep vim vim-X11-7.4.629-8.el7_9.x
使用sqoop一直卡在：mapreduce.Job: Running job: job_1703173956074_0001
修改hadoop配置 vi /usr/local/software/hadoop-2.9.2/etc/hadoop/yarn-site.xml # 添加如下 &lt;configuration&gt; &lt;property&gt; &lt;name&gt;yarn.nodemanager.res
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot