如何解决Pyspark dataframe列包含字典数组,要使字典中的每个键成为一列
我目前有一个这样的数据框:
+-------+-------+-------+-------+
| Id |value_list_of_dicts |
+-------+-------+-------+-------+
| 1 |[{"val1":0,"val2":0},|
| |{"val1":2,"val2":5}] |
+-------+-------+-------+-------+
| 2 |[{"val1":9,"val2":10},|
| |{"val1":1,"val2":2}] |
+-------+-------+-------+-------+
每个列表正好包含30个词典,其值可能不同,但键名始终相同。我希望我的数据框看起来像这样:
+-------+-------+-------+
| Id |val1 |val2 |
+-------+-------+-------+
| 1 | 0 | 0 |
+-------+-------+-------+
| 1 | 2 | 5 |
+-------+-------+-------+
| 2 | 9 | 10 |
+-------+-------+-------+
| 2 | 1 | 2 |
+-------+-------+-------+
做到这一点的最佳方法是什么?
解决方法
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import *
from datetime import datetime
from pyspark.sql import *
from collections import *
from pyspark.sql.functions import udf,explode
from pyspark.sql.types import StringType
df= spark.createDataFrame(
[
(1,[{"val1":0,"val2":0},{"val1":2,"val2":5}]),(2,[{"val1":9,"val2":10},{"val1":1,"val2":2}])
],("ID","List")
)
df2 = df.select(df.ID,explode(df.List).alias("Column1") )
df2.withColumn("Val1",F.col("Column1").getItem("val1")).withColumn("Val2",F.col("Column1").getItem("val2")).show(truncate=False)
输出:
+---+-----------------------+----+----+
|ID |Column1 |Val1|Val2|
+---+-----------------------+----+----+
|1 |[val2 -> 0,val1 -> 0] |0 |0 |
|1 |[val2 -> 5,val1 -> 2] |2 |5 |
|2 |[val2 -> 10,val1 -> 9]|9 |10 |
|2 |[val2 -> 2,val1 -> 1] |1 |2 |
+ --- + ----------------------- + ---- + ---- +
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。