如何解决使用pyspark中另一个数据框的列创建数据框
root
|-- content: string (nullable = true)
|-- conversationId: long (nullable = true)
|-- date: string (nullable = true)
|-- id: long (nullable = true)
|-- lang: string (nullable = true)
|-- likeCount: long (nullable = true)
|-- media: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- duration: double (nullable = true)
| | |-- fullUrl: string (nullable = true)
| | |-- previewUrl: string (nullable = true)
| | |-- thumbnailUrl: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- variants: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- bitrate: long (nullable = true)
| | | | |-- contentType: string (nullable = true)
| | | | |-- url: string (nullable = true)
我要提取单独的数据框
- 没有媒体列的根数据框
- 没有变量列的媒体数据框
- 变体数据框
问题是我无法删除新variants数据框中的media列。
在media新数据框中，我不希望media列中的所有数据，我希望media对象成为其数据框中的新根，其字段为column ，并且与variants新数据框相同
这是我的代码：
tweets_df = spark.read.json(jsonFile)
media_df = tweets_df.select(col('media'))
variants_df = media_df.select(col('media.variants'))
# i tried also media_df = media_df.drop('media.variants')
media_df = media_df.drop('variants')
tweets_df = tweets_df.drop('media')
这是结果：
root
|-- variants: array (nullable = true)
| |-- element: array (containsNull = true)
| | |-- element: struct (containsNull = true)
| | | |-- bitrate: long (nullable = true)
| | | |-- contentType: string (nullable = true)
| | | |-- url: string (nullable = true)
root
|-- media: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- duration: double (nullable = true)
| | |-- fullUrl: string (nullable = true)
| | |-- previewUrl: string (nullable = true)
| | |-- thumbnailUrl: string (nullable = true)
| | |-- type: string (nullable = true) **IT'S NOT DELETED**
| | |-- variants: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- bitrate: long (nullable = true)
| | | | |-- contentType: string (nullable = true)
| | | | |-- url: string (nullable = true)
root
|-- content: string (nullable = true)
|-- conversationId: long (nullable = true)
|-- date: string (nullable = true)
|-- idOnSite: long (nullable = true)
|-- lang: string (nullable = true)
|-- likeCount: long (nullable = true)
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
