如何解决WeekOfYear 列在 SparkSQL 中变空
在这里,我正在为 spark.sql 编写 SQL 语句,但我无法将 WEEKOFYEAR 转换为一年中的第几周并在输出中获得空值 下面我展示了我使用的表达
输入数据
InvoiceNo,StockCode,Description,Quantity,InvoiceDate,UnitPrice,CustomerID,Country
536365,85123A,WHITE HANGING HEART T-LIGHT HOLDER,6,01-12-2010 8.26,2.55,17850,United Kingdom
536365,71053,WHITE METAL LANTERN,3.39,84406B,CREAM CUPID HEARTS COAT HANGER,8,2.75,84029G,KNITTED UNION FLAG HOT WATER BOTTLE,United Kingdom
SQL 代码
val summarySQlTest = spark.sql(
"""
|select Country,WEEKOFYEAR(InvoiceDate)as WeekNumber,|count(distinct(InvoiceNo)) as NumInvoices,|sum(Quantity) as TotalQuantity,|round(sum(Quantity*UnitPrice),2) as InvoiceValue
|from sales
|group by Country,WeekNumber
|""".stripMargin
).show()
期望的输出
+--------------+----------+-----------+-------------+------------+
| Country|WeekNumber|NumInvoices|TotalQuantity|InvoiceValue|
+--------------+----------+-----------+-------------+------------+
| Spain| 49| 1| 67| 174.72|
| Germany| 48| 11| 1795| 3309.75|
我得到的输出
+--------------+----------+-----------+-------------+------------+
| Country|WeekNumber|NumInvoices|TotalQuantity|InvoiceValue|
+--------------+----------+-----------+-------------+------------+
| Spain| null| 1| 67| 174.72|
| Germany| null| 11| 1795| 3309.75|
对于我使用的所需输出,但我想在 spark.sql 中解决相同的问题
如果有人能解释这里实际发生的事情,那就太好了 (to_date(col("InvoiceDate"),"dd-MM-yyyy H.mm")
val knowFunc= invoicesDF.withColumn("InvoiceDate",to_date(col("InvoiceDate"),"dd-MM-yyyy H.mm"))
.where("year(InvoiceDate) == 2010")
.withColumn("WeekNumber",weekofyear(col("InvoiceDate")))
.groupBy("Country","WeekNumber")
.agg(sum("Quantity").as("TotalQuantity"),round(sum(expr("Quantity*UnitPrice")),2).as("InvoiceValue")).show()
解决方法
您需要先将 InvoiceDate
列转换为日期类型(使用 to_date
),然后才能调用 weekofyear
。我想这也回答了您的最后一个问题。
val summarySQlTest = spark.sql(
"""
|select Country,WEEKOFYEAR(to_date(InvoiceDate,'dd-MM-yyyy H.mm')) as WeekNumber,|count(distinct(InvoiceNo)) as NumInvoices,|sum(Quantity) as TotalQuantity,|round(sum(Quantity*UnitPrice),2) as InvoiceValue
|from sales
|group by Country,WeekNumber
|""".stripMargin
).show()
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。