Pyspark-java.lang.OutOfMemoryError：写入csv文件时的Java堆空间

如何解决Pyspark-java.lang.OutOfMemoryError：写入csv文件时的Java堆空间

当尝试使用以下代码写入csv文件时

DF.coalesce(1).write.option("header","false").option("sep",",").option("escape",'"').option("ignoreTrailingWhiteSpace","false").option("ignoreLeadingWhiteSpace","false").mode("overwrite").csv(filename)

我遇到以下错误

ileFormatWriter.scala:169)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space

有人可以建议解决方法吗？

解决方法

尝试在提交火花的应用程序中增加executor.memory

类似这样的东西

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

,

对我来说，添加以下spark配置可以解决此问题

spark = SparkSession.builder.master('local[*]').config("spark.driver.memory","15g").appName('sl-app').getOrCreate()