如何禁用随机文件的磁盘写入？

如何解决如何禁用随机文件的磁盘写入？

嗨，我们有火花集群，在火花作业执行期间，当将中间数据写入 spark.local.dir 位置时，我得到了内存溢出，但是当我看到它们的执行器未使用的内存超过两倍时，而不是写入该目录，我们可以将数据存储到内存中吗？

在例外详情下方

Job aborted due to stage failure: Task 134555 in stage 32.0 Failed 4 times,most recent failure: Lost task 134555.3 in stage 32.0 (TID 151065,<<some worker node IP>>,executor 318): java.io.FileNotFoundException: /opt/spark/tmp/spark-98331af4-b923-4342-ae3e-93e764b02d4a/executor-a5874092-943d-4b57-b1d0-eab05a3d36c5/blockmgr-17e989d8-4657-4a4e-bc93-ea075cb45f61/0f/temp_shuffle_1f574b0e-617b-46db-a558-9937a911c90a (No space left on device)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at org.apache.spark.storage.diskBlockObjectWriter.initialize(diskBlockObjectWriter.scala:103)
    at org.apache.spark.storage.diskBlockObjectWriter.open(diskBlockObjectWriter.scala:116)
    at org.apache.spark.storage.diskBlockObjectWriter.write(diskBlockObjectWriter.scala:249)
    at org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:211)
    at org.apache.spark.shuffle.sort.ShuffleExternalSorter.closeAndGetSpills(ShuffleExternalSorter.java:419)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:230)
    at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:190)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:

在失败阶段的屏幕截图下方

解决方法

我的理解是无法禁用磁盘写入。

ShuffleMapTask 必须将中间 shuffle 映射输出文件（shuffle 块）写入 spark.local.dir 目录，以便（这个 shuffle 阶段的）reducers 可以完成他们的工作（没有双关语）打算）。

您可以尝试使用 External Shuffle Service 来减少磁盘使用量（这样每个执行程序都可以管理自己的与 shuffle 相关的目录，他们可以卸载它）。您可以在 Dynamic Resource Allocation 中稍微了解一下。