Spark 抛出错误 org.apache.spark.memory.SparkOutOfMemoryError：无法获取 262144 字节的内存，得到 0

如何解决Spark 抛出错误 org.apache.spark.memory.SparkOutOfMemoryError：无法获取 262144 字节的内存，得到 0

我正在使用 PySpark 代码运行 spark-submit pyspark_script.py --master yarn --num-executors 6 --driver-memory 30g --executor-memory 7g

我确保在纱线模式下运行它。仍然面临内存不足问题。

我的代码：

from pyspark import SparkContext,SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType,StructField,StringType,IntegerType,LongType,DoubleType
import os
conf = SparkConf().setAppName("PySpark App")
sc=SparkContext(conf=conf)
base_path=os.path.join(os.path.dirname(os.path.abspath(__file__)))
file_path=os.path.join(os.path.abspath("combine_ocssubsdump_20210526.csv"))
rdd1=sc.textFile("file:////mapr/samrat/test/samrat/combine_ocssubsdump_20210526.csv",5,use_unicode=False)
spark = SparkSession \
    .builder \
    .appName("recon") \
    .master("yarn") \
    .enableHiveSupport() \
    .getorCreate()

#schema=["SubscriberID","ParentID","Brand","SubBrand","ActivationDate","ActiveEndDate","CallBarringEndDate","SuspendEndDate","Status","AirtimeBalance","AirtimeBalanceExpiry","StatusDetail","ProductWallet","OfferDetails","fileProperties"]
schema = StructType([StructField("SubscriberID",StringType(),True),StructField("ParentID",StructField("Brand",StructField("SubBrand",StructField("ActivationDate",StructField("ActiveEndDate",StructField("CallBarringEndDate",StructField("SuspendEndDate",StructField("Status",StructField("AirtimeBalance",StructField("AirtimeBalanceExpiry",StructField("StatusDetail",StructField("ProductWallet",rdd1=rdd1.map(lambda x:x.split("|")).cache()
rdd2=rdd1.map(lambda x:(x[4].split(" ")[0],1))
rdd3=rdd2.reduceByKey(lambda a,b:a+b)
df1=spark.createDataFrame(data=rdd1,schema=schema)
df2=df1.select("SubscriberID","ActivationDate")
df3=df1.groupBy("Brand").count()
df3.show()