python – Spark – 为我的spark作业分配了多少个执行器和内核

Spark架构完全围绕执行器和核心的概念.我想看看在集群中运行的spark应用程序运行了多少执行程序和核心.

我试图在我的应用程序中使用下面的代码段,但没有运气.

val conf = new SparkConf().setAppName("ExecutorTestJob")
val sc = new SparkContext(conf)
conf.get("spark.executor.instances")
conf.get("spark.executor.cores")

有没有办法使用SparkContext Object或SparkConf对象等来获取这些值.

解决方法:

Scala(程序化方式)：

getExecutorStorageStatus和getExecutorMemoryStatus都返回执行程序的数量,包括驱动程序.
如下面的示例代码段.

/** Method that just returns the current active/registered executors
        * excluding the driver.
        * @param sc The spark context to retrieve registered executors.
        * @return a list of executors each in the form of host:port.
        */
       def currentActiveExecutors(sc: SparkContext): Seq[String] = {
         val allExecutors = sc.getExecutorMemoryStatus.map(_._1)
         val driverHost: String = sc.getConf.get("spark.driver.host")
         allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList
       }

sc.getConf.getInt("spark.executor.instances", 1)

同样获取所有属性并打印如下,您也可以获得核心信息..

sc.getConf.getAll.mkString("\n")

要么

sc.getConf.toDebugString

执行程序spark.driver.cores驱动程序的spark.executor.cores应该具有此值.

Python：

Above methods getExecutorStorageStatus and getExecutorMemoryStatus, In python api were not implemented

编辑
但是可以使用SparkSession公开的Py4J绑定来访问它.

sc._jsc.sc().getExecutorMemoryStatus()