无法在独立模式下运行 Spark 程序客户端和集群模式中的错误 client 模式的输出：cluster 模式的输出：

如何解决无法在独立模式下运行 Spark 程序客户端和集群模式中的错误 client 模式的输出：cluster 模式的输出：

我有一个 Ubuntu 服务器，我在其中运行了一个主服务器和一个从服务器（一个执行程序），它们显示在 8080 用户界面上。
我可以成功运行 spark-shell --master spark://foo.bar:7077，但我无法成功提交我的程序（fat jar）并且出现错误（独立模式）。

我有一个 Main 对象，它 extends App 而不是有一个 main 方法。并且都在一个 package myProject 中。我正在像这样运行我的程序：

 spark-submit --master spark://foo.bar:7077 \
--class myProject.Main \
--deploy-mode client \
--num-executors 1 \
--executor-memory 58g \
--executor-cores 39 \
--driver-memory 4g \
--driver-cores 2 \
--conf spark.driver.memoryOverhead=819m \
--conf spark.executor.memoryOverhead=819m \
target/scala-2.12/myProject-assembly-0.1.jar

client 模式的输出：

Exception in thread "main" java.lang.NoSuchMethodError: scala.App.$init$(Lscala/App;)V
        at mstproject.Main$.<init>(Main.scala:8)
        at mstproject.Main$.<clinit>(Main.scala)
        at mstproject.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
log4j:WARN No appenders Could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

我已经检查了 similar error，但是我的所有软件包似乎都与 Scala 2.12 兼容，正如我的 build.sbt 所示（我不确定我的 assemblyMergeStrategy ):

scalaVersion := "2.12.12"
libraryDependencies ++= Seq(
  "com.github.pathikrit" %% "better-files" % "3.9.1","org.scalatest" %% "scalatest" % "3.2.3" % Test,"org.apache.spark" %% "spark-core" % "2.4.8","org.apache.spark" %% "spark-sql" % "2.4.8","org.apache.spark" %% "spark-graphx" % "2.4.8","redis.clients" % "jedis" % "3.5.1","com.redislabs" %% "spark-redis" % "2.4.2"
)

assemblyMergeStrategy in assembly := {
  case PathList("org","aopalliance",xs @ _*) => MergeStrategy.last
  case PathList("javax","inject","servlet","activation",xs @ _*) => MergeStrategy.last
  case PathList("org","apache",xs @ _*) => MergeStrategy.last
  case PathList("com","google","esotericsoftware","codahale","yammer",xs @ _*) => MergeStrategy.last
  case "about.html" => MergeStrategy.rename
  case "meta-inf/ECLIPSEF.RSA" => MergeStrategy.last
  case "meta-inf/mailcap" => MergeStrategy.last
  case "meta-inf/mimetypes.default" => MergeStrategy.last
  case PathList("meta-inf",xs @ _*) =>
    xs map {_.toLowerCase} match {
      case "manifest.mf" :: Nil | "index.list" :: Nil | "dependencies" :: Nil =>
        MergeStrategy.discard
      case ps @ x :: xs if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
        MergeStrategy.discard
      case "plexus" :: xs =>
        MergeStrategy.discard
      case "services" :: xs =>
        MergeStrategy.filterdistinctLines
      case "spring.schemas" :: Nil | "spring.handlers" :: Nil =>
        MergeStrategy.filterdistinctLines
      case _ => MergeStrategy.first
    }
  case "application.conf" => MergeStrategy.concat
  case "reference.conf" => MergeStrategy.concat
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case _ => MergeStrategy.first
//  case x =>
//    val oldStrategy = (assemblyMergeStrategy in assembly).value
//    oldStrategy(x)
}

cluster 模式的输出：

log4j:WARN No appenders Could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/07/17 17:35:40 INFO SecurityManager: Changing view acls to: root
21/07/17 17:35:40 INFO SecurityManager: Changing modify acls to: root
21/07/17 17:35:40 INFO SecurityManager: Changing view acls groups to:
21/07/17 17:35:40 INFO SecurityManager: Changing modify acls groups to:
21/07/17 17:35:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
21/07/17 17:35:40 INFO Utils: Successfully started service 'driverClient' on port 34218.
21/07/17 17:35:40 INFO TransportClientFactory: Successfully created connection to foo.bar/10.0.8.137:7077 after 39 ms (0 ms spent in bootstraps)
21/07/17 17:35:41 INFO ClientEndpoint: Driver successfully submitted as driver-20210717173541-0003
21/07/17 17:35:41 INFO ClientEndpoint: ... waiting before polling master for driver state
21/07/17 17:35:46 INFO ClientEndpoint: ... polling master for driver state
21/07/17 17:35:46 INFO ClientEndpoint: State of driver-20210717173541-0003 is Failed
21/07/17 17:35:46 INFO ShutdownHookManager: Shutdown hook called
21/07/17 17:35:46 INFO ShutdownHookManager: Deleting directory /tmp/spark-c15b4457-664f-43b7-9699-b62839ec83c0

• Cluster 模式给出相同的输出，即使我给出一个随机 --class 名称。但是 client 模式只会发出错误。
• 在 Cluster 模式下提交会在 Master 的 UI（8080 端口）中添加一个状态为“Failed”的“Completed Driver”。
• 我可以在本地模式下成功运行我的程序。
• 在 client 模式下没有 master 和 worker 日志输出。
• 在 cluster 模式下。工作日志通知复制 jar 和驱动程序失败。
• 在 cluster 模式下，主日志给出以下输出：

21/07/17 17:45:08 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
21/07/17 17:45:08 INFO Master: Launching driver driver-20210717174508-0007 on worker worker-20210716205255-10.0.8.137-45558
21/07/17 17:45:12 INFO Master: Removing driver: driver-20210717174508-0007
21/07/17 17:45:14 INFO Master: 10.0.8.137:34806 got disassociated,removing it.
21/07/17 17:45:14 INFO Master: 10.0.8.137:38394 got disassociated,removing it.

我觉得在两个 deploy-mode 中都发生了同样的事情。 client 中的错误通过 spark-submit 可见，但 cluster 模式下的错误是隐藏的，必须从 Master UI 中检查（我不明白为什么它在 Master 和 Worker 日志文件中不可见） .

更新： 正如 Luis 所说，这是 Scala 不兼容的问题，我的 Spark 集群使用的是嵌入式 Scala 2.11 而不是 2.12。所以通过将我的 fat jar Scala 版本降级到 2.11 来修复它。