如何解决Livy 未连接到 EMR 主节点
我正在尝试将 r-studio 与 EMR 连接以使用 Livy 和 sparklyr 提交作业，我已经安装了 sparklyr 包，当我尝试使用 spark_connect 进行连接时，如下所示，我收到 lvy 连接失败错误。
library('sparklyr')
library('dplyr')
sc <- spark_connect( master = "https://<hostname>:8998",method = "livy",version = '2.4.5',app_name = "sparklyr",config = livy_config(config = spark_config(),username = NULL,password = NULL,negotiate = FALSE,custom_headers = list(`X-Requested-By` = "sparklyr"),proxy = NULL,curl_opts = NULL))
ERROR
Error in livy_connection(master,config,app_name,version,hadoop_version,:
Failed to launch livy session,session status is shutting_down
Logs
20/12/23 11:13:54 INFO StateStore$: Using BlackholeStateStore for recovery.
20/12/23 11:13:54 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
20/12/23 11:13:54 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
20/12/23 11:13:54 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
20/12/23 11:13:55 INFO WebServer: Starting server on https://ip-172-18-205-248.cihi.ca:8998
20/12/23 14:14:54 WARN InteractiveSession$: Enable HiveContext but no hive-site.xml found under classpath or user request.
20/12/23 14:14:54 INFO InteractiveSession$: Creating Interactive session 0: [owner: null,request: [kind: spark,proxyUser: Some(livy),jars: https://github.com/sparklyr/sparklyr/blob/feature/sparklyr-1.4.0/inst/java/sparklyr-2.4-2.11.jar?raw=true,heartbeatTimeoutInSecond: 0]]
20/12/23 14:14:54 INFO RpcServer: Connected to the port 10001
20/12/23 14:14:54 INFO InteractiveSessionManager: Registering new session 0
20/12/23 14:14:54 INFO InteractiveSessionManager: Registered new session 0
20/12/23 14:14:56 INFO LineBufferedStream: 20/12/23 14:14:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/12/23 14:15:57 INFO LineBufferedStream: Exception in thread "main" java.net.socketTimeoutException: connect timed out
20/12/23 14:15:57 INFO LineBufferedStream: at java.net.PlainSocketImpl.socketConnect(Native Method)
20/12/23 14:15:57 INFO LineBufferedStream: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
20/12/23 14:15:57 INFO LineBufferedStream: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
20/12/23 14:15:57 INFO LineBufferedStream: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
20/12/23 14:15:57 INFO LineBufferedStream: at java.net.socksSocketImpl.connect(SocksSocketImpl.java:392)
20/12/23 14:15:57 INFO LineBufferedStream: at java.net.socket.connect(Socket.java:607)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:284)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
20/12/23 14:15:57 INFO LineBufferedStream: at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:757)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:137)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$downloadResource$1(SparkSubmit.scala:389)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$13$$anonfun$apply$3.apply(SparkSubmit.scala:403)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$13$$anonfun$apply$3.apply(SparkSubmit.scala:403)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.collection.AbstractTraversable.map(Traversable.scala:104)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$13.apply(SparkSubmit.scala:403)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$13.apply(SparkSubmit.scala:402)
20/12/23 14:15:57 INFO LineBufferedStream: at scala.Option.map(Option.scala:146)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:402)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anon$3.run(SparkSubmit.scala:146)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anon$3.run(SparkSubmit.scala:144)
20/12/23 14:15:57 INFO LineBufferedStream: at java.security.AccessController.doPrivileged(Native Method)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
20/12/23 14:15:57 INFO LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/12/23 14:15:57 WARN ContextLauncher: Child process exited with code 1.
20/12/23 14:15:57 ERROR RScclient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:400)
at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:451)
at java.lang.Thread.run(Thread.java:748)
20/12/23 14:15:57 INFO RScclient: Failing pending job a040eee0-e14f-490b-a79f-ddb58b879797 due to shutdown.
20/12/23 14:15:57 INFO InteractiveSession: Stopping InteractiveSession 0...
20/12/23 14:15:57 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
20/12/23 14:15:59 ERROR SparkYarnApp: Error whiling refreshing YARN state
java.lang.IllegalStateException: spark-submit start Failed
at org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:192)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:267)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:264)
at scala.Option.getorElse(Option.scala:121)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:264)
at org.apache.livy.Utils$$anon$1.run(Utils.scala:97)
20/12/23 14:15:59 INFO InteractiveSession: Stopped InteractiveSession 0.
20/12/23 14:15:59 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411)
at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.io.IOException: Child process exited with code 1.
at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:400)
at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:451)
at java.lang.Thread.run(Thread.java:748)
20/12/23 14:26:54 INFO InteractiveSessionManager: Deleting InteractiveSession 0 because it was inactive for more than 720000.0 ms.
20/12/23 14:26:54 INFO InteractiveSessionManager: Deleting session 0
20/12/23 14:26:54 INFO InteractiveSession: Stopping InteractiveSession 0...
20/12/23 14:26:54 INFO InteractiveSession: Stopped InteractiveSession 0.
20/12/23 14:26:54 INFO InteractiveSessionManager: Deleted session 0
任何人都可以指导我了解缺少什么或我做错了什么吗？
谢谢。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
