微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

如何在我安装 Apache Spark和 PySpark的确切位置安装 Apache Toree?

如何解决如何在我安装 Apache Spark和 PySpark的确切位置安装 Apache Toree?

所以我在位置 /opt/spark/ (Ubuntu) 安装了 Spark。我正在尝试安装 Apache Toree 以便在 Jupyter 笔记本中使用 Spark。我相信 Apache Spark 和 Toree 的安装位置需要在同一个目录中,但我不能 pip install toree 进入同一个文件夹。所以我附上spark安装的目录信息-

deep@deep-VirtualBox:~$ cd /opt/spark/
deep@deep-VirtualBox:/opt/spark$ ls
bin  conf  data  examples  jars  kubernetes  LICENSE  licenses  
logs  NOTICE  python  R  README.md  RELEASE  sbin  work  yarn

错误 -

[I 01:40:14.396 LabApp] KernelRestarter: restarting kernel (4/5),keep random ports
Starting Spark Kernel with SPARK_HOME=/opt/spark/
21/01/27 01:40:16 WARN Utils: Your hostname,deep-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
21/01/27 01:40:16 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/01/27 01:40:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
    at org.apache.toree.Main$.<init>(Main.scala:24)
    at org.apache.toree.Main$.<clinit>(Main.scala)
    at org.apache.toree.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.classNotFoundException: scala.App$class
    at java.base/java.net.urlclassloader.findClass(urlclassloader.java:471)
    at java.base/java.lang.classLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.classLoader.loadClass(ClassLoader.java:522)
    ... 15 more
log4j:WARN No appenders Could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[W 01:40:17.410 LabApp] KernelRestarter: restart Failed
[W 01:40:17.410 LabApp] Kernel 5bac4936-6c30-45b3-bb78-c82469d58dd3 died,removing from map.
[W 01:40:56.400 LabApp] Timeout waiting for kernel_info reply from 5bac4936-6c30-45b3-bb78-c82469d58dd3
[E 01:40:56.401 LabApp] Error opening stream: HTTP 404: Not Found (Kernel does not exist: 5bac4936-6c30-45b3-bb78-c82469d58dd3)

你们中的任何人都可以帮我吗?我可能在这里做错了什么。感谢您的投入。

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。