微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

Jupyter Notebook中的PySpark问题

如何解决Jupyter Notebook中的PySpark问题

初始化时出现此错误。我已经设置了master和worker,然后启动了它。尽管在设置了master和spark之后,我没有启动spark-shell来直接运行它。有人可以帮忙吗..

代码

import findspark
import os
os.environ["SPARK_HOME"]="C:\Spark"
findspark.init()
from pyspark.sql import SparkSession
from pyspark import SparkConf,SparkContext

错误

IndexError                                Traceback (most recent call last)
c:\users\khan\appdata\local\programs\python\python37\lib\site-packages\findspark.py in init(spark_home,python_path,edit_rc,edit_profile)
    142     try:
--> 143         py4j = glob(os.path.join(spark_python,"lib","py4j-*.zip"))[0]
    144     except IndexError:

IndexError: list index out of range

During handling of the above exception,another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-7-db62de47bcf3> in <module>
      2 import os
      3 os.environ["SPARK_HOME"]="C:\Spark"
----> 4 findspark.init()
      5 from pyspark.sql import SparkSession
      6 from pyspark import SparkConf,SparkContext

c:\users\khan\appdata\local\programs\python\python37\lib\site-packages\findspark.py in init(spark_home,edit_profile)
    144     except IndexError:
    145         raise Exception(
--> 146             "Unable to find py4j,your SPARK_HOME may not be configured correctly"
    147         )
    148     sys.path[:0] = [spark_python,py4j]

Exception: Unable to find py4j,your SPARK_HOME may not be configured correctly

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。