如何解决如何在VS Code中设置pyspark?
我试图在VS-Code中运行pypsark,但似乎无法将我的环境指向正确的pyspark驱动程序和路径。当我在终端窗口中运行pyspark时,它看起来像这样:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR,use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.0.1
/_/
Using Python version 3.8.3 (default,Jul 2 2020 11:26:31)
SparkSession available as 'spark'.
File "/Users/myuser/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",line 477,in main
("%d.%d" % sys.version_info[:2],version))
Exception: Python in worker has different version 2.7 than that in driver 3.8,PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
在我的bash_profile中,我认为设置正确:
export SPARK_PATH=~/spark-1.6.0-bin-hadoop2.6
export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'
我发现它正在查看两个不同的版本,那么,我需要怎么做才能在VS代码中正确配置环境以运行pyspark?
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。