微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

如何在VS Code中设置pyspark?

如何解决如何在VS Code中设置pyspark?

我试图在VS-Code中运行pypsark,但似乎无法将我的环境指向正确的pyspark驱动程序和路径。当我在终端窗口中运行pyspark时,它看起来像这样:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR,use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/

Using Python version 3.8.3 (default,Jul  2 2020 11:26:31)
SparkSession available as 'spark'.

当我用Vs代码运行文件时,出现此错误

File "/Users/myuser/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",line 477,in main
    ("%d.%d" % sys.version_info[:2],version))
Exception: Python in worker has different version 2.7 than that in driver 3.8,PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

在我的bash_profile中,我认为设置正确:

export SPARK_PATH=~/spark-1.6.0-bin-hadoop2.6
export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

我发现它正在查看两个不同的版本,那么,我需要怎么做才能在VS代码中正确配置环境以运行pyspark?

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。