在Windows 10上，如何在不带Databricks但带wasbs的情况下从带有Apache Spark的Azure Blob中读取文件？

如何解决在Windows 10上，如何在不带Databricks但带wasbs的情况下从带有Apache Spark的Azure Blob中读取文件？

我有azure-storage-8.6.0.jar和hadoop-azure-3.0.1.jar。我不断从其他论坛上看到，我必须像https://github.com/hning86/articles/blob/master/hadoopAndWasb.md一样修改hadoop的etc文件夹中的core-site.xml文件。我什至不知道我什至需要下载所有hadoop才能运行Spark。我以为我需要的只是hadoop / bin中的winutils.exe。

    spark.read.load(f"wasbs://{container_name}@{storage_account_name}.blob.core.windows.net/{container_name}/myfile.txt" )



Py4JJavaError: An error occurred while calling o53.load.
: java.io.IOException: No FileSystem for scheme: wasbs
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:46)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:366)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:286)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:286)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:232)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

解决方法

如果要使用pyspark从Windows 10上的Azure blob存储中读取CSV文件，请参考以下步骤

1. 安装pyspark

pip install pyspark

1. 代码（创建.py文件）

from pyspark.sql import SparkSession
import traceback

try:

    spark = SparkSession.builder.getOrCreate()
    conf = spark.sparkContext._jsc.hadoopConfiguration()
    conf.set("fs.wasbs.impl","org.apache.hadoop.fs.azure.NativeAzureFileSystem")
    spark.conf.set('fs.azure.account.key.<account name>.blob.core.windows.net','<account key>')
    df = spark.read.option("header",True).csv(
        'wasbs://<container name>@<account name>.blob.core.windows.net/<directory name>/<file name>')
    df.show()

except Exception as exp:
    print("Exception occurred")
    print(traceback.format_exc())

1. 运行代码

cd <your python or env path>\Scripts
spark-submit --packages org.apache.hadoop:hadoop-azure:3.2.1,com.microsoft.azure:azure-storage:8.6.5 <your py file path>