如何解决如何使用 Python SDK 将大文件~100mb上传到 Azure blob 存储?
我使用的是最新的 Azure 存储 SDK (azure-storage-blob-12.7.1)。它适用于较小的文件,但对于大于 30MB 的较大文件会引发异常。
azure.core.exceptions.ServiceResponseError: ('Connection aborted.',timeout('写操作超时'))
from azure.storage.blob import BlobServiceClient,PublicAccess,BlobProperties,ContainerClient
def upload(file):
settings = read_settings()
connection_string = settings['connection_string']
container_client = ContainerClient.from_connection_string(connection_string,'backup')
blob_client = container_client.get_blob_client(file)
with open(file,"rb") as data:
blob_client.upload_blob(data)
print(f'{file} uploaded to blob storage')
upload('crashes.csv')
解决方法
当我尝试上传 ~180MB .txt 文件时,您的代码似乎对我有用。但是,如果上传小文件对您有用,我认为将大文件分成小部分上传可能是一种解决方法。试试下面的代码:
from azure.storage.blob import BlobClient
storage_connection_string=''
container_name = ''
dest_file_name = ''
local_file_path = ''
blob_client = BlobClient.from_connection_string(storage_connection_string,container_name,dest_file_name)
#upload 4 MB for each request
chunk_size=4*1024*1024
if(blob_client.exists):
blob_client.delete_blob()
blob_client.create_append_blob()
with open(local_file_path,"rb") as stream:
while True:
read_data = stream.read(chunk_size)
if not read_data:
print('uploaded')
break
blob_client.append_block(read_data)
结果:
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。