如何解决无法打开大于内存的 HDF5 文件... ValueError
我有很多来自 nyc.gov 的纽约出租车的 .csv,一个 .csv = 年-月。在那里,我抓取了 15 个 csvs 并从中制作了 HDF5:
import h5py
import pandas as pd
import os
import glob
import numpy as np
import vaex
from tqdm import tqdm_notebook as tqdm
#hdf = pd.hdfstore('c:/Projekty/H5Edu/NYCTaxi/NYCTaxi.hp')
#df1 = pd.read_csv('path nejake csvcko')
#hdf.put('DF1',df1,format = 'table',data_columns = True)
csv_list = np.sort(np.array(glob.glob('G:\\NYCTaxi\\*.csv')))[::-1]
csv_list = csv_list[20:39]
output_dir = 'c:\\Datasety\\YelowTaxi\\DataH5\\'
for file in tqdm(csv_list,leave=False,desc='Converting to hdf5...'):
# Setting up the files,and directories
#zip_file = ZipFile(file)
output_file = file.split('\\')[-1][:-3]+'hdf5'
output = output_dir + output_file
#output = output_file
# Check if a converted file already exists: if it does skip it,otherwise read in the raw csv and convert it
if (os.path.exists(output) and os.path.isfile(output)):
pass
else:
# Importing the data into pandas
#pandas_df = [pd.read_csv(file,index_col=None,header=0)][0]
pandas_df = [pd.read_csv(file,header=0,low_memory=False)][0]
# Rename some columns to match the more well kNown dataset from
# http://stat-computing.org/dataexpo/2009/the-data.html
# Importing the data from pandas to vaex
vaex_df = vaex.from_pandas(pandas_df,copy_index=False)
# Export the data with vaex to hdf5
vaex_df.export_hdf5(path=output,progress=False)
接下来我制作一个大的 HDF5:
import re
import glob
import vaex
import numpy as np
def tryint(s):
try:
return int(s)
except:
return s
def alphanum_key(s):
""" Turn a string into a list of string and number chunks.
"z23a" -> ["z",23,"a"]
"""
return [ tryint(c) for c in re.split('([0-9]+)',s) ]
hdf5_list = glob.glob('c:\\Datasety\\YelowTaxi\\DataH5\\*.hdf5')
hdf5_list.sort(key=alphanum_key)
hdf5_list = np.array(hdf5_list)
#assert len(hdf5_list) == 3,"Incorrect number of files"
# This is an important step
master_df = vaex.open_many(hdf5_list)
# exporting
#master_df.export_hdf5(path='c:\\Datasety\\YelowTaxi\\DataH5\\Spojene.hd5',progress=True)
master_df.export_hdf5(path='c:\\Datasety\\YelowTaxi\\DataH5\\Spojene.hdf5',progress=True)
到目前为止,一切正常,我可以打开输出文件 Spojene.hdf5。
接下来,我将新的 .csv 附加到 Spojene.hdf5:
for file in csv_list:
#file = csv_list[0]
df2 = pd.read_csv(file,low_memory=False)
filename = 'c:\\Datasety\\YelowTaxi\\DataH5\\Spojene.hdf5'
df2.to_hdf(filename,'data',append=True)
但是,当我将新的 .csv 附加到 Spojene.hdf5 时,我无法打开它:
df = vaex.open('c:\\Datasety\\YelowTaxi\\DataH5\\Spojene.hdf5')
ValueError: 第一列的长度为 289184484,而列表的长度为 60107988
请问,我该怎么办?
解决方法
我认为这与 pandas 如何创建 hdf5 文件有关。根据 vaex 的 documentation 你不能用 vaex 打开 HDF5 文件,如果它是通过 to_hdf
pandas 方法创建的。如果您附加到现有的 HDF5 文件,我认为它是相同的。
为避免此错误,您可以重用逻辑,将 Pandas 数据帧转换为 vaex 数据帧,将其导出到 HDF5,然后使用 open_many
。这样的事情应该可以工作:
main_hdf5_file_path = "c:\\Datasety\\YelowTaxi\\DataH5\\Spojene.hdf5"
hdf5_files_created = []
for file in csv_list:
hdf5_file = file.replace(".csv",".hdf5")
# from_csv can take additional parameters to forward to pd.read_csv
# You can also use convert=True to convert it automatically to hdf5 without the export_hdf5
# Refer to https://vaex.readthedocs.io/en/docs/api.html#vaex.from_csv
df = vaex.from_csv(file)
df.export_hdf5(hdf5_file)
hdf5_files_created.append(hdf5_file)
hdf5_to_read = hdf5_files_created + [main_hdf5_file_path]
final_df = vaex.open_many(hdf5_to_read)
final_df.export_hdf5(main_hdf5_file_path)
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。