如何解决切片大文件,删除重复项并使用 Pandas 合并到输出中
所以,我有一个包含 12.5 亿个特征的地理包。该文件实际上并不包含几何图形,并且只有一个属性“id”,它是一个唯一的 id。有很多重复项,我想删除重复的“id”并仅保留唯一值。由于存在大量数据(geopackage 包含 19 GB),我进行了切片。我尝试了多处理,但没有奏效,而且会出现问题,因为我必须跟踪唯一的“id”,而多处理不允许这样做(至少据我所知)。
我有什么:
import fiona
import geopandas as gpd
import pandas as pd
# import numpy as np
slice_count = 200
start = 0
end = slice_count
fname = "path/Output.gpkg"
file_gpd = gpd.read_file(fname,rows=slice(start,end))
chunk = pd.DataFrame(file_gpd)
chunks = pd.DataFrame()
only_ids = pd.DataFrame(columns=['id'])
loop = True
while loop:
try:
# Dropping duplicates in current dataset
chunk = chunk.drop_duplicates(subset=['id'])
# Extract only unique IDS from chunk variable to save memory
only_ids_in_chunk = pd.DataFrame()
only_ids_in_chunk['id'] = chunk['id']
only_ids = only_ids.append(only_ids_in_chunk)
only_ids = only_ids.drop_duplicates(subset=['id'])
# If we want to make another file which have all values unique
# we must store somewhere what we have in chunk variable,to be able to load new chunk
# Because we must not have all chunks in memory at the same time
del chunk
# Load next chunk
start += slice_count
end += slice_count
file_gpd = gpd.read_file(fname,end))
chunk = pd.DataFrame(file_gpd)
if len(chunk) == 0:
print(len(only_ids))
loop = False
else:
pass
except Exception:
loop = False
print("Iteration is stopped")
我遇到了一个无限循环。我认为使用 if 语句会发现块的长度何时等于 0 或切片何时结束。
解决方法
所以,这是最终的脚本。我遇到的问题是,当您使用 geopandas 对 geopackage 文件进行切片时,当您走到最后时,它会从头开始并且不会停止。所以我在代码末尾添加了 if 语句来覆盖这一点。
import fiona
import geopandas as gpd
import pandas as pd
import logging
import time
slice_count = 20000000
start = 0
end = slice_count
fname = "/Output.gpkg"
chunk = gpd.read_file(fname,rows=slice(start,end),ignore_geometry=True)
chunks = pd.DataFrame()
only_ids = pd.DataFrame(columns=['id'])
loop = True
chunk_num = 1
while loop:
start_time = time.time()
# Dropping duplicates in current dataset
chunk = chunk.drop_duplicates(subset=['id'])
only_ids = only_ids.append(chunk)
only_ids = only_ids.drop_duplicates(subset=['id'])
# delete chunk to save memory
del chunk
# Load next chunk
start += slice_count
end += slice_count
chunk = gpd.read_file(fname,ignore_geometry=True)
FORMAT = '%(asctime)s:%(name)s:%(levelname)s - %(message)s'
logging.basicConfig(format=FORMAT,level=logging.INFO)
logging.info(f"Chunk {chunk_num} done")
print(f"Duration: {time.time() - start_time}")
chunk_num += 1
if len(chunk) != slice_count:
chunk = chunk.drop_duplicates(subset=['id'])
only_ids = only_ids.append(chunk)
only_ids = only_ids.drop_duplicates(subset=['id'])
del chunk
break
only_ids.to_csv('output.csv')
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。