处理大数据集时 IPython 内核重启

如何解决处理大数据集时 IPython 内核重启

我正在尝试处理混合变量（整数和字符串）的大型 csv（56、72,000），最终目标是将初始 csv 划分为更小、更易于管理的文档。鉴于数据类型的多样性，我打算将所有变量转换为字符串，然后从那里导出。当我尝试处理这些数据时，IPython 内核将在几分钟后重新启动。以下是 csv 的示例：

[ [ 'region code','state','state code','compound name','total compound' ]
[ '04','GA','13','nitrogen dioxide','5.5' ]
[ '04','Sulfur dioxide','1.3' ]
[ '05','MI','26','PM10 Primary','0.095' ]
[ '05','VOC','4.23' ]
[ '04','Elemental Carbon','0.0062' ]
[ '09','CA','05','PM2.5 secondary','0.0012' ] ]

代码：

import csv
import numpy as np

file = "/User/Downloads/large_file.csv"
   
data = list( csv.reader( open( large_file.csv ) ) )

headers = np.array( data[ 0 ][ : ],dtype = 'str' )
all_data = np.array( data[ 1:],dtype = 'str' ) #error occurring here

locate_state = np.where( headers == 'state' )

state_name = ( all_data [ :,locate_state ] )[ :,0 ]

state_mask = np.where( state_name == 'GA' )

state_data = all_data[ state_mask ]

with open( 'test_Export.csv','w',encoding = 'UTF8',newline = '') as f:
    writer = csv.writer( f )
    writer.writerow( headers )
    write.writerows( state_data )

如何处理此数据集并导出较小的子集？