使用文件列表构建 tf.data 管道，即pickle 数据框

分类：编程问答

如何解决使用文件列表构建 tf.data 管道，即pickle 数据框

我无法为腌制数据帧列表构建 tf.data 管道（在 python 3.7.7 和 Windows 10 上使用 Tensorflow 2.1.0）。要开始使用以下代码：

import numpy as np
import pandas as pd
import os,pickle
import tensorflow as tf
from tensorflow.keras.preprocessing.sequence import pad_sequences
from sklearn.preprocessing import MultiLabelBinarizer

MAX_LEN = 8
NULL_VALUE = -1

# creating the sample dataframe
df = pd.DataFrame({'SOURCE': [[0,0],[0,1,2,2],1],[-1,-1,-1],1]],'TIME': [[0,7,7]],'DATE': [[0,'CAT_AF1': [[1,3,4],[1,4,3],'CAT_AF2': [[1,'PROD': [[1,'PROD_Y': [[1,[1],[7],[3],[0],[2]]})

FEATURES = [col for col in df.columns if col != 'PROD_Y']

# saving the same dataframe as multiple pickle files
os.mkdir('data')
os.mkdir(os.path.join('data','pickles'))

for n in range(10):
    with open(os.path.join('data','pickles','pickle-n-{n}.pkl'.format(n=n)),'wb') as f:
        pickle.dump(df,f,pickle.HIGHEST_PROTOCOL)

def read_pickles(filename):
    with open(filename,'rb') as f:
        df_ = pickle.load(f)
    return df_

def pad_(x):
    return pad_sequences([x],dtype=np.int32,maxlen=MAX_LEN,padding='pre',truncating='pre',value=NULL_VALUE)[0]

def parse_files(filename):
    df_ = read_pickles(filename)
    MLB = MultiLabelBinarizer(classes=list(set([j for i in df_['PROD_Y'].tolist() for j in i])))
    MLB.fit(y=df_['PROD_Y'])
    for col in FEATURES:
        df_[col] = df_[col].apply(pad_)
    df_['PROD_Y'] = MLB.transform(df_['PROD_Y']).tolist()
    
    return dict(df_) 
    # return tf.data.Dataset.from_tensor_slices(dict(df_)) 
    # return df_.to_dict('list')

def parse_(filename):
    out = tf.py_function(parse_files,inp=[filename],Tout=[tf.float32 for k in FEATURES]) # not sure how to write the dict returned into Tout argument
    return out

list_of_files = [os.path.join('data',f) for f in os.listdir(os.path.join('data','pickles'))]

dataset = tf.data.Dataset.from_tensor_slices(list_of_files)
dataset_2 = dataset.map(parse_)
a = [d for d in dataset_2]

在最后一行，它在 read_pickles 中抛出以下错误：

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte

我尝试在 filename.decode('utf-8') 中使用 read_pickles 但这也会导致另一个错误： AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'decode'

当我搜索它时，我找到了这些，但它们对问题没有帮助：

完整回溯：

2021-05-16 19:59:01.568800: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-05-16 19:59:04.926696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-05-16 19:59:06.100711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2021-05-16 19:59:06.100787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-05-16 19:59:06.109139: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-05-16 19:59:06.114618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-05-16 19:59:06.117511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-05-16 19:59:06.124338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-05-16 19:59:06.129912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-05-16 19:59:06.142492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-05-16 19:59:06.142600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-16 19:59:06.143031: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-05-16 19:59:06.144206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2021-05-16 19:59:06.144245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-05-16 19:59:06.144265: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-05-16 19:59:06.144281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-05-16 19:59:06.144299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-05-16 19:59:06.144314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-05-16 19:59:06.144328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-05-16 19:59:06.144343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-05-16 19:59:06.144390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-16 19:59:06.866245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-16 19:59:06.866285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2021-05-16 19:59:06.866300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2021-05-16 19:59:06.866531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4755 MB memory) -> physical GPU (device: 0,name: GeForce GTX 1660 Ti,pci bus id: 0000:01:00.0,compute capability: 7.5)
2021-05-16 19:59:07.063641: W tensorflow/core/framework/op_kernel.cc:1643] Invalid argument: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
Traceback (most recent call last):

  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\ops\script_ops.py",line 234,in __call__
    return func(device,token,args)

  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\ops\script_ops.py",line 123,in __call__
    ret = self._func(*args)

  File "stack-overflow-tf-data-question.py",line 33,in parse_files
    df_ = read_pickles(filename)

  File "stack-overflow-tf-data-question.py",line 25,in read_pickles
    with open(filename,'rb') as f:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte


2021-05-16 19:59:07.063812: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at iterator_ops.cc:941 : Invalid argument: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
Traceback (most recent call last):

  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\ops\script_ops.py",'rb') as f:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte


     [[{{node EagerPyFunc}}]]
stack-overflow-tf-data-question.py:25: DeprecationWarning: path should be string,bytes,or os.PathLike,not tensorflow.python.framework.ops.EagerTensor
  with open(filename,'rb') as f:
Traceback (most recent call last):
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\eager\context.py",line 1897,in execution_mode
    yield
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\data\ops\iterator_ops.py",line 659,in _next_internal
    output_shapes=self._flat_output_shapes)
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\ops\gen_dataset_ops.py",line 2479,in iterator_get_next_sync
    _ops.raise_from_not_ok_status(e,name)
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\framework\ops.py",line 6606,in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code,message),None)
  File "<string>",line 3,in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
Traceback (most recent call last):

  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\ops\script_ops.py",'rb') as f:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte


     [[{{node EagerPyFunc}}]] [Op:IteratorGetNextSync]

During handling of the above exception,another exception occurred:

Traceback (most recent call last):
  File "stack-overflow-tf-data-question.py",line 52,in <module>
    a = [d for d in dataset_2]
  File "stack-overflow-tf-data-question.py",in <listcomp>
    a = [d for d in dataset_2]
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\data\ops\iterator_ops.py",line 630,in __next__
    return self.next()
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\data\ops\iterator_ops.py",line 674,in next
    return self._next_internal()
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\data\ops\iterator_ops.py",line 665,in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec,ret)
  File "C:\Users\D\Miniconda3\envs\Dev\lib\contextlib.py",line 130,in __exit__
    self.gen.throw(type,value,traceback)
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\eager\context.py",line 1900,in execution_mode
    executor_new.wait()
  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\eager\executor.py",line 67,in wait
    pywrap_tensorflow.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
Traceback (most recent call last):

  File "C:\Users\D\Miniconda3\envs\Dev\lib\site-packages\tensorflow_core\python\ops\script_ops.py",'rb') as f:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte


     [[{{node EagerPyFunc}}]]

感谢您的帮助！ TIA！

解决方法

将 read_pickles 函数替换为

def read_pickles(filename):
    with open(filename,'rb',encoding="utf8") as f:
        df_ = pickle.load(f)
    return df_

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
VM VirtualBox中安装centos时报错：设置基础软件仓库时出错
参考1 参考2 解决方案 # 点击安装源 协议选择 http:// 路径填写 mirrors.aliyun.com/centos/8.3.2011/BaseOS/x86_64/os URL类型 软件库URL 其他路径 # 版本 7 mirrors.aliyun.com/centos/7/os/x86
kafka常见问题
报错1 [root@slave1 data_mocker]# kafka-console-consumer.sh --bootstrap-server slave1:9092 --topic topic_db [2023-12-19 18:31:12,770] WARN [Consumer clie
hive常见问题
错误1 # 重写数据 hive (edu)&gt; insert overwrite table dwd_trade_cart_add_inc &gt; select data.id, &gt; data.user_id, &gt; data.course_id, &gt; date_format(
装载数据时报错：Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
错误1 hive (edu)&gt; insert into huanhuan values(1,&#39;haoge&#39;); Query ID = root_20240110071417_fe1517ad-3607-41f4-bdcf-d00b98ac443e Total jobs = 1
flume常见问题
报错1：执行到如下就不执行了，没有显示Successfully registered new MBean. [root@slave1 bin]# /usr/local/software/flume-1.9.0/bin/flume-ng agent -n a1 -c /usr/local/softwa
hadoop集群常见错误
虚拟及没有启动任何服务器查看jps会显示jps，如果没有显示任何东西 [root@slave2 ~]# jps 9647 Jps 解决方案 # 进入/tmp查看 [root@slave1 dfs]# cd /tmp [root@slave1 tmp]# ll 总用量 48 drwxr-xr-x. 2
hive常见错误
报错1 hive&gt; show databases; OK Failed with exception java.io.IOException:java.lang.RuntimeException: Error in configuring object Time taken: 0.474 se
Linux常见错误
报错1 [root@localhost ~]# vim -bash: vim: 未找到命令 安装vim yum -y install vim* # 查看是否安装成功 [root@hadoop01 hadoop]# rpm -qa |grep vim vim-X11-7.4.629-8.el7_9.x
使用sqoop一直卡在：mapreduce.Job: Running job: job_1703173956074_0001
修改hadoop配置 vi /usr/local/software/hadoop-2.9.2/etc/hadoop/yarn-site.xml # 添加如下 &lt;configuration&gt; &lt;property&gt; &lt;name&gt;yarn.nodemanager.res
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot