如何将Dask Delayed应用于以下数据类型转换函数？

分类：编程问答

如何解决如何将Dask Delayed应用于以下数据类型转换函数？

我正在使用庞大的数据集解决https://www.kaggle.com/c/ieee-fraud-detection问题。因此，在进行任何机器学习之前，我想通过将每个属性都设置为正确的类型来减少数据集的大小。所以下面是代码片段：

def reduce_mem_usage(df):
    """ iterate through all the columns of a dataframe and modify the data type
        to reduce memory usage.        
    """
    start_mem = df.memory_usage().sum().compute() / 1024**2
    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))

    for col in df.columns:
        col_type = df[col].dtype

        if col_type != object:
            c_min = df[col].min().compute()
            c_max = df[col].max().compute()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = df[col].astype(np.int16)
                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                    df[col] = df[col].astype(np.int32)
                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                    df[col] = df[col].astype(np.int64)  
            else:
                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                    df[col] = df[col].astype(np.float16)
                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = df[col].astype(np.float32)
                else:
                    df[col] = df[col].astype(np.float64)
        else: df[col] = df[col].astype('category')

    end_mem = df.memory_usage().sum().compute()/ 1024**2
    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))

    return df

我通过了如下数据集：（test_transaction是一个简单的数据框）

test_transaction = reduce_mem_usage(test_transaction)
test_transaction.to_csv(base + 'test_transaction.csv',single_file = True)

问题是，它是如此之大以至于需要永远。因此，我决定使用Dask.Delayed将其并行化。所以我写了下面的代码：

from dask import delayed

def reduce_mem_usage(df):
    """ iterate through all the columns of a dataframe and modify the data type
        to reduce memory usage.        
    """
    start_mem = df.memory_usage().sum() / 1024**2
    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))

    for col in df.columns:
        col_type = df[col].dtype

        if col_type != object:
            c_min = df[col].min()
            c_max = df[col].max()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = delayed(df[col].astype)(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = delayed(df[col].astype)(np.int16)
                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                    df[col] = delayed(df[col].astype)(np.int32)
                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                    df[col] = delayed(df[col].astype)(np.int64)  
            else:
                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                    df[col] = delayed(df[col].astype)(np.float16)
                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = delayed(df[col].astype)(np.float32)
                else:
                    df[col] = delayed(df[col].astype)(np.float64)
        else: df[col] = delayed(df[col].astype)('category')

    Sum = 0
    for col in df.columns:
        Sum += delayed(df[col].memory_usage)()
     
    print(Sum.compute()/ 1024**2)
    
    #end_mem = df.memory_usage().sum()/ 1024**2
    #print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
    #print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))

    return df

我通过了：

#base = 'E:\Study Material\Python_Machine_AI\Machine Learning\Python_ML_programs\IEEE_Fraud_Detection\\'
test = reduce_mem_usage(test_identity.compute())
#test_identity.to_csv(base + 'test_identity.csv',single_file = True)

我在这里没有得到任何结果，它表明：（无变化）

Memory usage of dataframe is 44.39 MB
44.394248962402344

当我执行以下操作时：

test.head()

显示：

TransactionID   id-01   id-02   id-03   id-04   id-05   id-06   id-07   id-08   id-09   ... id-31   id-32   id-33   id-34   id-35   id-36   id-37   id-38   DeviceType  DeviceInfo
0   Delayed('astype-039b7a66-2d77-43f2-a258-a412ad...   Delayed('astype-0125ce2c-b588-4a6c-ab32-53176b...   Delayed('astype-5979d7a0-d0e8-44b5-bd1a-28fd25...   Delayed('astype-544f031f-ef72-4d62-9db0-57aa7f...   Delayed('astype-26f740dc-2480-48b0-b754-f83434...   Delayed('astype-d42d8543-f23f-4829-a6c5-66409c...   Delayed('astype-878fd007-6e16-4ae8-bf57-1c8eb9...   Delayed('astype-2670a2b2-60f2-4c52-8ae3-306311...   Delayed('astype-3af3977e-860c-4f8e-b657-2e46e3...   Delayed('astype-9295cd86-9806-45b0-993f-4ed362...   ... Delayed('astype-a1478260-6344-4855-b7c3-bc317e...   Delayed('astype-13aa3bbd-cab7-421b-b7ec-f78371...   Delayed('astype-be95a488-5174-40f0-9bf7-a36bbf...   Delayed('astype-b3f81da4-b137-4095-a1b0-6a8411...   Delayed('astype-d04e8ed3-7690-47b5-987a-50b667...   Delayed('astype-0a70e20d-015e-4905-a14d-0ea9a7...   Delayed('astype-34e55bf4-20e3-4e96-bc87-25de00...   Delayed('astype-26797d19-0532-4f5b-8d2d-d7f295...   Delayed('astype-dedcc518-d216-491e-a124-bf2560...   Delayed('astype-3d28bd90-3aa3-47d8-a066-963c42...
1   Delayed('astype-039b7a66-2d77-43f2-a258-a412ad...   Delayed('astype-0125ce2c-b588-4a6c-ab32-53176b...   Delayed('astype-5979d7a0-d0e8-44b5-bd1a-28fd25...   Delayed('astype-544f031f-ef72-4d62-9db0-57aa7f...   Delayed('astype-26f740dc-2480-48b0-b754-f83434...   Delayed('astype-d42d8543-f23f-4829-a6c5-66409c...   Delayed('astype-878fd007-6e16-4ae8-bf57-1c8eb9...   Delayed('astype-2670a2b2-60f2-4c52-8ae3-306311...   Delayed('astype-3af3977e-860c-4f8e-b657-2e46e3...   Delayed('astype-9295cd86-9806-45b0-993f-4ed362...   ... Delayed('astype-a1478260-6344-4855-b7c3-bc317e...   Delayed('astype-13aa3bbd-cab7-421b-b7ec-f78371...   Delayed('astype-be95a488-5174-40f0-9bf7-a36bbf...   Delayed('astype-b3f81da4-b137-4095-a1b0-6a8411...   Delayed('astype-d04e8ed3-7690-47b5-987a-50b667...   Delayed('astype-0a70e20d-015e-4905-a14d-0ea9a7...   Delayed('astype-34e55bf4-20e3-4e96-bc87-25de00...   Delayed('astype-26797d19-0532-4f5b-8d2d-d7f295...   Delayed('astype-dedcc518-d216-491e-a124-bf2560...   Delayed('astype-3d28bd90-3aa3-47d8-a066-963c42...
2   Delayed('astype-039b7a66-2d77-43f2-a258-a412ad...   Delayed('astype-0125ce2c-b588-4a6c-ab32-53176b...   Delayed('astype-5979d7a0-d0e8-44b5-bd1a-28fd25...   Delayed('astype-544f031f-ef72-4d62-9db0-57aa7f...   Delayed('astype-26f740dc-2480-48b0-b754-f83434...   Delayed('astype-d42d8543-f23f-4829-a6c5-66409c...   Delayed('astype-878fd007-6e16-4ae8-bf57-1c8eb9...   Delayed('astype-2670a2b2-60f2-4c52-8ae3-306311...   Delayed('astype-3af3977e-860c-4f8e-b657-2e46e3...   Delayed('astype-9295cd86-9806-45b0-993f-4ed362...   ... Delayed('astype-a1478260-6344-4855-b7c3-bc317e...   Delayed('astype-13aa3bbd-cab7-421b-b7ec-f78371...   Delayed('astype-be95a488-5174-40f0-9bf7-a36bbf...   Delayed('astype-b3f81da4-b137-4095-a1b0-6a8411...   Delayed('astype-d04e8ed3-7690-47b5-987a-50b667...   Delayed('astype-0a70e20d-015e-4905-a14d-0ea9a7...   Delayed('astype-34e55bf4-20e3-4e96-bc87-25de00...   Delayed('astype-26797d19-0532-4f5b-8d2d-d7f295...   Delayed('astype-dedcc518-d216-491e-a124-bf2560...   Delayed('astype-3d28bd90-3aa3-47d8-a066-963c42...
3   Delayed('astype-039b7a66-2d77-43f2-a258-a412ad...   Delayed('astype-0125ce2c-b588-4a6c-ab32-53176b...   Delayed('astype-5979d7a0-d0e8-44b5-bd1a-28fd25...   Delayed('astype-544f031f-ef72-4d62-9db0-57aa7f...   Delayed('astype-26f740dc-2480-48b0-b754-f83434...   Delayed('astype-d42d8543-f23f-4829-a6c5-66409c...   Delayed('astype-878fd007-6e16-4ae8-bf57-1c8eb9...   Delayed('astype-2670a2b2-60f2-4c52-8ae3-306311...   Delayed('astype-3af3977e-860c-4f8e-b657-2e46e3...   Delayed('astype-9295cd86-9806-45b0-993f-4ed362...   ... Delayed('astype-a1478260-6344-4855-b7c3-bc317e...   Delayed('astype-13aa3bbd-cab7-421b-b7ec-f78371...   Delayed('astype-be95a488-5174-40f0-9bf7-a36bbf...   Delayed('astype-b3f81da4-b137-4095-a1b0-6a8411...   Delayed('astype-d04e8ed3-7690-47b5-987a-50b667...   Delayed('astype-0a70e20d-015e-4905-a14d-0ea9a7...   Delayed('astype-34e55bf4-20e3-4e96-bc87-25de00...   Delayed('astype-26797d19-0532-4f5b-8d2d-d7f295...   Delayed('astype-dedcc518-d216-491e-a124-bf2560...   Delayed('astype-3d28bd90-3aa3-47d8-a066-963c42...
4   Delayed('astype-039b7a66-2d77-43f2-a258-a412ad...   Delayed('astype-0125ce2c-b588-4a6c-ab32-53176b...   Delayed('astype-5979d7a0-d0e8-44b5-bd1a-28fd25...   Delayed('astype-544f031f-ef72-4d62-9db0-57aa7f...   Delayed('astype-26f740dc-2480-48b0-b754-f83434...   Delayed('astype-d42d8543-f23f-4829-a6c5-66409c...   Delayed('astype-878fd007-6e16-4ae8-bf57-1c8eb9...   Delayed('astype-2670a2b2-60f2-4c52-8ae3-306311...   Delayed('astype-3af3977e-860c-4f8e-b657-2e46e3...

这意味着每列都正确延迟了，但是我无法以正确的方式调用.compute（）。我希望所有列都正确转换为正确的类型，并且还要显示最终的内存大小。 我应该怎么做??

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
VM VirtualBox中安装centos时报错：设置基础软件仓库时出错
参考1 参考2 解决方案 # 点击安装源 协议选择 http:// 路径填写 mirrors.aliyun.com/centos/8.3.2011/BaseOS/x86_64/os URL类型 软件库URL 其他路径 # 版本 7 mirrors.aliyun.com/centos/7/os/x86
kafka常见问题
报错1 [root@slave1 data_mocker]# kafka-console-consumer.sh --bootstrap-server slave1:9092 --topic topic_db [2023-12-19 18:31:12,770] WARN [Consumer clie
hive常见问题
错误1 # 重写数据 hive (edu)&gt; insert overwrite table dwd_trade_cart_add_inc &gt; select data.id, &gt; data.user_id, &gt; data.course_id, &gt; date_format(
装载数据时报错：Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
错误1 hive (edu)&gt; insert into huanhuan values(1,&#39;haoge&#39;); Query ID = root_20240110071417_fe1517ad-3607-41f4-bdcf-d00b98ac443e Total jobs = 1
flume常见问题
报错1：执行到如下就不执行了，没有显示Successfully registered new MBean. [root@slave1 bin]# /usr/local/software/flume-1.9.0/bin/flume-ng agent -n a1 -c /usr/local/softwa
hadoop集群常见错误
虚拟及没有启动任何服务器查看jps会显示jps，如果没有显示任何东西 [root@slave2 ~]# jps 9647 Jps 解决方案 # 进入/tmp查看 [root@slave1 dfs]# cd /tmp [root@slave1 tmp]# ll 总用量 48 drwxr-xr-x. 2
hive常见错误
报错1 hive&gt; show databases; OK Failed with exception java.io.IOException:java.lang.RuntimeException: Error in configuring object Time taken: 0.474 se
Linux常见错误
报错1 [root@localhost ~]# vim -bash: vim: 未找到命令 安装vim yum -y install vim* # 查看是否安装成功 [root@hadoop01 hadoop]# rpm -qa |grep vim vim-X11-7.4.629-8.el7_9.x
使用sqoop一直卡在：mapreduce.Job: Running job: job_1703173956074_0001
修改hadoop配置 vi /usr/local/software/hadoop-2.9.2/etc/hadoop/yarn-site.xml # 添加如下 &lt;configuration&gt; &lt;property&gt; &lt;name&gt;yarn.nodemanager.res
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot