微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦！

python处理csv数据的方法

分类：Python作者：编程之家

本文实例讲述了python处理csv数据的方法。分享给大家供大家参考。具体如下：

Python代码：

复制代码 代码如下:

#coding=utf-8
__author__ = 'dehua.li'
from datetime import *
import datetime
import csv
import sys
import time
import string
import os
import os.path
import pylab as plt
rootdir='/nethome/dehua.lI/OrderlifeCycleData/xingzheng'
writeFileDir="/nethome/dehua.lI/OrderlifeMyWork/xingzheng/csv"
heyueFile="/nethome/dehua.lI/OrderlifeCycleData/heyue_150128.csv"
ms_acked="1"
msg=[]
ex=[]
def getTheDate(date):
    [filenamePart1,filenamePart2]=string.split(filename,'.')
    [filenamePart11,filenamePart12,filenamePart13]=string.split(filenamePart1,'_')
    return filenamePart13
LocalTime=datetime.datetime.fromtimestamp(time.mktime(time.strptime("2014-11-04 20:59:59","%Y-%m-%d %H:%M:%s")))
for parent,dirname,filenames in os.walk(rootdir):
    for filename in filenames:
        fileNameWrite=os.path.join(writeFileDir,filename)
        print fileNameWrite
        csvfile00=open(fileNameWrite,'wb')
        writer1=csv.writer(csvfile00)
        writer1.writerow(['FeedCode','OrderId','Status','LocalTime','Time','Exchange'])
        fileName=os.path.join(parent,filename)
        [filenamePart1,'.')
        [filenamePart11,'_')
        #filenamePart11_filenamePart12_filenamePart13.filenamePart2:dongzheng_orderlifeCycleData_20150111.csv
        print fileName
        with open(fileName,'rb') as csvfile:
            reader=csv.reader(csvfile)
            CsvItem=[row for row in reader]
            for item in CsvItem:
                if item[3]=='TPO':
                    #print " filter TPO "
                    continue
                if item[12]=='Sent':
                    [tm_local,ms_local]=string.split(item[15],'.')
                    [tm_localup,ms_localup]=string.split(item[19],'.')
                    LocalTime=datetime.datetime.fromtimestamp(time.mktime(time.strptime(tm_local,"%Y-%m-%d %H:%M:%s")))
                    LocalUpdate=datetime.datetime.fromtimestamp(time.mktime(time.strptime(tm_localup,"%Y-%m-%d %H:%M:%s")))
                    tm=int(((LocalTime-LocalUpdate).seconds))*1000
                    ms_sent=str(int(ms_local)-int(ms_localup)+tm)
                    if int(ms_sent)>10*60*1000:
                        print "ms_sent>600000"
                        continue
                    if(int(ms_local)-int(ms_localup)+tm)<0:
                        print 'wrong1'
                    msg=[]
                    msg.append(item[0])
                    msg.append(item[1])
                    msg.append(item[12])
                    msg.append(item[15])
                    msg.append(ms_sent)
                    with open(heyueFile,'rb') as csvfile1:
                        reader=csv.reader(csvfile1)
                        CsvItem=[row for row in reader]
                        for Item in CsvItem:
                            if Item[1]==item[0]:
                                msg.append(Item[3])
                                writer1.writerow(msg)
                                #print 'write ok'
                                ex=Item[3]
                                break
                    csvfile1.close()
                    with open(fileName,'rb') as csvfile22:
                        reader=csv.reader(csvfile22)
                        CsvItem2=[row for row in reader]
                        for item_ in CsvItem2:
                            if item_[12]=='Acked' and item_[1]==item[1]:
                                [tm_local2,ms_local2]=string.split(item_[15],'.')
                                LocalTime2=datetime.datetime.fromtimestamp(time.mktime(time.strptime(tm_local2,"%Y-%m-%d %H:%M:%s")))
                                tm2=int(((LocalTime2-LocalTime).seconds))*1000
                                ms_acked=str(int(ms_local2)-int(ms_local)+tm2)
                                if int(ms_acked)>10*60*1000:
                                    print "MSacked>600000"
                                    continue
                                msg=[]
                                msg.append(item_[0])
                                msg.append(item_[1])
                                msg.append(item_[12])
                                msg.append(item_[15])
                                msg.append(ms_acked)
                    with open(heyueFile,'rb') as csvfile111:
                        reader=csv.reader(csvfile111)
                        CsvItem=[row for row in reader]
                        for Item in CsvItem:
                            if Item[1]==item[0]:
                                msg.append(Item[3])
                                writer1.writerow(msg)
                                #print 'write ok'
                                break
                                #print "write ok"
                    csvfile22.close()
        csvfile.close()
        csvfile00.close()

复制代码 代码如下:

#coding=utf-8
#__author__ = 'dehua.li'
from datetime import *
import datetime
import csv
import sys
import time
import string
import os
import os.path
import pylab as plt
def median(lst):
    even = (0 if len(lst) % 2 else 1) + 1
    half = (len(lst) - 1) / 2
    return sum(sorted(lst)[half:half + even]) / float(even)
def mean(lst):
    if len(lst)==0:
        return 0
    return sum(lst)/len(lst)
nightLine="21:01:00"
morningLine="09:01:00"
def getTheDate(date):
    [filenamePart1,'_')
    return filenamePart13
def afterOneMin(time):
    [tm_local,ms_local]=string.split(time,'.')
    [ymd,hms]=string.split(tm_local,' ')
    flag=0
    if hms>"21:01:00":
        flag=1
    elif hms>"09:01:00" and hms<"20:00:00":
        flag=1
    elif hms>"00:00:00" and hms<"05:00:00":
        flag=1
    return flag
rootdir="/nethome/dehua.lI/OrderlifeMyWork/xingzheng/csv"
#csvfileMaxMin = open('e:\dehua.li\csv\__xingzhenMaxMin.csv','wb')
#writer1 = csv.writer(csvfileMaxMin)
#writer1.writerow(['FeedCode','date','SentMaxTime','SentMaxLocalTime','SentMinTime','SentMinLocalTime','SentMeanTime','SentMedian','AckedMaxTime','AckedMaxLocalTime','AckedMinTime','AckedMinLocalTime','AckedMeanTime','AckedMedianTime','Exchange'])
#writer1.writerow(['FeedCode','Exchange'])
msg=[]
codeList=list()
orderList=list()
itemSentList=[]
itemAckedList=[]
FeedCode=[]
exchange=[]
zhengshangSentMedian=0
zhengshangSentMean=0
zhengshangAckedMedian=0
zhengshangAckedMean=0
dashangSentMedian=0
dashangSentMean=0
dashangAckedMedian=0
dashangAckedMean=0
shangqiSentMedian=0
shangqiSentMean=0
shangqiAckedMedian=0
shangqiAckedMean=0
zhongjinSentMedian=0
zhongjinSentMean=0
zhongjinAckedMedian=0
zhongjinAckedMean=0
zhengshangSent=[]
zhengshangAcked=[]
dashangSent=[]
dashangAcked=[]
shangqiSent=[]
shangqiAcked=[]
zhongjinSent=[]
zhongjinAcked=[]
zhengshangSentMedianAll=[]
zhengshangSentMeanAll=[]
zhengshangAckedMedianAll=[]
zhengshangAckedMeanAll=[]
dashangSentMedianAll=[]
dashangSentMeanAll=[]
dashangAckedMedianAll=[]
dashangAckedMeanAll=[]
shangqiSentMedianAll=[]
shangqiSentMeanAll=[]
shangqiAckedMedianAll=[]
shangqiAckedMeanAll=[]
zhongjinSentMedianAll=[]
zhongjinSentMeanAll=[]
zhongjinAckedMedianAll=[]
zhongjinAckedMeanAll=[]
zhengshang='0'
dashang='0'
shangqi='0'
zhongjin='0'
with open('/nethome/dehua.lI/OrderlifeCycleData/heyue_150128.csv','rb') as csvfile:
    reader=csv.reader(csvfile)
    csvItem=[row for row in reader]
    zhengshang=csvItem[300][3]
    dashang=csvItem[5][3]
    shangqi=csvItem[165][3]
    zhongjin=csvItem[435][3]
    #for item in csvItem:
    #    if item[3]==zhengshang:
     #       print item
for parent,filenames in os.walk(rootdir):
    for filename in filenames:
        fileName=os.path.join(rootdir,filename)
        csvfile1=open(fileName,'rb')
        reader=csv.reader(csvfile1)
        CsvItem=[row for row in reader]
        for item in CsvItem:
            if item[0]=='FeedCode':
                continue
            if item[0] not in codeList:
                codeList.append(item[0])
                #print CsvItem[15]
            if len(item)<=5:
                print fileName
                print item
                print '++++++++++++++++++++++++++++++'
            #if afterOneMin(item[3])==0:
            #    print item[3]
            #    continue
            if item[5]==zhengshang and item[2]=='Sent':
                zhengshangSent.append(int(item[4]))
            elif item[5]==zhengshang and item[2]=='Acked':
                zhengshangAcked.append(int(item[4]))
            elif item[5]==dashang and item[2]=='Sent':
                 dashangSent.append(int(item[4]))
            elif item[5]==dashang and item[2]=='Acked':
                 dashangAcked.append(int(item[4]))
            elif item[5]==shangqi and item[2]=='Sent':
                 shangqiSent.append(int(item[4]))
                 if int(item[4])>=600000:
                     print "------------"
                     print item
            elif item[5]==shangqi and item[2]=='Acked':
                 shangqiAcked.append(int(item[4]))
            elif item[5]==zhongjin and item[2]=='Sent':
                 zhongjinSent.append(int(item[4]))
            elif item[5]==zhongjin and item[2]=='Acked':
                 zhongjinAcked.append(int(item[4]))
            else:
                 print "wrong info"
                 print item
        if mean(shangqiSent)>420000:
            print sum(shangqiSent)
            print len(shangqiSent)
            print item
            print fileName
            print shangqiSent
        zhengshangSentMedian=median(zhengshangSent)
        zhengshangSentMean=mean(zhengshangSent)
        zhengshangAckedMedian=median(zhengshangAcked)
        zhengshangAckedMean=mean(zhengshangAcked)
        dashangSentMedian=median(dashangSent)
        dashangSentMean=mean(dashangSent)
        dashangAckedMedian=median(dashangAcked)
        dashangAckedMean=mean(dashangAcked)
        shangqiSentMedian=median(shangqiSent)
        shangqiSentMean=mean(shangqiSent)
        shangqiAckedMedian=median(shangqiAcked)
        shangqiAckedMean=mean(shangqiAcked)
        zhongjinSentMedian=median(zhongjinSent)
        zhongjinSentMean=mean(zhongjinSent)
        zhongjinAckedMedian=median(zhongjinAcked)
        zhongjinAckedMean=mean(zhongjinAcked)
        #if mean(shangqiSent)>70:
        #    print '================================'
        #    print fileName
        #print codeList
        '''
        for listItem in codeList:
            itemSentList=[]
            itemAckedList=[]
            for item in CsvItem:
                if item[0]==listItem and item[2]=='Sent':
                    itemSentList.append(int(item[4]))
                    exchange=item[5]
                elif  item[0]==listItem and item[2]=='Acked':
                    itemAckedList.append(int(item[4]))
            #print itemSentList
            itemmaxSent=max(itemSentList)
            itemminSent=min(itemSentList)
            itemAvgSent=sum(itemSentList)/len(itemSentList)
            itemmaxAcked=max(itemAckedList)
            itemminAcked=min(itemAckedList)
            itemAvgAcked=sum(itemAckedList)/len(itemAckedList)
            SentMedian=median(itemSentList)
            AckedMedian=median(itemAckedList)
            msg=[]
            msg.append(listItem)         #0
            msg.append("2015/01/14")   #1
            msg.append(itemmaxSent)     #2
            msg.append(itemminSent)     #3
            msg.append(itemAvgSent)     #4
            msg.append(SentMedian)     #5
            msg.append(itemmaxAcked)    #6
            msg.append(itemminAcked)     #7
            msg.append(itemAvgAcked)     #8
            msg.append(AckedMedian)      #9
            msg.append(exchange)          #10
            if len(msg)>15:
               print "------------------------------"
               print msg
            writer1.writerow(msg)
         '''
        zhengshangSentMedianAll.append(zhengshangSentMedian)
        zhengshangSentMeanAll.append(zhengshangSentMean)
        zhengshangAckedMedianAll.append(zhengshangAckedMedian)
        zhengshangAckedMeanAll.append(zhengshangAckedMean)
        dashangSentMedianAll.append(dashangSentMedian)
        dashangSentMeanAll.append(dashangSentMean)
        dashangAckedMedianAll.append(dashangAckedMedian)
        dashangAckedMeanAll.append(dashangAckedMean)
        shangqiSentMedianAll.append(shangqiSentMedian)
        shangqiSentMeanAll.append(shangqiSentMean)
        shangqiAckedMedianAll.append(shangqiAckedMedian)
        shangqiAckedMeanAll.append(shangqiAckedMean)
        zhongjinSentMedianAll.append(zhongjinSentMedian)
        zhongjinSentMeanAll.append(zhongjinSentMean)
        zhongjinAckedMedianAll.append(zhongjinAckedMedian)
        zhongjinAckedMeanAll.append(zhongjinAckedMean)
plt.figure(1)
plt.figure(2)
plt.figure(3)
plt.figure(4)
plt.figure(1)
plt.title('SentMean r-zhengshang b-dashang,green-shangqi grey-zhongjin')
plt.plot(range(1,len(zhengshangSentMeanAll)+1),zhengshangSentMeanAll,'r')
plt.plot(range(1,len(dashangSentMeanAll)+1),dashangSentMeanAll,'b')
plt.plot(range(1,len(shangqiSentMeanAll)+1),shangqiSentMeanAll,'g')
plt.plot(range(1,len(zhongjinSentMeanAll)+1),zhongjinSentMeanAll,'grey')
plt.savefig('/nethome/dehua.lI/OrderlifeMyWork/xingzheng/data_noTPO_in10minutes/SentMean.png')
plt.figure(2)
plt.title('SentMedian r-zhengshang b-dashang,len(zhengshangSentMedianAll)+1),zhengshangSentMedianAll,len(dashangSentMedianAll)+1),dashangSentMedianAll,len(shangqiSentMedianAll)+1),shangqiSentMedianAll,len(zhongjinSentMedianAll)+1),zhongjinSentMedianAll,'grey')
plt.savefig('/nethome/dehua.lI/OrderlifeMyWork/xingzheng/data_noTPO_in10minutes/SentMedian.png')
plt.figure(3)
plt.title('AckedMean r-zhengshang b-dashang,len(zhengshangAckedMeanAll)+1),zhengshangAckedMeanAll,len(dashangAckedMeanAll)+1),dashangAckedMeanAll,len(shangqiAckedMeanAll)+1),shangqiAckedMeanAll,len(zhongjinAckedMeanAll)+1),zhongjinAckedMeanAll,'grey')
plt.savefig('/nethome/dehua.lI/OrderlifeMyWork/xingzheng/data_noTPO_in10minutes/AckedMean.png')
plt.figure(4)
plt.title('AckedMedian r-zhengshang b-dashang,len(zhengshangAckedMedianAll)+1),zhengshangAckedMedianAll,len(dashangAckedMedianAll)+1),dashangAckedMedianAll,len(shangqiAckedMedianAll)+1),shangqiAckedMedianAll,len(zhongjinAckedMedianAll)+1),zhongjinAckedMedianAll,'grey')
plt.savefig('/nethome/dehua.lI/OrderlifeMyWork/xingzheng/data_noTPO_in10minutes/AckedMedian.png')
plt.show()
print 'over'

希望本文所述对大家的Python程序设计有所帮助。

您可能感兴趣的文章:

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

空气阻力对乒乓球运动轨迹的影响
在前一篇博客中我们介绍了加侧旋的乒乓球弧圈技术的模拟，本文我们关注的是高吊弧圈和削球弧线这两种轨迹原理，并且引入了空气阻力对乒乓球运动轨迹的影响。通过对空气阻力和马格努斯力的模拟，我们可以看到不同的弧线曲线。对于乒乓球爱好者而言，可以通过这种模拟的结果，来制定比赛中有可能用到的策略，比如低长弧圈球、
解决python报错：ModuleNotFoundError: No module named '_sysconfigdata_x86_64_conda_linux_gnu'
在近期conda的版本更新中，有可能会删除路径下的_sysconfigdata_x86_64_conda_linux_gnu模块相关备份文件，而在其他的一些软件的运行过程中有可能依赖于这个备份文件，这就会导致运行过程报错模块无法找到的问题。最终我们通过重新复制备份这个文件解决了该问题。
python3之lambda表达式
本文主要展示了一些lambda表达式的使用示例，通过这些示例，我们可以了解到lambda表达式的常用语法以及使用的场景。在一般情况下，其最大的意义还是简化了python中函数的定义方式，在可读性上远不如def所定义的函数。
使用jax加速Hamming Distance的计算
本文通过对比Jax和Numpy计算Normalized Hamming Distance的过程来对比了Jax所实现的Numpy的GPU版本所带来的加速效果。实际上在维度比较小的时候，Numpy还是有非常轻量级的优势，此时GPU的加速效果并没有很好的体现出来。但是在规模较大的输入场景下，GPU的并行加
用CUDA写出比Numpy更快的规约求和函数
我们知道GPU加速在可并行化程度比较高的算法中，能够发挥出比较大的作用，展示出明显的加速效果，而对于一些线程之间存在依赖这样的场景就不一定能够起到很大的加速作用。CUDA官方针对此类问题，提供了atomic的内置函数解决方案，包含有求和、求最大值等常用函数。而这些函数的特点就在于，线程与线程之间需要
超过Numpy的速度有多难？试试Numba的GPU加速
Numpy这个库在Python编程中非常的常用，不仅在性能上补足了Python语言的一些固有缺陷，还具有无与伦比的强大生态。但是即使都是使用Python，Numpy也未必就达到了性能的巅峰，对于我们自己日常中使用到的一些计算的场景，针对性的使用CUDA的功能来进行GPU的优化，是可以达到比Numpy
使用autopep8自动规范化python3代码
本文主要通过介绍两个python中常用的编码规范格式化工具：autopep8和black来讲解python编程中一些快速处理编程规范问题的方法，同时也说明了这些软件的局限性。编程规范也是人为制定的，事实上在实际项目中，也不是所有的编程规范都需要满足，这就需要项目的组织者或者领导者有自己的基本判断。结
乒乓球之弧圈理论
本文通过对马格努斯力的模拟，来理解乒乓球的弧圈原理。并且在这个理论基础之上对比了几种场景下的乒乓球轨迹，比如乒乓球的运动速度，或者日常所说的撞击对乒乓球轨迹的影响。还有乒乓球的角速度，也就是日常我们所说的摩擦对乒乓球轨迹的影响。还有一个在乒乓球界堪称变革的小球改大球对乒乓球这项运动可能带来的影响。由
在pyqt5中展示pyecharts生成的图像
本文通过一个实际的散点图案例，展示了如何使用pyqt5嵌套一个pyecharts图层的方法，通过这个技巧，可以在pyqt5的框架中也实现精美的数据可视化的功能模块。
python并行计算之mpi4py的安装与基本使用
在这篇文章中，我们并没有介绍非常多的MPI的应用，其实在普通的并行或者分布式任务中，掌握了各个进程的任务处理过程以及进程间的通信方法就足够了。总体来说，MPI是一个非常通用也非常高效的并行计算软件。有了这些专业的并行化任务调度软件，我们就可以专注于专业任务的代码和算法上，而不需要过多的去关注并行任务
苹果市值2025年有望达4万亿美元
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot