zoukankan      html  css  js  c++  java
  • python3 读取大文件分解成若干小文件

    有个数据实在太大了,有1.7G,打开慢,改文件也慢,我们将其分解成若干个中等文件

    #!/usr/bin/env python3
    # -*- coding: utf-8 -*-
     
    f = open("123.sql",'r',encoding='utf-8')
    readlist = []
    i = 0
    i=i+1
    filename = "mytest_{0}.sql".format(i)
    wf=open(filename, 'w', encoding='utf-8')
    linecount=0
    while 1:
        try:
            line = f.readline()
            wf.write(line)
            linecount=linecount+1       
            if(linecount >;= 200000):
                linecount = 0
                print(filename)
                i=i+1
                wf.close()
                filename = "mytest_{0}.sql".format(i)
                wf=open(filename, 'w', encoding='utf-8')                
            if not line:
                break        
        except Exception as e:
            print("read except:" + str(e))
            continue
            #print("read except:" + str(e))
     
    f.close()
     
    print(filename)
    wf.close()        
     
     

    有一个问题,之前的sql文件保存为utf-8格式,实际上在文件中夹杂中部分乱码,读取时会出错,所以需要try屏蔽

  • 相关阅读:
    分布式消息系统Kafka初步
    webService 学习
    一个周末掌握IT前沿技术之node.js篇
    mount
    dd
    fsck
    resize2fs
    mkfs
    parted
    tune2fs
  • 原文地址:https://www.cnblogs.com/luhouxiang/p/4940056.html
Copyright © 2011-2022 走看看