zoukankan      html  css  js  c++  java
  • 如何使用临时文件?

    需求:
    某项目中,我们传感器采集数据,每收集到1G数据后,做数据分析,最终只保存分析结果。这样很大的临时数据如果常驻内存,将消耗大量内存资源。我们可以使用临时文件来存储这些临时数据(外部存储:硬盘)
    临时文件不用命名,县城关闭后会自动删除。

    思路:
    使用标准库中的tempfile下的TemporaryFile,NamedTemporaryFile

    代码:

    from tempfile import TemporaryFile , NamedTemporaryFile # 这两个都是创建临时文件,只是named这个可以对临时文件进行命名
    # 创建一个临时文件的对象,在文件系统中是找不到这个临时文件的,只能由对象f来访问
    f = TemporaryFile()
    # 生成临时数据并放入临时文件中去
    f.write(b'abcdef'* 100000)  # 注意python3中要加b,写入到磁盘中去
    f.seek(0)  # 把文件的指针移到文件的开头
    # 根据具体的需求,把临时文件中的数据读入到内存中
    f.read(100) # b'abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcd'
    
    # 创建一个可以在文件系统中找到的临时文件,使用NamedTemporaryFile
    ntf = NamedTemporaryFile()  # 返回的不直接是一个文件对象
    print(ntf.name)  # 描述了在文件系统下的name属性
    ntf = NamedTemporaryFile(delete=False) # 这样在临时文件关闭后,不会进行自动删除
    
    ==============================================================================
    
    >>> from tempfile import TemporaryFile,NamedTemporaryFile
    
    >>> TemporaryFile?
    Signature:
    TemporaryFile(
        mode='w+b',
        buffering=-1,
        encoding=None,
        newline=None,
        suffix=None,
        prefix=None,
        dir=None,
    )
    Docstring:
    Create and return a temporary file.
    Arguments:
    'prefix', 'suffix', 'dir' -- as for mkstemp.
    'mode' -- the mode argument to io.open (default "w+b").
    'buffering' -- the buffer size argument to io.open (default -1).
    'encoding' -- the encoding argument to io.open (default None)
    'newline' -- the newline argument to io.open (default None)
    The file is created as mkstemp() would do it.
    
    Returns an object with a file-like interface.  The file has no
    name, and will cease to exist when it is closed.
    File:      /usr/lib/python3.6/tempfile.py
    Type:      function
    
    >>> tf = TemporaryFile()
    
    >>> tf.write(b'*' * 1024 * 1024)
    1048576
    
    >>> tf.write(b'*' * 1024 * 1024)
    1048576
    
    >>> tf.write(b'*' * 1024 * 1024)
    1048576
    
    >>> tf.write(b'*' * 1024 * 1024)
    1048576
    
    >>> tf.seek(0)
    0
    
    >>> tf.read(512)
    b'********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************'
    
    >>> tf.close()
    
    >>> tf = TemporaryFile()
    
    >>> tf.write(b'*' * 1024 * 1024 * 1024)
    1073741824
    
    >>> tf.close()
    
    >>> TemporaryFile
    <function tempfile.TemporaryFile(mode='w+b', buffering=-1, encoding=None, newline=None, suffix=None, prefix=None, dir=None)>
    
    >>> import os
    
    >>> NamedTemporaryFile?
    Signature:
    NamedTemporaryFile(
        mode='w+b',
        buffering=-1,
        encoding=None,
        newline=None,
        suffix=None,
        prefix=None,
        dir=None,
        delete=True,
    )
    Docstring:
    Create and return a temporary file.
    Arguments:
    'prefix', 'suffix', 'dir' -- as for mkstemp.
    'mode' -- the mode argument to io.open (default "w+b").
    'buffering' -- the buffer size argument to io.open (default -1).
    'encoding' -- the encoding argument to io.open (default None)
    'newline' -- the newline argument to io.open (default None)
    'delete' -- whether the file is deleted on close (default True).
    The file is created as mkstemp() would do it.
    
    Returns an object with a file-like interface; the name of the file
    is accessible as its 'name' attribute.  The file will be automatically
    deleted when it is closed unless the 'delete' argument is set to False.
    File:      /usr/lib/python3.6/tempfile.py
    Type:      function
    
    >>> ntf = NamedTemporaryFile()
    
    >>> ntf.name
    '/tmp/tmph3tgjae_'
    
    >>> ll /tmp/tmph3tgjae_
    -rw------- 1 richardo 0 11月  4 09:59 /tmp/tmph3tgjae_
    
    >>> import tempfile
    
    >>> tempfile.gettempdir()
    '/tmp'
    
    >>> tempfile.gettempprefix()
    'tmp'
    
    >>> ntf.close()
    
    >>> ll /tmp/tmph3tgjae_
    ls: 无法访问'/tmp/tmph3tgjae_': 没有那个文件或目录
    
    >>> ntf2 = NamedTemporaryFile(delete=False)
    
    >>> ntf2.name
    '/tmp/tmpnpjilyx5'
    
    >>> ll  /tmp/tmpnpjilyx5
    -rw------- 1 richardo 0 11月  4 10:01 /tmp/tmpnpjilyx5
    
    >>> ntf2.close()
    
    >>> ll  /tmp/tmpnpjilyx5
    -rw------- 1 richardo 0 11月  4 10:01 /tmp/tmpnpjilyx5
    
    
  • 相关阅读:
    红黑树(二)插入
    HDU 3415 Max Sum of Max-K-sub-sequence(单调队列)
    Codeforces 433 Div.2(A、B、C、D)
    Codeforces 846D Monitor(简单二分+二维BIT)
    hihoCoder 1403 后缀数组一·重复旋律(后缀数组+单调队列)
    CF 787D Legacy(线段树思想构图+最短路)
    HDU 6165 FFF at Valentine(Tarjan缩点+拓扑排序)
    Codeforces 165E Compatible Numbers(二进制+逆序枚举)
    Codeforces 672D Robin Hood(二分好题)
    HITOJ 2739 The Chinese Postman Problem(欧拉回路+最小费用流)
  • 原文地址:https://www.cnblogs.com/Richardo-M-Q/p/13326340.html
Copyright © 2011-2022 走看看