zoukankan      html  css  js  c++  java
  • python里如何计算大文件的md5

    在python3中,有了一个hashlib,可以用来计算md5,这里先给出一个简单的例子:

    import hashlib
    
    sstr="i love hanyu"
    print(hashlib.md5(sstr).hexdigest())

    很遗憾的,出错了,错误信息是:

    C:Python35python.exe C:/pylearn/bottlelearn/3.py
    Traceback (most recent call last):
      File "C:/pylearn/bottlelearn/3.py", line 4, in <module>
        print(hashlib.md5(sstr).hexdigest())
    TypeError: Unicode-objects must be encoded before hashing
    
    Process finished with exit code 1

    这里主要是考虑到传入的编码不同,会导致md5出问题,所以,要求传入前进行统一的编码,修改如下:

     import hashlib
     hashlib.sha256(str(random.getrandbits(256)).encode('utf-8')).hexdigest()
    import hashlib
    
    with open(hash_file) as file:
        control_hash = file.readline().rstrip("
    ")
    
    wordlistfile = open(wordlist, "rb")
    # ...
    for line in wordlistfile:
        if hashlib.md5(line.rstrip(b'
    
    ')).hexdigest() == control_hash:

    下面,来看看如何计算大文件的md5,如果只是简单的把文件都入到内存中,大文件会导致出现大问题,编码如下:

    import hashlib
    
    def hash_bytestr_iter(bytesiter, hasher, ashexstr=False):
        for block in bytesiter:
            hasher.update(block)
        return (hasher.hexdigest() if ashexstr else hasher.digest())
    
    def file_as_blockiter(afile, blocksize=65536):
        with afile:
            block = afile.read(blocksize)
            while len(block) > 0:
                yield block
                block = afile.read(blocksize)
    
    
    [(fname, hash_bytestr_iter(file_as_blockiter(open(fname, 'rb')), hashlib.md5()))
        for fname in fnamelst]
  • 相关阅读:
    hiho150周
    hdu1011
    hiho1055/hdu1561
    bat脚本启动exe并打开文件后退出 + 中文乱码
    hiho1080
    hiho1079
    java异常处理——基础篇
    找不到要编译的文件——path环境变量配置
    MVC——studying
    轻松搞定EasyUI
  • 原文地址:https://www.cnblogs.com/aomi/p/7047214.html
Copyright © 2011-2022 走看看