zoukankan      html  css  js  c++  java
  • python 内存监控模块之memory_profiler

    0. memory_profiler是干嘛的

    This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for python programs. It is a pure python module and has the psutil module as optional (but highly recommended) dependencies.

    memory_profiler是监控python进程的神器,它可以分析出每一行代码所增减的内存状况。

    1. 入门例子

    #del3.py

    import time
    @profile
    def my_func():
        a = [1] * (10 ** 6)
        b = [2] * (2 * 10 ** 7)
        time.sleep(10)
        del b
        del a
        print "+++++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    $python -m memory_profiler del3.py
    +++++++++
    Filename: del3.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
         2   10.293 MiB    0.000 MiB   @profile
         3                             def my_func():
         4   17.934 MiB    7.641 MiB       a = [1] * (10 ** 6)
         5  170.523 MiB  152.590 MiB       b = [2] * (2 * 10 ** 7)
         6  170.527 MiB    0.004 MiB       time.sleep(10)
         7   17.938 MiB -152.590 MiB       del b
         8   10.305 MiB   -7.633 MiB       del a
         9   10.309 MiB    0.004 MiB       print "+++++++++"

    代码执行一遍,然后给出具体代码在某一步占用的内存,通过内存加减可以看出某个对象的大小。

    2. 对象不删除,直接赋值内存是否会继续增长

    #对比1

    @profile
    def my_func():
        a = 'a' * 1024 * 1024 * 1024;
        a = 'a' * 1024 * 1024
        a = 'a' * 1024
        del a
        print "+++++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    Line #    Mem usage    Increment   Line Contents
    ================================================
         1   10.293 MiB    0.000 MiB   @profile
         2                             def my_func():
         3 1034.301 MiB 1024.008 MiB       a = 'a' * 1024 * 1024 * 1024;
         4   11.285 MiB -1023.016 MiB       a = 'a' * 1024 * 1024
         5   11.285 MiB    0.000 MiB       a = 'a' * 1024
         6   11.285 MiB    0.000 MiB       del a
         7   11.289 MiB    0.004 MiB       print "+++++++++"

    #对比2

    @profile
    def my_func():
        a = 'a' * 1024 * 1024 * 1024;
        del a
        a = 'a' * 1024 * 1024
        del a
        a = 'a' * 1024
        del a
        print "+++++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    Line #    Mem usage    Increment   Line Contents
    ================================================
         1   10.293 MiB    0.000 MiB   @profile
         2                             def my_func():
         3 1034.301 MiB 1024.008 MiB       a = 'a' * 1024 * 1024 * 1024;
         4   10.297 MiB -1024.004 MiB       del a
         5   11.285 MiB    0.988 MiB       a = 'a' * 1024 * 1024
         6   11.285 MiB    0.000 MiB       del a
         7   11.285 MiB    0.000 MiB       a = 'a' * 1024
         8   11.285 MiB    0.000 MiB       del a
         9   11.289 MiB    0.004 MiB       print "+++++++++"

    结论:是否 del对象没有影响,新赋的值会替代旧的值

    3. 对象赋值是否会增加同样的内存

    #对比1

    @profile
    def my_func():
        a = 'a' * 1024 * 1024 * 1024;
        b = a
        del a
        print "+++++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    Line #    Mem usage    Increment   Line Contents
    ================================================
         1   10.293 MiB    0.000 MiB   @profile
         2                             def my_func():
         3 1034.301 MiB 1024.008 MiB       a = 'a' * 1024 * 1024 * 1024;
         4 1034.301 MiB    0.000 MiB       b = a
         5 1034.301 MiB    0.000 MiB       del a
         6 1034.305 MiB    0.004 MiB       print "+++++++++"

    #对比2

    @profile
    def my_func():
        a = 'a' * 1024 * 1024 * 1024;
        b = a
        del a
        del b
        print "+++++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    Line #    Mem usage    Increment   Line Contents
    ================================================
         1   10.297 MiB    0.000 MiB   @profile
         2                             def my_func():
         3 1034.305 MiB 1024.008 MiB       a = 'a' * 1024 * 1024 * 1024;
         4 1034.305 MiB    0.000 MiB       b = a
         5 1034.305 MiB    0.000 MiB       del a
         6   10.301 MiB -1024.004 MiB       del b
         7   10.305 MiB    0.004 MiB       print "+++++++++"

    结论,把a赋值给b,内存没有增加。但是只删除其中一个对象的时候,内存不会减。

    4. 另一种等价的启动方式

    from memory_profiler import profile
    @profile(precision=4)
    def my_func():
        a = 'a' * 1024 * 1024 * 1024;
        del a
        a = 'a' * 1024 * 1024
        del a
        a = 'a' * 1024
        del a
        print "+++++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    $python -m memory_profiler del3.py
    +++++++++
    Filename: del3.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
         2  10.3867 MiB   0.0000 MiB   @profile(precision=4)
         3                             def my_func():
         4 1034.3945 MiB 1024.0078 MiB       a = 'a' * 1024 * 1024 * 1024;
         5  10.3906 MiB -1024.0039 MiB       del a
         6  11.3789 MiB   0.9883 MiB       a = 'a' * 1024 * 1024
         7  11.3789 MiB   0.0000 MiB       del a
         8  11.3789 MiB   0.0000 MiB       a = 'a' * 1024
         9  11.3789 MiB   0.0000 MiB       del a
        10  11.3828 MiB   0.0039 MiB       print "+++++++++"

    5. 非python内置对象例子

    from memory_profiler import profile
    import networkx as nx
    
    @profile(precision=4)
    def my_func():
        a = 'a' * 1024 * 1024 * 1024;
        del a
        G = nx.Graph()
        G.add_node(1)
        G.add_nodes_from([i for i in range(10000)])
        G.add_nodes_from([i for i in range(10000, 20000)])
        G.add_edges_from([(1,2), (1,4), (2, 9), (4, 1), (3, 8)])
        del G
        print "++++++"
    
    if __name__ == '__main__':
        my_func()

    结果

    $python del3.py
    ++++++
    Filename: del3.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
         4  23.4844 MiB   0.0000 MiB   @profile(precision=4)
         5                             def my_func():
         6 1047.4922 MiB 1024.0078 MiB       a = 'a' * 1024 * 1024 * 1024;
         7  23.4883 MiB -1024.0039 MiB       del a 
         8  23.4883 MiB   0.0000 MiB       G = nx.Graph()
         9  23.4883 MiB   0.0000 MiB       G.add_node(1)
        10  31.3359 MiB   7.8477 MiB       G.add_nodes_from([i for i in range(10000)]) 
        11  36.9219 MiB   5.5859 MiB       G.add_nodes_from([i for i in range(10000, 20000)]) 
        12  36.9219 MiB   0.0000 MiB       G.add_edges_from([(1,2), (1,4), (2, 9), (4, 1), (3, 8)])
        13  25.9219 MiB -11.0000 MiB       del G
        14  25.9258 MiB   0.0039 MiB       print "++++++"

    6. 类怎么使用呢

    #del4.py

    from memory_profiler import profile
    
    class people:
        name = ''
        age = 0
        __weight = 0
    
        def __init__(self,n,a,w):
            self.name = n
            self.age = a
            self.__weight = w
    
        @profile(precision=4)
        def speak(self):
            a = 'a' * 1024
            b = 'b' * 1024 * 1024
            print("%s is speaking: I am %d years old" % (self.name,self.age))
    
    
    
    if __name__ == '__main__':
        p = people('tom', 10, 30)
        p.speak()

    结果

    $python del4.py
    tom is speaking: I am 10 years old
    Filename: del4.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
        13   9.4219 MiB   0.0000 MiB       @profile(precision=4)
        14                                 def speak(self):  
        15   9.4258 MiB   0.0039 MiB           a = 'a' * 1024
        16  10.4297 MiB   1.0039 MiB           b = 'b' * 1024 * 1024
        17  10.4336 MiB   0.0039 MiB           print("%s is speaking: I am %d years old" % (self.name,self.age)) 

    7. 随时间内存统计

    #test.py

    import time
    
    @profile
    def test1():
        n = 10000
        a = [1] * n
        time.sleep(1)
        return a
    
    @profile
    def test2():
        n = 100000
        b = [1] * n
        time.sleep(1)
        return b
    
    if __name__ == "__main__":
        test1()
        test2()

    test.py 里有两个两个待分析的函数(@profile标识),为了形象地看出内存随时间的变化,每个函数内sleep 1s,执行

    mprof run test.py

    如果执行成功,结果这样

    $ mprof run test.py
    mprof: Sampling memory every 0.1s
    running as a Python program...

    结果会生成一个.dat文件,如"mprofile_20160716170529.dat",里面记录了内存随时间的变化,可用下面的命令以图片的形式展示出来:

    mprof plot

    8. API

    memory_profiler提供很多包给第三方代码,如

    >>> from memory_profiler import memory_usage
    >>> mem_usage = memory_usage(-1, interval=.2, timeout=1)
    >>> print(mem_usage)
        [7.296875, 7.296875, 7.296875, 7.296875, 7.296875]

    memory_usage(proc=-1, interval=.2, timeout=None)返回一段时间的内存值,其中proc=-1表示此进程,这里可以指定特定的进程号;interval=.2表示监控的时间间隔是0.2秒;timeout=1表示总共的时间段为1秒。那结果就返回5个值。

    如果要返回一个函数的内存消耗,示例

    def f(a, n=100):
         import time
         time.sleep(2)
         b = [a] * n
         time.sleep(1)
         return b
    
    from memory_profiler import memory_usage
    print memory_usage((f, (2,), {'n' : int(1e6)}))

    这里执行了 f(1, n=int(1e6)) ,并返回在执行此函数时的内存消耗。

    9. 优化实例

    对比str & int

    from datetime import datetime
    @profile
    def my_func():
        beg = datetime.now()
        a = {}
        for i in range(1000000):
            a[i] = i
            #a[str(i)] = i
        print "+++++++++"
        del a
        print "+++++++++"
        end = datetime.now()
        print "time:", end - beg
    
    if __name__ == '__main__':
        my_func()

    用a[i] = i,结果

    +++++++++
    +++++++++
    time: 0:06:14.790899
    Filename: int.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
         2   14.727 MiB    0.000 MiB   @profile
         3                             def my_func():
         4   14.734 MiB    0.008 MiB       beg = datetime.now()
         5   14.734 MiB    0.000 MiB       a = {}
         6   94.031 MiB   79.297 MiB       for i in range(1000000):
         7   94.031 MiB    0.000 MiB           a[i] = i
         8                                     #a[str(i)] = i
         9   86.402 MiB   -7.629 MiB       print "+++++++++"
        10   38.398 MiB  -48.004 MiB       del a
        11   38.398 MiB    0.000 MiB       print "+++++++++"
        12   38.398 MiB    0.000 MiB       end = datetime.now()
        13   38.406 MiB    0.008 MiB       print "time:", end - beg

    用a[str(i)] = i,结果

    +++++++++
    +++++++++
    time: 0:06:00.288052
    Filename: int.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
         2   14.723 MiB    0.000 MiB   @profile
         3                             def my_func():
         4   14.730 MiB    0.008 MiB       beg = datetime.now()
         5   14.730 MiB    0.000 MiB       a = {}
         6  140.500 MiB  125.770 MiB       for i in range(1000000):
         7                                     #a[i] = i
         8  140.500 MiB    0.000 MiB           a[str(i)] = i
         9  132.871 MiB   -7.629 MiB       print "+++++++++"
        10   38.539 MiB  -94.332 MiB       del a
        11   38.539 MiB    0.000 MiB       print "+++++++++"
        12   38.539 MiB    0.000 MiB       end = datetime.now()
        13   38.547 MiB    0.008 MiB       print "time:", end - beg
  • 相关阅读:
    Android实现自适应正方形GridView
    关于ViewPager被嵌套在ScrollView中不显示的问题
    能够兼容ViewPager的ScrollView
    二二叉搜索树学习
    linux添加静态路由表,重新启动继续有效
    基于Cocos2dx + box2d 愤怒的小鸟的实现Demo
    2014牡丹江区域赛H(特里)ZOJ3826
    AIX加入能telnet远程连接方法的帐户
    函数返回值1的数量
    JQuery日记6.5 Javascript异步模式(一)
  • 原文地址:https://www.cnblogs.com/kaituorensheng/p/5669861.html
Copyright © 2011-2022 走看看