zoukankan      html  css  js  c++  java
  • Redis内存碎片清理

    当Redis中清理了大量的Key之后原先Redis申请的内存(used_memory_rss)将继续持有而不会释放,此时查看内存信息将会看到存在大量的内存碎片。那么,Redis的内存碎片可以清理么,该如何清理呢?

    翻看了Redis的相关资料发现,Redis4版本之后开始支持内存碎片的清理,于是进行了一次测试,内容如下:

    1.  搭建Redis

    搭建一个Redis,版本为4.0.14.搭建步骤参考历史博文或微信公众号,步骤相对简单,没有太多幺蛾子,很快便可以搭建成功。

    2.  插入一堆Key,使其内存占用很大

    可以批量写一个循环,插入大量key。

    3.  删除90%以上的key

    循环删除key或在创建key时设置过期时间,待key删除或过期之后,可以查看内存的情况。

    127.0.0.1:6379> info memory 
    # Memory
    used_memory:137040696
    used_memory_human:130.69M
    used_memory_rss:11705876480
    used_memory_rss_human:10.90G
    used_memory_peak:12091169848
    used_memory_peak_human:11.26G
    used_memory_peak_perc:1.13%
    used_memory_overhead:3473184
    used_memory_startup:786648
    used_memory_dataset:133567512
    used_memory_dataset_perc:98.03%
    total_system_memory:16862617600
    total_system_memory_human:15.70G
    used_memory_lua:37888
    used_memory_lua_human:37.00K
    maxmemory:12000000000
    maxmemory_human:11.18G
    maxmemory_policy:noeviction
    mem_fragmentation_ratio:85.42
    mem_allocator:jemalloc-4.0.3
    active_defrag_running:0
    lazyfree_pending_objects:0
     

    可以发现实际使用内存为130.69M,而Redis申请的内存为10.90G,碎片率mem_fragmentation_ratio为85.42,相当高了

    4.  清理内存碎片

    默认情况下自动清理碎片的参数是关闭的,可以按如下命令查看

    127.0.0.1:6379> config get activedefrag 
    1) "activedefrag"
    2) "no"

    启动自动清理内存碎片

    127.0.0.1:6379> config set  activedefrag yes
    OK

    开启后再查看内存信息

    127.0.0.1:6379> info memory
    # Memory
    used_memory:138029408
    used_memory_human:131.64M
    used_memory_rss:5052907520
    used_memory_rss_human:4.71G
    used_memory_peak:12091169848
    used_memory_peak_human:11.26G
    used_memory_peak_perc:1.14%
    used_memory_overhead:3752728
    used_memory_startup:786648
    used_memory_dataset:134276680
    used_memory_dataset_perc:97.84%
    total_system_memory:16862617600
    total_system_memory_human:15.70G
    used_memory_lua:37888
    used_memory_lua_human:37.00K
    maxmemory:12000000000
    maxmemory_human:11.18G
    maxmemory_policy:noeviction
    mem_fragmentation_ratio:36.61
    mem_allocator:jemalloc-4.0.3
    active_defrag_running:0
    lazyfree_pending_objects:0
     

    此时redis占用的内存used_memory_rss已降低至4.71G了,内存碎片为36.61

    5. 查看内存分配情况

    此时也可以查看内存分配情况,其中重要的指标是查看bins里的util,此时可以发现当前最大的已达到0.998(1除外)

    127.0.0.1:6379> memory malloc-stats
    ___ Begin jemalloc statistics ___
    Version: 4.0.3-0-ge9192eacf8935e29fc62fddc2701f7942b1cc02c
    Assertions disabled
    Run-time option settings:
      opt.abort: false
      opt.lg_chunk: 21
      opt.dss: "secondary"
      opt.narenas: 8
      opt.lg_dirty_mult: 3 (arenas.lg_dirty_mult: 3)
      opt.stats_print: false
      opt.junk: "false"
      opt.quarantine: 0
      opt.redzone: false
      opt.zero: false
      opt.tcache: true
      opt.lg_tcache_max: 15
    CPUs: 2
    Arenas: 8
    Pointer size: 8
    Quantum size: 8
    Page size: 4096
    Min active:dirty page ratio per arena: 8:1
    Maximum thread-cached size class: 32768
    Chunk size: 2097152 (2^21)
    Allocated: 138983464, active: 149237760, metadata: 133846144, resident: 299532288, mapped: 5274861568
    Current active ceiling: 153092096
    
    arenas[0]:
    assigned threads: 1
    dss allocation precedence: secondary
    min active:dirty page ratio: 8:1
    dirty pages: 36435:4043 active:dirty, 348100 sweeps, 737670 madvises, 4686933 purged
                                allocated      nmalloc      ndalloc    nrequests
    small:                       51173416     27946257     27800354    215995984
    large:                       20701184      6397067      6396713      7682104
    huge:                        67108864          754          753          754
    total:                      138983464     34344078     34197820    223678842
    active:                     149237760
    mapped:                    5270667264
    metadata: mapped: 130830336, allocated: 115776
    bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                       8   0         2952         4753         4384     18226389          369            1  512   1 0.720         2600         2566            1            0
                      16   1       699600      5702102      5658377     34777916        43725          392  256   1 0.435        61712        64807        22136          251
                      24   2       824592      9648532      9614174     75492339        34358          441  512   3 0.152       102398       100732        16561      1083971
                      32   3          800         1828         1803     11077167           25            1  128   1 0.195         1031          883            1            0
                      40   4          400          893          883      1285046           10            1  512   5 0.019          787          791            1            0
                      48   5         1824        10123        10085     23813168           38            1  256   3 0.148         6390         7039           18            9
                      56   6        11200       665763       665563      3654283          200           60  512   7 0.006        11325        18434         1302          135
                      64   7        44480      2221901      2221206      8140167          695           62   64   1 0.175        38634        43809        34702         1105
                      80   8      2243120      3241050      3213011     12640545        28039          154  256   5 0.711        36308        40673        12556          268
                      96   9      1248384      1173523      1160519      5632612        13004          116  128   3 0.875        17234        22442         9049          828
                     112  10       257152       497094       494798       378079         2296            9  256   7 0.996        10382        20700         1942          529
                     128  11       173056       241148       239796      1500956         1352           43   32   1 0.982        11994        19268         7491         1149
                     160  12       510880       676589       673396      1867045         3193           25  128   5 0.997        12036        20344         5279          974
                     192  13        86784       163154       162702       104853          452            8   64   3 0.882         8095        19909         2538          885
                     224  14      1050784       937414       932723       882636         4691           37  128   7 0.990        14534        21327         7237          653
                     256  15       177920       122377       121682       108334          695           44   16   1 0.987        11216        18094         7648         1025
                     320  16       158720       160687       160191      1395080          496           10   64   5 0.775         8282        19932         2458          806
                     384  17       108288        87166        86884        56781          282           10   32   3 0.881         7364        18594         2691          830
                     448  18       520576       311865       310703       235276         1162           19   64   7 0.955         9885        20257         4849         1277
                     512  19       366080       139077       138362       136708          715           91    8   1 0.982        16677        21203        17307          758
                     640  20       296320       136674       136211      1387569          463           15   32   5 0.964         8781        19327         4294         1194
                     768  21       754176       164055       163073       154637          982           62   16   3 0.989        13751        19514        10232          853
                     896  22       507136       142840       142274       104081          566           19   32   7 0.930         8831        19297         4488         1232
                    1024  23       460800        81240        80790        71715          450          113    4   1 0.995        11411        17538        20204         1932
                    1280  24       967680       142375       141619      3997592          756           48   16   5 0.984        12880        19162         8852          974
                    1536  25       806400        83240        82715      1363965          525           66    8   3 0.994        11744        17875        10296         1083
                    1792  26       507136        70931        70648        50310          283           18   16   7 0.982         8140        17915         4403         1138
                    2048  27       540672        52915        52651        40011          264          132    2   1 1             8604        16582        26425         1888
                    2560  28      1487360       115947       115366      2688774          581           73    8   5 0.994        15012        19969        14410          962
                    3072  29      1022976        70646        70313        58822          333           84    4   3 0.991        10445        17189        17546         2050
                    3584  30      1634304        76807        76351        66030          456           57    8   7 1            11190        17542         9538         1175
                    4096  31       487424        56959        56840        43668          119          119    1   1 1             9176        16863        56959            0
                    5120  32      6154240       145461       144259      2675097         1202          301    4   5 0.998        17704        22138        36076         1483
                    6144  33      1198080        45380        45185        83315          195           98    2   3 0.994         8050        16148        22721         1759
                    7168  34       824320        32518        32403      1310845          115           29    4   7 0.991         6598        14416         8063         1715
                    8192  35     18800640       355210       352915       362972         2295         2295    1   2 1            39103        42128       355210            0
                   10240  36      2232320        76168        75950        66295          218          109    2   5 1            11960        17361        37917         1769
                   12288  37      2039808        52860        52694        40296          166          166    1   3 1             8941        16632        52860            0
                   14336  38      1964032        36992        36855        24610          137           69    2   7 0.992         6879        15285        18422         1777
    large:          size ind    allocated      nmalloc      ndalloc    nrequests      curruns
                   16384  39       688128         7548         7506         7760           42
                   20480  40       880640         9391         9348      1293561           43
                   24576  41       688128         6279         6251         6367           28
                   28672  42       602112         4187         4166         4261           21
                   32768  43      2818048        17937        17851        18430           86
                   40960  44      1433600      6306450      6306415      6306450           35
                   49152  45      1277952         5117         5091         5117           26
                   57344  46       860160         5916         5901         5916           15
                   65536  47       393216         2953         2947         2953            6
                   81920  48      1310720         8244         8228         8244           16
                   98304  49       294912         5037         5034         5037            3
                  114688  50       573440         4224         4219         4224            5
                  131072  51       393216         2233         2230         2233            3
                  163840  52      1638400         2798         2788         2798           10
                  196608  53      1179648         1740         1734         1740            6
                  229376  54       229376         1162         1161         1162            1
                  262144  55       524288          872          870          872            2
                  327680  56       327680         1290         1289         1290            1
                  393216  57            0          782          782          782            0
                  458752  58            0          520          520          520            0
                  524288  59       524288          400          399          400            1
                  655360  60       655360          606          605          606            1
                  786432  61            0          371          371          371            0
                  917504  62            0          196          196          196            0
                 1048576  63      2097152          224          222          224            2
                 1310720  64      1310720          328          327          328            1
                 1572864  65            0          179          179          179            0
                 1835008  66            0           83           83           83            0
    huge:           size ind    allocated      nmalloc      ndalloc    nrequests   curhchunks
                 2097152  67            0          110          110          110            0
                 2621440  68            0          263          263          263            0
                 3145728  69            0           86           86           86            0
                 3670016  70            0           91           91           91            0
                 4194304  71            0           45           45           45            0
                 5242880  72            0           55           55           55            0
                 6291456  73            0           26           26           26            0
                 7340032  74            0           15           15           15            0
                 8388608  75            0           13           13           13            0
                10485760  76            0            8            8            8            0
                12582912  77            0           13           13           13            0
                14680064  78            0            2            2            2            0
                16777216  79            0            4            4            4            0
                20971520  80            0            2            2            2            0
                25165824  81            0            2            2            2            0
                29360128  82            0            2            2            2            0
                33554432  83            0            4            4            4            0
                41943040  84            0            2            2            2            0
                50331648  85            0            2            2            2            0
                58720256  86            0            2            2            2            0
                67108864  87     67108864            7            6            7            1
                         ---
    --- End jemalloc statistics ---

    6.  手动清理

    其实还可以手动清理,可以采用如下命令

    127.0.0.1:6379> memory purge
    OK

    7. 相关参数配置说明

    内存清理相关参数如下,可以使用config get的方式查看对应的值

    # Enabled active defragmentation
    # 碎片整理总开关
    # activedefrag yes
    
    # Minimum amount of fragmentation waste to start active defrag
    # 内存碎片达到多少的时候开启整理
    active-defrag-ignore-bytes 100mb
    
    # Minimum percentage of fragmentation to start active defrag
    # 碎片率达到百分之多少开启整理
    active-defrag-threshold-lower 10
    
    # Maximum percentage of fragmentation at which we use maximum effort
    # 碎片率小余多少百分比开启整理
    active-defrag-threshold-upper 100
    
    # Minimal effort for defrag in CPU percentage
    active-defrag-cycle-min 25
    
    # Maximal effort for defrag in CPU percentage
    active-defrag-cycle-max 75

    至此,Redis4.0.14版本的内存碎片清理就测试完成了。

  • 相关阅读:
    常见网络攻击手段原理分析
    admins.py总结比较,转
    django的表与表之间的关系详细讲解
    django中的@login_required
    安装指定版本的第三方库
    在django中使用logging
    django的manytomany总结
    manyToManyField理解和用法
    django的多对一,一对一,多对多关系
    python 的os的总结
  • 原文地址:https://www.cnblogs.com/gjc592/p/12180024.html
Copyright © 2011-2022 走看看