zoukankan      html  css  js  c++  java
  • 小知识:Linux如何删除大量小文件

    环境:RHEL 6.5 + Oracle 11.2.0.4
    需求:使用df -i巡检发现Inodes使用率过高,需要清理删除文件来解决。如果Inodes满,该目录将不能写,即使df -h查看还有剩余空间。

    1.问题现象

    Oracle的adump下记录的是sys的登陆审计信息,特点是小碎文件非常多,经常会遇到使用rm -rf *命令删除不了,报错-bash: /bin/rm: Argument list too long
    这是因为通配符*在执行时会替换为具体的文件名,例如rm -rf file1 file2 file3 ...,如果文件数量过多,就容易出现这个错误。
    比如在下面的环境中,adump目录下文件已达到114万+,执行rm -rf *命令时就会报这个错误:

    [oracle@jystdrac2 adump]$ pwd
    /opt/app/oracle/admin/crmdb/adump
    [oracle@jystdrac2 adump]$ ls|wc -l
    1149787
    [oracle@jystdrac2 adump]$ rm -rf *
    -bash: /bin/rm: Argument list too long
    [oracle@jystdrac2 adump]$ du -sh
    4.4G    
    

    2.解决方案

    清楚了问题现象,解决方案就从除去rm -rf *命令的方式之外,还有哪些方法可用,如果通过网络搜索,可能会找到结合find命令再去执行rm的方式,但其实效率非常差,具体写法这里就不列出了,因为我们通常也不会这样处理。那么如何较为效率的删除大批小文件呢?结合网络的经验,并实测验证,最终总结了两种常见的解决方案,效率上也都尚可。

    方案一:巧用rsync的方式达到删除目的

    建立一个空文件夹,使用rsync --delete-before -d <空文件夹> <需要清理删除小文件的目录>命令最终达到删除大批小文件的目的。下面演示具体操作:

    [oracle@jystdrac2 adump]$ mkdir /data/null
    [oracle@jystdrac2 adump]$ ls -l /data/null
    total 0
    [oracle@jystdrac2 ~]$ nohup rsync --delete-before -d /data/null/ /opt/app/oracle/admin/crmdb/adump/ &
    

    使用man rsync查看rsync命令相关的参数说明如下:

    -d, --dirs                  transfer directories without recursing
    --delete-before         receiver deletes before transfer (default)
    

    方案二:使用find命令的delete参数

    使用find <需要清理删除小文件的目录> -type f -delete命令直接删除大批小文件。

    使用man find查看find命令相关的参数说明如下:

           -type c
                  File is of type c:
    
                  b      block (buffered) special
    
                  c      character (unbuffered) special
    
                  d      directory
    
                  p      named pipe (FIFO)
    
                  f      regular file
    
                  l      symbolic  link; this is never true if the -L option or the -follow option is in effect, unless the symbolic link is broken.  If you want to
                         search for symbolic links when -L is in effect, use -xtype.
    
                  s      socket
    
                  D      door (Solaris)
    
           -delete
                  Delete files; true if removal succeeded.  If the removal failed, an error message is issued.  If -delete fails, find’s exit status will be nonzero
                  (when it eventually exits).  Use of -delete automatically turns on the ‘-depth’ option.
    
                  Warnings:  Don’t forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything
                  below the starting points you specified.  When testing a find command line that you later intend to use with -delete, you should explicitly  spec-
                  ify -depth in order to avoid later surprises.  Because -delete implies -depth, you cannot usefully use -prune and -delete together.
    

    下面演示具体操作:

    [oracle@jystdrac1 adump]$ nohup find /opt/app/oracle/admin/crmdb/adump/ -type f -delete &
    

    可以参考下面的命令来简单监控删除过程中Inodes使用率的变化:

    while true; do df -i /; sleep 10; done
    

    比如我这里节点jystdrac1使用的find方法,节点jystdrac2使用的rsync方法,实际观察Inodes释放速度区别并不大:

    # 使用的find方法,观察Inodes释放速度:
    [oracle@jystdrac1 ~]$ while true; do df -i /; sleep 10; done
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1519124 287772   85% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1519015 287881   85% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1513880 293016   84% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1511132 295764   84% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1502434 304462   84% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1494583 312313   83% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1489111 317785   83% /
    Filesystem                        Inodes   IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 1487629 319267   83% /
    
    # 使用的rsync方法,观察Inodes释放速度:
    [oracle@jystdrac2 ~]$ while true; do df -i /; sleep 10; done
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 963029 843867   54% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 955037 851859   53% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 953088 853808   53% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 950523 856373   53% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 948754 858142   53% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 944613 862283   53% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 942619 864277   53% /
    Filesystem                        Inodes  IUsed  IFree IUse% Mounted on
    /dev/mapper/vg_linuxbase-lv_root 1806896 938510 868386   52% /
    

    既然两种方式差异不算大,那就根据需求或个人习惯选择即可。我自己更倾向于使用方案二,因为这样无需创建空目录,操作上也更直观。
    最后再总结下删除大量小文件的方法:

    # 方案一:
    mkdir <空文件夹>
    rsync --delete-before -d <空文件夹> <需要清理删除小文件的目录>
    # 方案二:
    find <需要清理删除小文件的目录> -type f -delete
    

    相对来说这两种方式都比较效率,但由于整体小文件也是比较多,所以实际可以选择nohup放到后台执行。

  • 相关阅读:
    POJ 2251 Dungeon Master
    HDU 3085 Nightmare Ⅱ
    CodeForces 1060 B Maximum Sum of Digits
    HDU 1166 敌兵布阵(树状数组)
    HDOJ 2050 折线分割平面
    HDU 5879 Cure
    HDU 1878 欧拉回路
    HDU 6225 Little Boxes
    ZOJ 2971 Give Me the Number
    HDU 2680 Choose the best route
  • 原文地址:https://www.cnblogs.com/jyzhao/p/13308221.html
Copyright © 2011-2022 走看看