zoukankan      html  css  js  c++  java
  • hadoop fs:du & count统计hdfs文件(目录下文件)大小的用法

    hadoop fs 更多用法,请参考官网:http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html

    以下是我的使用hadoop fs -du统计文件时使用的记录:

    [t@dv00938 ~]$ hadoop fs -ls /jc_rc/rc_hive_db/llcfpd_s_join_n_over0innerdoor_lishui
    17/10/15 12:14:41 INFO hdfs.PeerCache: SocketCache disabled.
    Found 113 items
    -rwxrwx---+  3 t_user t_user_group  282171728 2017-10-14 20:38 /t_user/my_hive_db/my_hive_table_test01/000000_0
    -rwxrwx---+  3 t_user t_user_group  281446475 2017-10-14 20:38 /t_user/my_hive_db/my_hive_table_test01/000001_0
    -rwxrwx---+  3 t_user t_user_group  281021562 2017-10-14 20:38 /t_user/my_hive_db/my_hive_table_test01/000002_0
    -rwxrwx---+  3 t_user t_user_group  280834172 2017-10-14 20:38 /t_user/my_hive_db/my_hive_table_test01/000003_0
    -rwxrwx---+  3 t_user t_user_group  280411919 2017-10-14 20:38 /t_user/my_hive_db/my_hive_table_test01/000004_0
    -rwxrwx---+  3 t_user t_user_group  279749295 2017-10-14 20:38 /t_user/my_hive_db/my_hive_table_test01/000005_0
    ...
    [t@dv00938
    ~]$ hadoop fs -du /t_user/my_hive_db/my_hive_table_test01 17/10/15 12:14:54 INFO hdfs.PeerCache: SocketCache disabled. 282171728 /t_user/my_hive_db/my_hive_table_test01/000000_0 281446475 /t_user/my_hive_db/my_hive_table_test01/000001_0 281021562 /t_user/my_hive_db/my_hive_table_test01/000002_0 280834172 /t_user/my_hive_db/my_hive_table_test01/000003_0 280411919 /t_user/my_hive_db/my_hive_table_test01/000004_0 279749295 /t_user/my_hive_db/my_hive_table_test01/000005_0 ...
    [t@dv00938
    ~]$ hadoop fs -du -h /t_user/my_hive_db/my_hive_table_test01 17/10/15 12:17:05 INFO hdfs.PeerCache: SocketCache disabled. 269.1 M /t_user/my_hive_db/my_hive_table_test01/000000_0 268.4 M /t_user/my_hive_db/my_hive_table_test01/000001_0 268.0 M /t_user/my_hive_db/my_hive_table_test01/000002_0 267.8 M /t_user/my_hive_db/my_hive_table_test01/000003_0 267.4 M /t_user/my_hive_db/my_hive_table_test01/000004_0 266.8 M /t_user/my_hive_db/my_hive_table_test01/000005_0 ...
    [t@dv00938
    ~]$ hadoop fs -du -s /t_user/my_hive_db/my_hive_table_test01 17/10/15 12:16:52 INFO hdfs.PeerCache: SocketCache disabled. 29959873142 /t_user/my_hive_db/my_hive_table_test01 [t@dv00938 ~]$ hadoop fs -du -s -h /t_user/my_hive_db/my_hive_table_test01 17/10/15 12:17:17 INFO hdfs.PeerCache: SocketCache disabled. 27.9 G /t_user/my_hive_db/my_hive_table_test01

     hadoop fs -count

    [dx@d-143 conf]$ hadoop fs -count /user/dx/
           45241       110279        26340662670 /user/dx
    [dx@d-143 conf]$ hadoop fs -count /user/dx/*
              12            4              45844 /user/dx/.Trash
              15          502         2382220393 /user/dx/.sparkStaging
               1            2           28116579 /user/dx/base_data
           45212       109771        23930279854 /user/dx/streaming
  • 相关阅读:
    [iPad]PencilKit教程3、检查,修改和构造PencilKit绘图
    [iPad]PencilKit教程2、PencilKit的新功能
    [iPad]PencilKit教程1、PencilKit介绍
    为什么要谨慎使用Arrays.asList、ArrayList的subList?
    精通高并发与多线程,却不会用ThreadLocal?
    拥抱Kubernetes,再见了,SpringBoot @Scheduled
    从小公司进入大厂,我都做对了哪些事?
    总结我的Java朋友
    MySQL如何实时同步数据到ES?试试这款阿里开源的神器!
    基于Java访问数据库
  • 原文地址:https://www.cnblogs.com/yy3b2007com/p/7670550.html
Copyright © 2011-2022 走看看