zoukankan      html  css  js  c++  java
  • hive归档分区

    归档hive历史分区不会减少hdfs存储空间,但是可以有效减轻hadoop namenode的压力,尤其在于小文件比较多的情况下。

    $mkdir $HIVE_HOME/auxlib

    $ cp /opt/cdh-5.3.6/hadoop-2.5.0/share/hadoop/tools/lib/hadoop-archives-2.5.0-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1/auxlib/hadoop-archives-2.5.0-cdh5.3.6.jar

    hive (chavin)>  set hive.archive.enabled=true;                                    
    hive (chavin)> alter table emp archive partition(country='china',state='beijing');
    intermediate.archived is hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ARCHIVED
    intermediate.original is hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ORIGINAL
    Creating data.har for hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing
    in hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing/.hive-staging_hive_2017-05-24_13-23-51_914_3548751700804069937-1/-ext-10000/partlevel
    Please wait... (this may take a while)
    Moving hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing/.hive-staging_hive_2017-05-24_13-23-51_914_3548751700804069937-1/-ext-10000/partlevel to hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ARCHIVED
    Moving hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing to hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ORIGINAL
    Moving hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ARCHIVED to hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing
    Moved: 'hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ORIGINAL' to trash at: hdfs://db01:8020/user/hadoop/.Trash/Current
    OK
    Time taken: 2.17 seconds

    可以看到,emp表下分区文件已经被打包成data.har文件了。

    image

  • 相关阅读:
    关于微信三点定位法
    PHP 取302跳转后真实 URL 的两种方法
    前端布局神器display:flex
    JS实现document.ready
    为什么无返回值的链表的插入操作头结点一定要用指向指针的指针
    常量字符串和指针
    C语言中指针数组和数组指针的区别
    二维数组简介与使用
    访问者模式
    解释器模式
  • 原文地址:https://www.cnblogs.com/wcwen1990/p/7191709.html
Copyright © 2011-2022 走看看