zoukankan      html  css  js  c++  java
  • 【解决】hive动态添加partitions不能超过100的问题

    Author: kwu 

    【解决】hive动态添加partitions不能超过100的问题,全量动态生成partitions超过100会出现例如以下异常:

    The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

    解决100限制,可设置例如以下參数:

    set hive.exec.dynamic.partition.mode=nonstrict;
    SET hive.exec.max.dynamic.partitions=100000; 
    SET hive.exec.max.dynamic.partitions.pernode=100000; 
    insert overwrite table ods.cms_entity PARTITION (DAY)
    select 
    ENTITY_ID  ,
    ENTITY_NAME  ,
    ENTITY_DESC  ,
    ENTITY_TYPE  ,
    ENTITY_PID  ,
    ENTITY_TIME  ,
    ENTITY_PRIORITY  ,
    ENTITY_STATUS  ,
    ENTITY_CHANNEL  ,
    ENTITY_EDITOR  ,
    ENTITY_TEMPLATE  ,
    ENTITY_URL    ,
    ENTITY_CATEGORY  ,
    ENTITY_PARAM  ,
    ENTITY_SHORTNAME  ,
    ENTITY_SUBTYPE  ,
    ENTITY_COMPDELAY  ,
    day
    from stage.cms_entity_by_day;
    


    设置动态partitions为100000,该语句插入几年按天的partitions超过1000个,成功运行语句。

  • 相关阅读:
    Spark2 Dataset DataFrame空值null,NaN判断和处理
    Spark2 文件处理和jar包执行
    &与&&, |与||区别
    Scala实现乘法口诀
    Hive desc
    Hive FUNCTIONS函数
    Hive show
    MySQL行列转换拼接
    MySQL字符串连接
    SQLServer之索引简介
  • 原文地址:https://www.cnblogs.com/lcchuguo/p/5274469.html
Copyright © 2011-2022 走看看