zoukankan      html  css  js  c++  java
  • 【解决】hive动态添加partitions不能超过100的问题

    Author: kwu 

    【解决】hive动态添加partitions不能超过100的问题,全量动态生成partitions超过100会出现例如以下异常:

    The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

    解决100限制,可设置例如以下參数:

    set hive.exec.dynamic.partition.mode=nonstrict;
    SET hive.exec.max.dynamic.partitions=100000; 
    SET hive.exec.max.dynamic.partitions.pernode=100000; 
    insert overwrite table ods.cms_entity PARTITION (DAY)
    select 
    ENTITY_ID  ,
    ENTITY_NAME  ,
    ENTITY_DESC  ,
    ENTITY_TYPE  ,
    ENTITY_PID  ,
    ENTITY_TIME  ,
    ENTITY_PRIORITY  ,
    ENTITY_STATUS  ,
    ENTITY_CHANNEL  ,
    ENTITY_EDITOR  ,
    ENTITY_TEMPLATE  ,
    ENTITY_URL    ,
    ENTITY_CATEGORY  ,
    ENTITY_PARAM  ,
    ENTITY_SHORTNAME  ,
    ENTITY_SUBTYPE  ,
    ENTITY_COMPDELAY  ,
    day
    from stage.cms_entity_by_day;
    


    设置动态partitions为100000,该语句插入几年按天的partitions超过1000个,成功运行语句。

  • 相关阅读:

    es -Aggregations
    散列表学习
    SpringCloud (一)——基本的搭建
    1.5 中不支持 lambda 表达 java
    区块链 (一)——基础
    线性表
    Redis(四)——消息队列
    rune is alias of int32
    wall time
  • 原文地址:https://www.cnblogs.com/lcchuguo/p/5274469.html
Copyright © 2011-2022 走看看