zoukankan      html  css  js  c++  java
  • 【解决】hive动态添加partitions不能超过100的问题

    Author: kwu 

    【解决】hive动态添加partitions不能超过100的问题,全量动态生成partitions超过100会出现例如以下异常:

    The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

    解决100限制,可设置例如以下參数:

    set hive.exec.dynamic.partition.mode=nonstrict;
    SET hive.exec.max.dynamic.partitions=100000; 
    SET hive.exec.max.dynamic.partitions.pernode=100000; 
    insert overwrite table ods.cms_entity PARTITION (DAY)
    select 
    ENTITY_ID  ,
    ENTITY_NAME  ,
    ENTITY_DESC  ,
    ENTITY_TYPE  ,
    ENTITY_PID  ,
    ENTITY_TIME  ,
    ENTITY_PRIORITY  ,
    ENTITY_STATUS  ,
    ENTITY_CHANNEL  ,
    ENTITY_EDITOR  ,
    ENTITY_TEMPLATE  ,
    ENTITY_URL    ,
    ENTITY_CATEGORY  ,
    ENTITY_PARAM  ,
    ENTITY_SHORTNAME  ,
    ENTITY_SUBTYPE  ,
    ENTITY_COMPDELAY  ,
    day
    from stage.cms_entity_by_day;
    


    设置动态partitions为100000,该语句插入几年按天的partitions超过1000个,成功运行语句。

  • 相关阅读:
    [转载]解析用户生命周期价值:LTV
    [整理]VS2013常用插件
    Extjs4 tabpanel 中表单定宽居中
    idea创建ssm项目
    shiro框架的学习
    mysql
    springboot
    idea配置
    Spring MVC
    关于web乱码问题
  • 原文地址:https://www.cnblogs.com/lcchuguo/p/5274469.html
Copyright © 2011-2022 走看看