zoukankan      html  css  js  c++  java
  • hive:默认允许动态分区个数为100,超出抛出异常:

    在创建好一个分区表后,执行动态分区插入数据,抛出了错误:

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by
    hive.exec.max.dynamic.partitions and 
    hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
     2017-11-06 14:54:31,381 FATAL [IPC Server handler 9 on 27102] org.apache.hadoop.mapred.TaskAttemptListenerImpl: 
    Task: attempt_1500969698103_8078701_m_000278_0 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
    Hive Runtime Error while processing row {"gridid":"148420952","longitude":"120.315241543934","latitude":"30.191598257425056",
    "objectid":"888665533","cellname":"7777建设_1","scrsrp":"-91.67163276672363","gridx":"12001","gridy":"7825","doortype":null,"biuldingid":null,"calibrategridid":"309","nobjectid":"193619203","ncrsrp":"-122.38408386707306","height":"0","p_group":0} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:180) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:174) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
    {"gridid":"148420952","longitude":"120.315241543934","latitude":"30.191598257425056","objectid":"998867665","cellname":"777建设_1","scrsrp":"-91.67163276672363","gridx":"12001","gridy":"7825","doortype":null,"biuldingid":null,"calibrategridid":"309","nobjectid":"193619203","ncrsrp":"-122.38408386707306","height":"0","p_group":0} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]:
    Fatal error occurred when node tried to create too many dynamic partitions.
    The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode.
    Maximum was set to: 100 at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:936) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:713) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:111) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552) ... 9 more

    解决方案:

    在执行插入数据到分区时,添加参数设置:

    set hive.exec.dynamic.partition=true;
    set hive.exec.dynamic.partition.mode=nonstrict;
    set hive.exec.max.dynamic.partitions.pernode=10000;
    set hive.exec.max.dynamic.partitions=10000;
    set hive.exec.max.created.files=10000;
  • 相关阅读:
    ASPxGridView之ASPxGridViewExporter
    Asp.net中,从弹出窗体取选择值
    ASPxGridView中常用操作
    asp.net Webconfig
    白皮书 CPU卡基本知识
    Linux 中的计时 转自IBM china
    网络无缝融合技术(转)
    UMA相关的网站
    几本不错的书
    手机基带芯片供应商严阵以待,备战3G市场
  • 原文地址:https://www.cnblogs.com/yy3b2007com/p/7794564.html
Copyright © 2011-2022 走看看