zoukankan      html  css  js  c++  java
  • Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode

    动态分区数太大的问题:[Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.


    hive> insert into table sogouq_test partition(query_time) select user_id,query_word,query_order,click_order,url,query_time from sogouq_test_tmp;
    Total MapReduce jobs = 3
    Launching Job 1 out of 3
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_1409113942738_0026, Tracking URL = http://centos1:8088/proxy/application_1409113942738_0026/
    Kill Command = /home/hadoop-2.2/bin/hadoop job  -kill job_1409113942738_0026
    Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
    2014-08-27 03:55:16,868 Stage-1 map = 0%,  reduce = 0%
    [Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.
    Sample of 100 partitions created under hdfs://centos1:8020/hive/scratchdir/hive_2014-08-27_03-55-09_118_348369539322185503-1/_tmp.-ext-10002:
            .../query_time=20111230000005
            .../query_time=20111230000007
            .../query_time=20111230000008
            .../query_time=20111230000009
            .../query_time=20111230000010
            .../query_time=20111230000011


    查看最大分区数:

    hive> set hive.exec.max.dynamic.partitions.pernode;
    hive.exec.max.dynamic.partitions.pernode=100

    将该參数设置的大一点,问题即解决。

  • 相关阅读:
    POJ3352 Road Construction (双连通分量)
    强连通分量(模板)
    图基本概念
    Air Raid(最小路径覆盖)
    Machine Schedule(最小覆盖)
    hdoj 1564 Play a game
    nyoj 483 Nightmare【bfs+优先队列】
    hdoj 1083 Courses【匈牙利算法】
    hdoj 2036 改革春风吹满地
    nyoj 353 3D dungeon
  • 原文地址:https://www.cnblogs.com/mthoutai/p/7396911.html
Copyright © 2011-2022 走看看