zoukankan      html  css  js  c++  java
  • [SparkSQL] hive.exec.max.dynamic.partitions配置不起作用

    参考:https://issues.apache.org/jira/browse/SPARK-21574

    问题描述:

    动态分区插入的时候报错

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Number of dynamic partitions created is 1483, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1483.
        at org.apache.hadoop.hive.ql.metadata.Hive.getValidPartitionsInPath(Hive.java:1782)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1825)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.sql.hive.client.Shim_v2_1.loadDynamicPartitions(HiveShim.scala:1169)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(HiveClientImpl.scala:804)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply(HiveClientImpl.scala:802)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply(HiveClientImpl.scala:802)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:221)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:220)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:266)
        at org.apache.spark.sql.hive.client.HiveClientImpl.loadDynamicPartitions(HiveClientImpl.scala:802)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(HiveExternalCatalog.scala:946)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply(HiveExternalCatalog.scala:934)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply(HiveExternalCatalog.scala:934)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
        ... 33 more

    解决办法

    you should use `--conf` when you start your Spark apps/shell.

  • 相关阅读:
    菜鸟版JAVA设计模式—从买房子看代理模式
    NTP工作机制及时间同步的方法
    Java工厂模式
    圣魔大战3(Castle Fantisia)艾伦希亚战记完美攻略
    对javabean的内省操作
    插入排序(insertion sort)
    中英文对照 —— 十二星座
    中英文对照 —— 十二星座
    数学归纳法的相关证明
    数学归纳法的相关证明
  • 原文地址:https://www.cnblogs.com/144823836yj/p/14819473.html
Copyright © 2011-2022 走看看