zoukankan      html  css  js  c++  java
  • hive mapjoin优化

    默认为10MB,如果大于该值不会执行mapjoin,hive语句中直接设置的mapjoin也不再起作用。
    参考hive wiki把hive.auto.convert.join.noconditionaltask.size

    修改大一些就ok。
    官方解释为:
    hive.auto.convert.join.noconditionaltask
    * Default Value: true
    * Added In: 0.11.0 with HIVE-3784 (default changed to true with HIVE-4146)

    Whether Hive enables the optimization about converting common join into mapjoin based on the input file size. If this parameter is on, and the sum of size for n-1 of the tables/partitions for an n-way join is smaller than the size specified by hive.auto.convert.join.noconditionaltask.size, the join is directly converted to a mapjoin (there is no conditional task).
    hive.auto.convert.join.noconditionaltask.size
    * Default Value: 10000000
    * Added In: 0.11.0 with HIVE-3784

    If hive.auto.convert.join.noconditionaltask is off, this parameter does not take effect. However, if it is on, and the sum of size for n-1 of the tables/partitions for an n-way join is smaller than this size, the join is directly converted to a mapjoin (there is no conditional task). The default is 10MB.

    https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

  • 相关阅读:
    运算符重载
    责任链模式
    MFC一些常见面试问题
    浅拷贝&深拷贝
    下雨的效果
    本地时间使用与倒计时
    钟表效果
    一种水纹波浪效果
    一个相册效果
    在Flash中管理鼠标右键
  • 原文地址:https://www.cnblogs.com/jacksu-tencent/p/4044285.html
Copyright © 2011-2022 走看看