zoukankan      html  css  js  c++  java
  • RHadoop: REDUCE capability required is more than the supported max container capability in the cluster

    I have not used RHadoop. However I've had a very similar problem on my cluster, and this problem seems to be linked only to MapReduce.

    The maxContainerCapability in this log refers to the yarn.scheduler.maximum-allocation-mbproperty of your yarn-site.xml configuration. It is the maximum amount of memory that can be used in any container.

    The mapResourceReqt and reduceResourceReqt in your log refer to the mapreduce.map.memory.mb and mapreduce.reduce.memory.mb properties of your mapred-site.xml configuration. It is the memory size of the containers that will be created for a Mapper or a Reducer in mapreduce.

    If the size of your Reducer's container is set to be greater than yarn.scheduler.maximum-allocation-mb, which seems to be the case here, your job will be killed because it is not allowed to allocate so much memory to a container.

    Check your configuration at http://[your-resource-manager]:8088/conf and you should normally find these values and see that this is the case.

    Maybe your new environment has these values set to 4096 Mb (which is quite big, the default in Hadoop 2.7.1 being 1024).

    Solution

    You should either lower the mapreduce.[map|reduce].memory.mb values down to 1024, or if you have lots of memory and want huge containers, raise the yarn.scheduler.maximum-allocation-mb value to 4096. Only then MapReduce be able to create containers.

    I hope this helps.

    简而言之:yarn-site.xml 中的yarn.scheduler.maximum-allocation-mb配置的值 >= mapred-site.xml 中mapreduce.map.memory.mb、mapreduce.reduce.memory.mb 的值

  • 相关阅读:
    c++之模板
    javascript 学习 四 函数参数
    javascript 学习三 语句
    jsvascript 学习 二 操作符
    javascript 学习一
    Highcharts areaspline 图表 动态更新数据
    js 数据类型的初步理解
    angular ng-class 三种方法
    angular 与 highcharts 结合使用
    angular 单选按钮之间的默认选中问题
  • 原文地址:https://www.cnblogs.com/zhouj-happy/p/6710295.html
Copyright © 2011-2022 走看看