The required MAP capability is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: <memory:2048, vCores:2> maxContainerCapability:<memory:1024, vCores

yarn内存设置问题

hive查询时出现

Ended Job = job_1544003470555_0007 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

然后进行yarn测试

hadoop jar hadoop-mapreduce-examples-3.0.0-cdh6.0.0.jar pi 2 10

报错

18/12/11 17:58:56 INFO mapreduce.Job: Job job_1544003470555_0008 failed with state KILLED due to: The required MAP capability is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: <memory:2048, vCores:2> maxContainerCapability:<memory:1024, vCores:1>

解决方法

修改参数 调大虚拟内存，根据自己情况配置

mapreduce.map.memory.mb=2048
mapreduce.reduce.memory.mb=2048
yarn.nodemanager.vmem-pmem-ratio=3
参考yarn平台参数设置点击此处
这样map运行时的虚拟内存大小为 2048*3

类似这样的情况还有
设置Container的分配的内存大小，意味着ResourceManager只能分配给Container的内存

大于yarn.scheduler.minimum-allocation-mb=2G，
不能超过 yarn.scheduler.maximum-allocation-mb=8G  的值。

ResourceManager分配给container的CPU也要满足最小和最大值的条件，通过设置

yarn.scheduler.minimum-allocation-vcores=2
yarn.scheduler.maximum-allocation-vcores=8