zoukankan      html  css  js  c++  java
  • 【原创】大叔问题定位分享(33)oozie提交任务报错ArithmeticException: / by zero

    oozie提交workflow后执行task报错:

    2019-07-04 17:19:00,559 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM. 
    java.lang.ArithmeticException: / by zero
        at org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:981)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:873)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:252)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282)
        at java.lang.Thread.run(Thread.java:748)

    查看代码

    org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils

        public static int computeAvailableContainers(Resource available, Resource required, EnumSet<SchedulerResourceTypes> resourceTypes) {
            return resourceTypes.contains(SchedulerResourceTypes.CPU)?Math.min(available.getMemory() / required.getMemory(), available.getVirtualCores() / required.getVirtualCores()):available.getMemory() / required.getMemory();
        }

    应该是以下两者之一为0

    required.getMemory()
    required.getVirtualCores()

    注意之前的日志还有一行

    2019-07-04 17:18:58,557 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:0, vCores:1>

    问题在于required.getMemory()=0

    从job history server查看失败task对应的application的conf发现问题

                  <tr>
                    <td>
                      mapreduce.map.memory.mb
                    </td>
                    <td>
                      0
                    </td>
                    <td>
                      job.xml &#11013; programatically
                    </td>
                  </tr>

    查看oozie代码发现

    org.apache.oozie.action.hadoop.JavaActionExecutor

        private static void injectLauncherProperties(Configuration srcConf, Configuration launcherConf) {
            for (Map.Entry<String, String> entry : srcConf) {
                if (entry.getKey().startsWith("oozie.launcher.")) {
                    String name = entry.getKey().substring("oozie.launcher.".length());
                    String value = entry.getValue();
                    // setting original KEY
                    launcherConf.set(entry.getKey(), value);
                    // setting un-prefixed key (to allow Hadoop job config
                    // for the launcher job
                    launcherConf.set(name, value);
                }
            }
        }

    所有的hadoop配置,都需要增加 oozie.launcher. 前缀(大量的组件都是这么搞得),即需要传递参数

    oozie.launcher.mapreduce.map.memory.mb=1024

    问题解决

  • 相关阅读:
    flume 使用遇到问题及解决
    定时任务 Linux cron job 初步使用
    java instrumentation &JVMTI
    Java远程执行Shell命令
    No input clusters found in output/ZS_TEST_OUTPUT3404268420/clusters-0/part-randomSeed. Check your -c argument.
    asp.net core 中读取post 方式来的内容
    C#程序 权限不够的解决方案
    wamp下安装https 实现 ssl 协议,主要是编写小程序通讯
    如何让thinkpad X1C 用U盘 安装上专业版win10
    php 5.4 5.5 如何连接 ms sqlserver
  • 原文地址:https://www.cnblogs.com/barneywill/p/11140996.html
Copyright © 2011-2022 走看看