zoukankan      html  css  js  c++  java
  • Hadoop本地编写的jar包放到集群执行时报错处理

    错误描述:

    020-03-24 22:45:23,204 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1585061045134_0001_01_000001 is : 1
    2020-03-24 22:45:23,205 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1585061045134_0001_01_000001 and exit code: 1
    ExitCodeException exitCode=1: 
    	at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
    	at org.apache.hadoop.util.Shell.run(Shell.java:901)
    	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
    	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:294)
    	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:491)
    	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
    	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)

    错误原因:查看日志发现有这样一条信息:conf.Configuration: resource-types.xml not found 还有这个: bad substitution 查了半天发现是提交任务出错了!
    解决办法:
    hadoop3的配置和hadoop2的配置基本相同,但是,在hadoop3中还需要在mapred-site.xml上新增一些东西:没有添加以下内容会找不到各种各样的类!添加之后问题解决。
    <property>
      <name>yarn.app.mapreduce.am.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.map.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.reduce.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
     <property>
            <name>mapreduce.application.classpath</name>
            <value>
                ${HADOOP_HOME}/etc/hadoop,
                ${HADOOP_HOME}/share/hadoop/common/*,
                ${HADOOP_HOME}/share/hadoop/common/lib/*,
                ${HADOOP_HOME}/share/hadoop/hdfs/*,
                ${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
                ${HADOOP_HOME}/share/hadoop/mapreduce/*,
                ${HADOOP_HOME}/share/hadoop/mapreduce/lib/*,
                ${HADOOP_HOME}/share/hadoop/yarn/*,
                ${HADOOP_HOME}/share/hadoop/yarn/lib/*
            </value>
    </property>


  • 相关阅读:
    背景(北极狐)
    【面试】java基础
    C#分享海报生成
    jetson nano 4gb记录
    jetson nano 2gb相关问题
    分布式系列-分布式ID
    Mybatis-Plus 多租户模式忽略某个方法
    IDEA 集成 Docker 插件实现一键远程部署 SpringBoot 应用,无需三方依赖,开源微服务全栈有来商城线上部署方式
    《将博客搬至CSDN》
    elk收集docker容器的json格式日志
  • 原文地址:https://www.cnblogs.com/syq816/p/12563191.html
Copyright © 2011-2022 走看看