zoukankan      html  css  js  c++  java
  • Hadoop本地编写的jar包放到集群执行时报错处理

    错误描述:

    020-03-24 22:45:23,204 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1585061045134_0001_01_000001 is : 1
    2020-03-24 22:45:23,205 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1585061045134_0001_01_000001 and exit code: 1
    ExitCodeException exitCode=1: 
    	at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
    	at org.apache.hadoop.util.Shell.run(Shell.java:901)
    	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
    	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:294)
    	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:491)
    	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
    	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)

    错误原因:查看日志发现有这样一条信息:conf.Configuration: resource-types.xml not found 还有这个: bad substitution 查了半天发现是提交任务出错了!
    解决办法:
    hadoop3的配置和hadoop2的配置基本相同,但是,在hadoop3中还需要在mapred-site.xml上新增一些东西:没有添加以下内容会找不到各种各样的类!添加之后问题解决。
    <property>
      <name>yarn.app.mapreduce.am.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.map.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.reduce.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
     <property>
            <name>mapreduce.application.classpath</name>
            <value>
                ${HADOOP_HOME}/etc/hadoop,
                ${HADOOP_HOME}/share/hadoop/common/*,
                ${HADOOP_HOME}/share/hadoop/common/lib/*,
                ${HADOOP_HOME}/share/hadoop/hdfs/*,
                ${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
                ${HADOOP_HOME}/share/hadoop/mapreduce/*,
                ${HADOOP_HOME}/share/hadoop/mapreduce/lib/*,
                ${HADOOP_HOME}/share/hadoop/yarn/*,
                ${HADOOP_HOME}/share/hadoop/yarn/lib/*
            </value>
    </property>


  • 相关阅读:
    day06.2-软链接与硬链接
    day06.1-磁盘管理
    day05.3-Linux进程管理
    day05.2-Vim编辑器
    day05.1-文件归档与压缩
    day04-Linux系统中用户控制及文件权限管理方法
    day03-Linux操作系统目录结构
    day02.2-常用Linux命令整理
    BST_traverse(中序遍历,前序遍历,后序遍历)
    JS_DOM_practice with Pokemon
  • 原文地址:https://www.cnblogs.com/syq816/p/12563191.html
Copyright © 2011-2022 走看看