zoukankan      html  css  js  c++  java
  • sparkStreaming运行yarn,提交任务报错

    op.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    
    
    Container exited with a non-zero exit code 1
    
    19/04/29 19:06:35 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1556531951737_0004_01_000002 on host: df3. Exit status: 1. Diagnostics: Exception from container-launch.
    Container id: container_1556531951737_0004_01_000002
    Exit code: 1
    Stack trace: ExitCodeException exitCode=1: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
        at org.apache.hadoop.util.Shell.run(Shell.java:507)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    View Code

    解决:

    这种错误一般就是yarn会检查虚拟cpu和虚拟内存,如果没满足要求,那么就会杀掉任务

    因此在yarn-site.xml中配置:

    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
  • 相关阅读:
    java的访问控制修饰符
    js将一个数组分割成二维子数组
    java的数组
    Java 变量类型
    java
    java的 %和 ^
    Python第三方库 -> 由于是ipynb格式,所以没有对应的输出结果
    获取浏览器历史记录 browserhistory
    虚拟机 CentOS 6.8 安装
    解决 django.db.utils.OperationalError: (1051, "Unknown table 'mydorm.users_studen t'")
  • 原文地址:https://www.cnblogs.com/niutao/p/10792596.html
Copyright © 2011-2022 走看看