zoukankan      html  css  js  c++  java
  • Spark优化之二:集群上运行jar程序,状态一直Accepted且不停止不报错

    如果运行Spark集群时状态一直为Accepted且不停止不报错,比如像下面这样的情况:

    15/06/14 11:33:33 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:34 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:35 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:36 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:37 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:38 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:39 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:40 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    15/06/14 11:33:41 INFO yarn.Client: Application report for application_1434263747091_0023 (state: ACCEPTED)
    

    一般是由于有多个用户同时向集群提交任务或一个用户向集群同时提交了多个任务导致Yarn资源的分配错误。解决这个问题,只需要更改Hadoop的配置文件:/etc/hadoop/conf/capacity-scheduler.xml,把选项:yarn.scheduler.capacity.maximum-am-resource-percent从0.1改成0.5。顾名思义,这个选项是增加Yarn可调度的资源量,当然也可以视具体情况增加更多。也可见,默认情况下,Yarn没有将很多资源分配给任务的能力。

  • 相关阅读:
    Spring 由哪些模块组成?
    spring 支持哪些 ORM 框架 ?
    @Autowired 注解有什么用?
    spring JDBC API 中存在哪些类?
    有哪些类型的通知(Advice)?
    @Component, @Controller, @Repository, @Service 有何区别?
    @Qualifier 注解有什么用?
    什么是 Aspect?
    什么是通知(Advice)?
    spring bean 容器的生命周期是什么样的?
  • 原文地址:https://www.cnblogs.com/shixiangwan/p/6195297.html
Copyright © 2011-2022 走看看