zoukankan      html  css  js  c++  java
  • java执行spark查询hbase的jar包出现错误提示:ob aborted due to stage failure: Master removed our application: FAILED

    执行java调用scala 打包后的jar时候出现异常

      

    /14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
    15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
    15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
    15/04/14 23:57:39 INFO AppClient$ClientActor: Executor updated: app-20150414235011-0003/9 is now EXITED (Command exited with code 1)
    15/04/14 23:57:39 INFO SparkDeploySchedulerBackend: Executor app-20150414235011-0003/9 removed: Command exited with code 1
    15/04/14 23:57:39 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED
    15/04/14 23:57:39 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
    15/04/14 23:57:39 INFO TaskSchedulerImpl: Cancelling stage 0
    15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
    Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
        at scala.Option.foreach(Option.scala:236)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1234)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

    问题1:

    /14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
    15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
    15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memor
    分析:这个是内存不足?
    我spark-env.sh的配置文件信息如下
    export JAVA_HOME=/home/hadoop/jdk1.7.0_75
    export SCALA_HOME=/home/hadoop/scala-2.11.6
    export HADOOP_HOME=/home/hadoop/hadoop-2.3.0-cdh5.0.2
    export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.3.0-cdh5.0.2/etc/hadoop
    export SPARK_CLASSPATH=/home/hadoop/hbase-0.96.1.1-cdh5.0.2/lib/*
    export SPARK_MASTER_IP=master
    export SPARK_MASTER_PORT=17077
    export SPARK_MASTER_WEBUI_PORT=18080
                 
    export SPARK_WORKER_CORES=1
    export SPARK_WORKER_MEMORY=1g
    export SPARK_WORKER_WEBUI_PORT=18081
    export SPARK_WORKER_INSTANCES=1


    问题2:
    15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
    这句话的代码:
     val count = hbaseRDD.count()
        println("HBase RDD Count:" + count)
        hbaseRDD.cache()
    问题3:
    in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
    有遇到过类似的或者知道怎么解决的可以留言下

  • 相关阅读:
    Java中的引用
    JVM参数调优
    GCRoots
    JVM体系结构
    死锁编码及定位分析
    线程池(Java中有哪些方法获取多线程)
    Synchronized和Lock的区别
    阻塞队列BlockingQueue
    CountDownLatch/CyclicBarrier/Semaphore
    浅谈二分
  • 原文地址:https://www.cnblogs.com/zhanggl/p/4428602.html
Copyright © 2011-2022 走看看