zoukankan      html  css  js  c++  java
  • [java][spark][spark streamming]java.util.concurrent.TimeoutException: Futures timed out

    spark streamming 程序提交到yarn 上运行

    报错

    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/mnt/disk3/hadoop/yarn/local/filecache/491/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/hdp/3.0.0.0-1634/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    19/10/24 12:02:09 ERROR ApplicationMaster: Uncaught exception: 
    java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
        at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
    View Code

    ==========

    我发现我的问题出在, 有一个特别大数据结构加载上.

    有一个30w 数据的map加载, 耗时超过了100s.

    - 优化这个加载

    - 把时间改长

  • 相关阅读:
    spring框架学习(六)AOP
    spring框架学习(五)注解
    spring框架学习(三)
    spring框架学习(四)自动装配
    spring框架学习(二)依赖注入
    解决Photoshop不支持webp格式图片问题
    计算机导论-5、6
    计算机导论-3、4
    计算机导论-1、2
    Android SDK Manager国内下载缓慢的问题
  • 原文地址:https://www.cnblogs.com/fadedlemon/p/11732084.html
Copyright © 2011-2022 走看看