zoukankan      html  css  js  c++  java
  • [java][spark][spark streamming]java.util.concurrent.TimeoutException: Futures timed out

    spark streamming 程序提交到yarn 上运行

    报错

    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/mnt/disk3/hadoop/yarn/local/filecache/491/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/hdp/3.0.0.0-1634/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    19/10/24 12:02:09 ERROR ApplicationMaster: Uncaught exception: 
    java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
        at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
    View Code

    ==========

    我发现我的问题出在, 有一个特别大数据结构加载上.

    有一个30w 数据的map加载, 耗时超过了100s.

    - 优化这个加载

    - 把时间改长

  • 相关阅读:
    php的序列化和反序列化有什么好处?
    解析PHP多种序列化与反序列化的方法
    PHP 数组和字符串互相转换实现方法
    PHP中foreach()用法汇总
    php实现伪静态的方法
    PHP中如何定义类及其成员属性与操作
    单例模式优缺点
    PHP 单例模式解析和实战
    主从复制之莫名少表
    诡异的 ERROR 1045 (28000): Access denied for user 错误
  • 原文地址:https://www.cnblogs.com/fadedlemon/p/11732084.html
Copyright © 2011-2022 走看看