zoukankan      html  css  js  c++  java
  • SPARK-18560

    ##### Receiver data can not be dataSerialized properly.
    
    ```
    16/11/22 19:20:15 ERROR executor.Executor: Exception in task 4.3 in stage 6.0 (TID 87)
    com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
        at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:243)
        at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:169)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1760)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    ```
    This problem stems from SPARK-13990, a patch to have Spark automatically pick the "best" serializer when caching RDDs.
    
    ```
    Affects Version/s:
    2.0.2
    Fix Version/s:
    2.0.3, 2.1.0
    ```

    SPARK-18560

  • 相关阅读:
    CF891E Lust
    Comet OJ 2019 夏季欢乐赛题解
    CF1098E Fedya the Potter
    CF1063F String Journey
    P4218 [CTSC2010]珠宝商
    AGC028 E
    51Nod 1584 加权约数和
    51Nod 1769 Clarke and math2
    Educational Codeforces Round 67
    斯特林数学习笔记
  • 原文地址:https://www.cnblogs.com/suanec/p/9881954.html
Copyright © 2011-2022 走看看