zoukankan      html  css  js  c++  java
  • SPARK-18560

    ##### Receiver data can not be dataSerialized properly.
    
    ```
    16/11/22 19:20:15 ERROR executor.Executor: Exception in task 4.3 in stage 6.0 (TID 87)
    com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
        at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:243)
        at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:169)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1760)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    ```
    This problem stems from SPARK-13990, a patch to have Spark automatically pick the "best" serializer when caching RDDs.
    
    ```
    Affects Version/s:
    2.0.2
    Fix Version/s:
    2.0.3, 2.1.0
    ```

    SPARK-18560

  • 相关阅读:
    poj2352树状数组解决偏序问题
    Python中的输入(input)和输出打印
    Python中的输入(input)和输出打印
    IP协议
    IP协议
    Python中的可迭代Iterable和迭代器Iterator
    Python中的可迭代Iterable和迭代器Iterator
    Python中的变量和数据类型
    Python中的变量和数据类型
    TCP的三次握手和四次挥手和UDP协议
  • 原文地址:https://www.cnblogs.com/suanec/p/9881954.html
Copyright © 2011-2022 走看看