zoukankan      html  css  js  c++  java
  • SPARK-18560

    ##### Receiver data can not be dataSerialized properly.
    
    ```
    16/11/22 19:20:15 ERROR executor.Executor: Exception in task 4.3 in stage 6.0 (TID 87)
    com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
        at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:243)
        at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:169)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1760)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    ```
    This problem stems from SPARK-13990, a patch to have Spark automatically pick the "best" serializer when caching RDDs.
    
    ```
    Affects Version/s:
    2.0.2
    Fix Version/s:
    2.0.3, 2.1.0
    ```

    SPARK-18560

  • 相关阅读:
    JavaScript知识回顾
    HTML和CSS相关知识回顾
    springmvc文件上传和下载
    jsp页面调试中的问题记录
    mybatis传参的几种方式
    ssm中调试遇到的坑
    idea新手日记
    Oracle安装
    mysql5安装
    Servlet 的原理----无脑笔记
  • 原文地址:https://www.cnblogs.com/suanec/p/9881954.html
Copyright © 2011-2022 走看看