zoukankan      html  css  js  c++  java
  • SPARK-18560

    ##### Receiver data can not be dataSerialized properly.
    
    ```
    16/11/22 19:20:15 ERROR executor.Executor: Exception in task 4.3 in stage 6.0 (TID 87)
    com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
        at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:243)
        at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:169)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1760)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:108)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    ```
    This problem stems from SPARK-13990, a patch to have Spark automatically pick the "best" serializer when caching RDDs.
    
    ```
    Affects Version/s:
    2.0.2
    Fix Version/s:
    2.0.3, 2.1.0
    ```

    SPARK-18560

  • 相关阅读:
    详解EBS接口开发之采购申请导入
    EBS HRMS数据表
    会计期间
    帐套和会计科目的理解
    oracle中动态SQL详解
    不同币种汇率转换
    API创建/更新员工联系电话
    API创建/更新员工薪水
    Android 圆形、圆角图片ImageView
    Knowledge Generation Model for Visual Analytics
  • 原文地址:https://www.cnblogs.com/suanec/p/9881954.html
Copyright © 2011-2022 走看看