zoukankan      html  css  js  c++  java
  • Spark问题记录

    Spark 多线程时的序列化问题  临时记录

     Exception in thread "Thread-28" org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
        at org.apache.spark.rdd.RDD.flatMap(RDD.scala:277)
        at org.apache.spark.api.java.JavaRDDLike$class.flatMap(JavaRDDLike.scala:109)
        at org.apache.spark.api.java.JavaRDD.flatMap(JavaRDD.scala:32)
        at com.main.java.MyThread.run(MyThread.java:30)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.NotSerializableException: org.apache.spark.api.java.JavaSparkContext
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164)
        ... 7 more
    hadoop@Node4:/usr/local/myjar$

    解决:发现是node3的/etc/hosts 多了一行  127.0.1.1 Node3

    去掉就解决了。

  • 相关阅读:
    POJ C程序设计进阶 编程题#4:Tomorrow never knows?
    POJ C程序设计进阶 编程题#3: 发票统计
    深度学习笔记(10)- 控制中心之电源管理、鼠标和触控板、键盘和语言
    深度学习笔记(09)- 控制中心之时间设置
    深度学习笔记(08)- 控制中心之网络设置
    深度学习笔记(07)- 控制中心之声音设置
    深度学习笔记(06)- 控制中心之个性化设置
    深度学习笔记(05)- 控制中心之显示与默认程序
    深度学习笔记(04)- 控制中心之首页与用户管理
    深度学习笔记(03)- 启动器
  • 原文地址:https://www.cnblogs.com/gnivor/p/4404499.html
Copyright © 2011-2022 走看看