zoukankan      html  css  js  c++  java
  • Spark问题记录

    Spark 多线程时的序列化问题  临时记录

     Exception in thread "Thread-28" org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
        at org.apache.spark.rdd.RDD.flatMap(RDD.scala:277)
        at org.apache.spark.api.java.JavaRDDLike$class.flatMap(JavaRDDLike.scala:109)
        at org.apache.spark.api.java.JavaRDD.flatMap(JavaRDD.scala:32)
        at com.main.java.MyThread.run(MyThread.java:30)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.NotSerializableException: org.apache.spark.api.java.JavaSparkContext
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164)
        ... 7 more
    hadoop@Node4:/usr/local/myjar$

    解决:发现是node3的/etc/hosts 多了一行  127.0.1.1 Node3

    去掉就解决了。

  • 相关阅读:
    andorid(3) 使用sqllite进行数据持久化
    android(1)--hello world中的layout与 onCreate()
    android(2)--listView
    linux 常用指令
    关于虚拟内存、驻留内存与共享内存——virt res shr之间的关系
    base64 和 md5
    python阿里云短信服务
    python邮件发送
    算法
    python3 字典
  • 原文地址:https://www.cnblogs.com/gnivor/p/4404499.html
Copyright © 2011-2022 走看看