zoukankan      html  css  js  c++  java
  • spark-submit command-line with --files

    spark提交任务

    bin/spark-submit --name  Test --class com.test.batch.modeltrainer.ModelTrainerMain  
      --master local --files /tmp/myobject.ser --verbose  /opt/test/lib/spark-test.jar 

    程序引用

    val serFile = SparkFiles.get("myobject.ser")

    错误提示

    Exception: 
    Exception in thread "main" java.lang.NullPointerException 
      at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37) 
      at org.apache.spark.SparkFiles$.get(SparkFiles.scala:31) 
      at com.test.batch.modeltrainer.ModelTrainerMain$.main(ModelTrainerMain.scala:37) 
      at com.test.batch.modeltrainer.ModelTrainerMain.main(ModelTrainerMain.scala) 
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
      at java.lang.reflect.Method.invoke(Method.java:606) 
      at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) 
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) 
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

    解决:

    SparkEnv is an internal class that is only meant to be used within Spark. Outside of Spark, it will be null because there are no executors or driver to start an environment for. Similarly, SparkFiles is meant to be used internally (though it's privacy settings should be modified to reflect that).

    只能在spark内去引用,在executors或driver去引用,在算子内

    sc.parallelize(1 to 100).map { i => SparkFiles.get("my.file") }.collect()

  • 相关阅读:
    图象处理算法(一)
    使用自定义类实现工程多语言
    SQL的数据类型
    SQL LEFT JOIN
    TADOCommand
    如何获得活动的数据
    TDateTime
    类定义(一)
    mxOutlookBar组件安装和使用
    TADOQuery
  • 原文地址:https://www.cnblogs.com/xiaoma0529/p/7068457.html
Copyright © 2011-2022 走看看