zoukankan      html  css  js  c++  java
  • spark-submit command-line with --files

    spark提交任务

    bin/spark-submit --name  Test --class com.test.batch.modeltrainer.ModelTrainerMain  
      --master local --files /tmp/myobject.ser --verbose  /opt/test/lib/spark-test.jar 

    程序引用

    val serFile = SparkFiles.get("myobject.ser")

    错误提示

    Exception: 
    Exception in thread "main" java.lang.NullPointerException 
      at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37) 
      at org.apache.spark.SparkFiles$.get(SparkFiles.scala:31) 
      at com.test.batch.modeltrainer.ModelTrainerMain$.main(ModelTrainerMain.scala:37) 
      at com.test.batch.modeltrainer.ModelTrainerMain.main(ModelTrainerMain.scala) 
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
      at java.lang.reflect.Method.invoke(Method.java:606) 
      at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) 
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) 
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

    解决:

    SparkEnv is an internal class that is only meant to be used within Spark. Outside of Spark, it will be null because there are no executors or driver to start an environment for. Similarly, SparkFiles is meant to be used internally (though it's privacy settings should be modified to reflect that).

    只能在spark内去引用,在executors或driver去引用,在算子内

    sc.parallelize(1 to 100).map { i => SparkFiles.get("my.file") }.collect()

  • 相关阅读:
    java实现模拟登陆
    springboot整合kafka(转)
    springboot使用缓存(三)
    springboot使用缓存(二)
    springboot使用缓存(一)
    springboot日期入参出参格式化注解:@DateTimeFormat 和 @JsonFormat (转)
    H5调用android相机拍照
    时间复杂度
    java guide面试突击版
    java面试手册-复制内容
  • 原文地址:https://www.cnblogs.com/xiaoma0529/p/7068457.html
Copyright © 2011-2022 走看看