zoukankan      html  css  js  c++  java
  • Spark-Submit提交作业过程

    1. spark-submit脚本

    exec $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"

    2. SparkSubmit中的main函数

    def main(args: Array[String]): Unit = {
        val appArgs = new SparkSubmitArguments(args)
        if (appArgs.verbose) {
          printStream.println(appArgs)
        }
        appArgs.action match {
          case SparkSubmitAction.SUBMIT => submit(appArgs)
          case SparkSubmitAction.KILL => kill(appArgs)
          case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
        }
      }

    Internally, each RDD is characterized by five main properties:
     - A list of partitions

      //Implemented by subclasses to return the set of partitions in this RDD. This method will only be called once, so it is safe to implement a time-consuming computation in it.
      protected def getPartitions: Array[Partition]

    - A function for computing each split

    //Implemented by subclasses to compute a given partition.
    def compute(split: Partition, context: TaskContext): Iterator[T]

     - A list of dependencies on other RDDs

    //Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only be called once, so it is safe to implement a time-consuming computation in it.
    protected def getDependencies: Seq[Dependency[_]] = deps

     - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)

    //Optionally overridden by subclasses to specify how they are partitioned. 
    @transient val partitioner: Option[Partitioner] = None

     - Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

    //Optionally overridden by subclasses to specify placement preferences.
    protected def getPreferredLocations(split: Partition): Seq[String] = Nil
  • 相关阅读:
    etc/ld.so.conf的使用说明
    COM编程入门转发
    OpenCV 中的图像结构IPLImage 和 Qt中QImage之间的转换源代码
    转载Linux启动过程详解2
    Linux添加环境变量与GCC编译器添加INCLUDE与LIB环境变量
    Linux下Opencv+Qt4
    OPENCV 中的图像旋转与缩放
    怎么用qt4绘制橡皮筋直线
    QT4+KDEVELOP+OPENCV配置
    Linux操作系统的头文件和库文件搜索路径
  • 原文地址:https://www.cnblogs.com/java-cjt/p/4593307.html
Copyright © 2011-2022 走看看