zoukankan      html  css  js  c++  java
  • Spark-Submit提交作业过程

    1. spark-submit脚本

    exec $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"

    2. SparkSubmit中的main函数

    def main(args: Array[String]): Unit = {
        val appArgs = new SparkSubmitArguments(args)
        if (appArgs.verbose) {
          printStream.println(appArgs)
        }
        appArgs.action match {
          case SparkSubmitAction.SUBMIT => submit(appArgs)
          case SparkSubmitAction.KILL => kill(appArgs)
          case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
        }
      }

    Internally, each RDD is characterized by five main properties:
     - A list of partitions

      //Implemented by subclasses to return the set of partitions in this RDD. This method will only be called once, so it is safe to implement a time-consuming computation in it.
      protected def getPartitions: Array[Partition]

    - A function for computing each split

    //Implemented by subclasses to compute a given partition.
    def compute(split: Partition, context: TaskContext): Iterator[T]

     - A list of dependencies on other RDDs

    //Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only be called once, so it is safe to implement a time-consuming computation in it.
    protected def getDependencies: Seq[Dependency[_]] = deps

     - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)

    //Optionally overridden by subclasses to specify how they are partitioned. 
    @transient val partitioner: Option[Partitioner] = None

     - Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

    //Optionally overridden by subclasses to specify placement preferences.
    protected def getPreferredLocations(split: Partition): Seq[String] = Nil
  • 相关阅读:
    在未排序的数组中找到第 k 个最大的元素
    区域和检索
    控制台画图程序(可更换笔刷版本)
    循环中的scanf处理了换行符怎么破
    strlen获取字符数组为什么是255
    宽字符输出中文,Devc++解决方法
    区间取最小值最大值-位值和
    模拟鼠标键盘-封装函数
    scanf("%d",a[i]+j)为什么不加取地址符号
    scanf需要多输入一行是什么问题
  • 原文地址:https://www.cnblogs.com/java-cjt/p/4593307.html
Copyright © 2011-2022 走看看