zoukankan      html  css  js  c++  java
  • Spark-Submit提交作业过程

    1. spark-submit脚本

    exec $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"

    2. SparkSubmit中的main函数

    def main(args: Array[String]): Unit = {
        val appArgs = new SparkSubmitArguments(args)
        if (appArgs.verbose) {
          printStream.println(appArgs)
        }
        appArgs.action match {
          case SparkSubmitAction.SUBMIT => submit(appArgs)
          case SparkSubmitAction.KILL => kill(appArgs)
          case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
        }
      }

    Internally, each RDD is characterized by five main properties:
     - A list of partitions

      //Implemented by subclasses to return the set of partitions in this RDD. This method will only be called once, so it is safe to implement a time-consuming computation in it.
      protected def getPartitions: Array[Partition]

    - A function for computing each split

    //Implemented by subclasses to compute a given partition.
    def compute(split: Partition, context: TaskContext): Iterator[T]

     - A list of dependencies on other RDDs

    //Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only be called once, so it is safe to implement a time-consuming computation in it.
    protected def getDependencies: Seq[Dependency[_]] = deps

     - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)

    //Optionally overridden by subclasses to specify how they are partitioned. 
    @transient val partitioner: Option[Partitioner] = None

     - Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

    //Optionally overridden by subclasses to specify placement preferences.
    protected def getPreferredLocations(split: Partition): Seq[String] = Nil
  • 相关阅读:
    Linux查看系统版本信息和设置远程终端登录
    Linux环境下安装JDK
    Windows10修改Tomcat服务端口和一台机器部署多个Tomcat
    [转]C#反射-Assembly.Load、LoadFrom与LoadFile进阶
    【转】C# lock的使用
    一个简单的C++程序及说明
    插入排序
    堆排序
    选择排序
    快速排序
  • 原文地址:https://www.cnblogs.com/java-cjt/p/4593307.html
Copyright © 2011-2022 走看看