zoukankan      html  css  js  c++  java
  • Spark运行模式:cluster与client

     

    When run SparkSubmit --class [mainClass], SparkSubmit will call a childMainClass which is

    1. client mode, childMainClass = mainClass

    2. standalone cluster mde, childMainClass = org.apache.spark.deploy.Client

    3. yarn cluster mode, childMainClass = org.apache.spark.deploy.yarn.Client

    The childMainClass is a wrapper of mainClass. The childMainClass will be called in SparkSubmit, and if cluster mode, the childMainClass will talk to the the cluster and launch a process on one woker to run the mainClass.
     
    ps. use "spark-submit -v" to print debug infos.
     
    Yarn client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn JavaWordCount.jar
    childMainclass: org.apache.spark.examples.JavaWordCount
    Yarn cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master yarn-cluster JavaWordCount.jar
    childMainclass: org.apache.spark.deploy.yarn.Client
     
    Standalone client: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 JavaWordCount.jar
    childMainclass: org.apache.spark.examples.JavaWordCount
    Stanalone cluster: spark-submit -v --class "org.apache.spark.examples.JavaWordCount" --master spark://aa01:7077 --deploy-mode cluster JavaWordCount.jar
    childMainclass: org.apache.spark.deploy.rest.RestSubmissionClient (if rest, else org.apache.spark.deploy.Client)
     
    Taking standalone spark as example, here is the client mode workflow. The mainclass run in the driver application which could be reside out of the cluster.
  • 相关阅读:
    Oracle数据库面试题【转载】
    年龄计算周岁
    丈夫的权力与妻子的职业水平
    JDK 8 and JRE 8 Supported Locales
    一笔画 奇点 偶点
    流水行船问题
    PL/SQL LOOP SAMPLE
    OpenCV——识别各省份地图轮廓
    OpenCV——轮廓面积及长度计算
    树莓派3安装opencv2程序无法运行
  • 原文地址:https://www.cnblogs.com/c-abc/p/7457430.html
Copyright © 2011-2022 走看看