zoukankan      html  css  js  c++  java
  • Spark中集群相关概念

    来源:http://spark.apache.org/docs/latest/cluster-overview.html

    TermMeaning
    Application

    User program built on Spark. Consists of a driver program and executors on the cluster.

    用户基于Spark构建的程序,由一个driver和集群中多个executor组成

    Application jar

    A jar containing the user's Spark application. In some cases users will want to create an "uber jar" containing their application along with its dependencies. The user's jar should never include Hadoop or Spark libraries, however, these will be added at runtime.

    包含用户Spark应用程序的jar文件。某些情况下用户会连同应用程序的依赖创建一个“超级jar”。这个jar文件不应该包含任何Hadoop或Spark库,因为它们会在运行时被加载

    Driver program

    The process running the main() function of the application and creating the SparkContext

    运行应用程序main()函数和创建SparkContext的进程

    Cluster manager

    An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)

    一个获取集群资源的外部服务,例如standalone,Mesos,YARN

    Deploy mode

    Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.

    指明driver进程的运行位置。在cluster模式中,由框架在集群中启动driver。在client模式中,由提交者在集群外启动driver

    Worker node

    Any node that can run application code in the cluster

    集群中任何可以运行应用程序的节点

    Executor

    A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.

    一个在worker 节点为application启动的进程,通过它运行tasks和将数据保存在内存或磁盘中。每一个application都有它自己的executors

    Task

    A unit of work that will be sent to one executor

    一个被发送到executor的工作单元

    Job

    A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. savecollect); you'll see this term used in the driver's logs.

    一个由Spark Action算子触发(例如save,collect)的多任务并行计算。可以在driver日志中看到这个词

    Stage

    Each job gets divided into smaller sets of tasks called stages that depend on each other (similar to the map and reduce stages in MapReduce); you'll see this term used in the driver's logs.

    每一个job根据tasks之间的依赖关系,划分为一组小的task,这组task就被称为stage可以在driver日志中看到这个词

    (渣翻-_-||)

  • 相关阅读:
    【摘录】使用实体框架、Dapper和Chain的仓储模式实现策略
    关于微软HttpClient使用,避免踩坑
    在网上摘录一段对于IOC的解析,比较直观,大家观摩观摩
    经历招聘中面试求职的点滴
    在面试中忽然发现DateTime的一些...
    GCC的内存边界对齐
    如何删除Weblogic域
    电蚊拍选购参考
    Localizing WPF with .resx files
    C#操作串口总结
  • 原文地址:https://www.cnblogs.com/144823836yj/p/10760841.html
Copyright © 2011-2022 走看看