zoukankan      html  css  js  c++  java
  • 看看spark的源码

    1. IntelliJ IDEA安装plugins,加入scala插件和SBT插件

    2. 下载spark的源代码,下载地址 https://github.com/apache/spark,可以使用git下载或者下载zip包

    3. 导入项目,开启 IntelliJ 之后选择 Import Project,而后选择 Spark 源代码,并将其导入为 SBT 项目,在之后的过程中 SBT 过程将会自动进行,下载作为依赖的 jar 包。

    4. 源代码已经准备好了。开干吧

      private def schedule(): Unit = {
          if (state != RecoveryState.ALIVE) {
            return
          }
          // Drivers take strict precedence over executors
          val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state == WorkerState.ALIVE))
          val numWorkersAlive = shuffledAliveWorkers.size
          var curPos = 0
          for (driver <- waitingDrivers.toList) { // iterate over a copy of waitingDrivers
            // We assign workers to each waiting driver in a round-robin fashion. For each driver, we
            // start from the last worker that was assigned a driver, and continue onwards until we have
            // explored all alive workers.
            var launched = false
            var numWorkersVisited = 0
            while (numWorkersVisited < numWorkersAlive && !launched) {
              val worker = shuffledAliveWorkers(curPos)
              numWorkersVisited += 1
              if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
                launchDriver(worker, driver)
                waitingDrivers -= driver
                launched = true
              }
              curPos = (curPos + 1) % numWorkersAlive
            }
          }
          startExecutorsOnWorkers()
        }
  • 相关阅读:
    系统相关的信息模块: import sys
    Mysql 解决emoji表情处理问题
    Servlet 个人理解
    Java BufferImage 获取像素矩阵 或 数组
    SVN 出现 R6034 错误 Python导致异常
    Mycat 入门
    Mycat 入门
    Nginx 架构
    Nginx 架构 性能优化
    Nginx 架构 性能优化
  • 原文地址:https://www.cnblogs.com/yangjiming/p/9767022.html
Copyright © 2011-2022 走看看