zoukankan      html  css  js  c++  java
  • [杂谈]Executor-1

    为了让Task对象能够重用,在Executor中,每一个分区数据都会有一个Task去进行计算,计算完以后,就要释放taslk的内存,包括堆内和堆外内存,并且,要把Thread-Local的内存也释放掉(TLA)。

    org.apache.spark.scheduler.Task
    run() {
    try {
    Utils.tryLogNonFatalError {
    // Release memory used by this thread for unrolling blocks
    SparkEnv.get.blockManager.memoryStore.releaseUnrollMemoryForThisTask(MemoryMode.ON_HEAP)
    SparkEnv.get.blockManager.memoryStore.releaseUnrollMemoryForThisTask(MemoryMode.OFF_HEAP)
    // Notify any tasks waiting for execution memory to be freed to wake up and try to
    // acquire memory again. This makes impossible the scenario where a task sleeps forever
    // because there are no other tasks left to notify it. Since this is safe to do but may
    // not be strictly necessary, we should revisit whether we can remove this in the future.
    val memoryManager = SparkEnv.get.memoryManager
    memoryManager.synchronized { memoryManager.notifyAll() }
    }
    } finally {
    TaskContext.unset()
    }
    }

  • 相关阅读:
    bzoj1221
    hdu3377
    bzoj3930
    bzoj3976
    bzoj4237
    fzu1977
    hdu1693
    ural1519
    bzoj1264
    回答自己的提问
  • 原文地址:https://www.cnblogs.com/ivanny/p/spark_executor_task_1.html
Copyright © 2011-2022 走看看