zoukankan      html  css  js  c++  java
  • [杂谈]Executor-1

    为了让Task对象能够重用,在Executor中,每一个分区数据都会有一个Task去进行计算,计算完以后,就要释放taslk的内存,包括堆内和堆外内存,并且,要把Thread-Local的内存也释放掉(TLA)。

    org.apache.spark.scheduler.Task
    run() {
    try {
    Utils.tryLogNonFatalError {
    // Release memory used by this thread for unrolling blocks
    SparkEnv.get.blockManager.memoryStore.releaseUnrollMemoryForThisTask(MemoryMode.ON_HEAP)
    SparkEnv.get.blockManager.memoryStore.releaseUnrollMemoryForThisTask(MemoryMode.OFF_HEAP)
    // Notify any tasks waiting for execution memory to be freed to wake up and try to
    // acquire memory again. This makes impossible the scenario where a task sleeps forever
    // because there are no other tasks left to notify it. Since this is safe to do but may
    // not be strictly necessary, we should revisit whether we can remove this in the future.
    val memoryManager = SparkEnv.get.memoryManager
    memoryManager.synchronized { memoryManager.notifyAll() }
    }
    } finally {
    TaskContext.unset()
    }
    }

  • 相关阅读:
    杂谈
    P1441 砝码称重
    P3159 [CQOI2012]交换棋子
    P5200 [USACO19JAN]Sleepy Cow Sorting
    P5201 [USACO19JAN]Shortcut
    P5196 [USACO19JAN]Cow Poetry
    20190922UVA测试
    P4014 分配问题
    P4012 深海机器人问题
    P2050 [NOI2012]美食节
  • 原文地址:https://www.cnblogs.com/ivanny/p/spark_executor_task_1.html
Copyright © 2011-2022 走看看