zoukankan      html  css  js  c++  java
  • executor.Executor: Managed memory leak detected; size = 37247642 bytes, TID = 5

    https://stackoverflow.com/questions/34359211/debugging-managed-memory-leak-detected-in-spark-1-6-0

    https://stackoverflow.com/questions/33518992/spark-executor-managed-memory-leak-detected

     
    I saw this exception while running spark streaming on EMR 16/04/14 13:49:10 WARN memory.TaskMemoryManager: leak 32.3 MB memory from org.apache.spark.unsafe.map.BytesToBytesMap@34158d5f 16/04/14 13:49:10 ERROR executor.Executor: Managed memory leak detected; size = 33816576 bytes, TID = 2942915 16/04/14 13:49:10 ERROR executor.Executor: Exception in task 22.0 in stage 35684.0 (TID 2942915) java.lang.OutOfMemoryError: Unable to acquire 262144 bytes of memory, got 220032 – Nipun Apr 20 '16 at 9:27 
        
    Were you able to resolve it? I am facing a similar issue of memory leak in spark 1.6.2 – Anchika Agarwal Mar 24 at 18:43
        
    I think in my case it was SPARK-14560. It is fixed in Spark 2.0.0 and we are using 2.1.0 now, so all is good. – Daniel Darabos Mar 25 at 20:10
        
    does all these comment mean, if I can not upgrade to the 'good' version yet due to company IT readiness, I can safely ignore the error? I am currently using spark 1.6.0 and having errors like this when some dataframe does 'distinct' and then 'head'. – Minnie Shi Aug 8 at 9:55 
        
    Well, if ignoring it is your only option, I suggest you ignore it :). It may lead to out-of-memory issues, being a warning about memory mis-accounting. – Daniel Darabos Aug 8 at 10:31
  • 相关阅读:
    Python
    按行读取文本,去重行
    Python字符串---固定长度分割字符串
    python终端打印带颜色的print
    Django安装与删除
    Python拼接字符串的7种方法
    CSS指定标签样式style选择器
    python 使用函数名的字符串调用函数(4种方法)
    test
    字符串常用操作
  • 原文地址:https://www.cnblogs.com/rocky-AGE-24/p/7512071.html
Copyright © 2011-2022 走看看