zoukankan      html  css  js  c++  java
  • spark

    Error

    ERROR TaskSetManager: Total size of serialized results of 8113 tasks (1131.0 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
    ERROR TaskSetManager: Total size of serialized results of 8114 tasks (1131.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
    ERROR TaskSetManager: Total size of serialized results of 8115 tasks (1131.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
    ERROR TaskSetManager: Total size of serialized results of 8116 tasks (1131.3 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
    

    Cause

    • caused by actions like RDD’s collect() that send big chunk of data to the driver(不一定是因为RDD的问题哦~)

    Solution

    • set by SparkConf: conf.set("spark.driver.maxResultSize", "3g")
    • set by spark-defaults.confspark.driver.maxResultSize 3g
    • set when calling spark-submit--conf spark.driver.maxResultSize=3g
  • 相关阅读:
    E. You Are Given Some Strings...
    神奇函数
    AC自动机再加强版
    AC自动机2
    AC自动机
    three arrays
    permutation 2
    string matching
    permutation 1
    equation
  • 原文地址:https://www.cnblogs.com/suanec/p/6169339.html
Copyright © 2011-2022 走看看