zoukankan      html  css  js  c++  java
  • Spark Standalone spark-default.conf

    Example:

    spark.master spark://master:7077

    spark.eventLog.enabled true

    spark.eventLog.dir hdfs://namenode:8021/directory

    spark.serializer org.apache.spark.serializer.KryoSerializer

    spark.driver.memory 5g

    spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

    spark.eventLog.enabled true
    spark.eventLog.dir hdfs://flashHadoop/spark/log
    spark.eventLog.compress true

    spark.local.dir=/data1/data/spark/local

    spark.ui.enabled true
    spark.ui.killEnabled false
    spark.ui.port 18080

    spark.history.ui.port 18080
    spark.history.fs.cleaner.enabled true
    spark.history.fs.logDirectory hdfs://flashHadoop/spark/log
    spark.history.fs.cleaner.interval 5d
    spark.history.fs.cleaner.maxAge 8d

    2018.12.17 Turning for driver and exec memory

    spark.driver.supervise true
    spark.driver.cores 2
    spark.driver.memory 3g

    spark.executor.cores 2
    spark.executor.memory 3g

    spark.driver.extraJavaOption -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX
    :+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=16 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPer
    cent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy

    spark.executor.extraJavaOptions -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=100 -XX:-ResizePLAB -XX
    :+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=16 -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1MixedGCLiveThresholdPer
    cent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy

    Spark Job Submit → Read default conf,,,, 调优后。

  • 相关阅读:
    [luoguP2342] 叠积木(并查集)
    [luoguP2147] [SDOI2008]Cave 洞穴勘测(并查集 || lct)
    [POJ1611]The Suspects(并查集)
    [POJ2912]Rochambeau(并查集)
    网站图片增强JS插件2.0(兼容IE&FF)
    jQuery简单纯文字提示条
    复选框--全选/全不选/反选。简易版
    Magic Grid ComboBox JQuery 版
    SWFObject 的基本使用方法
    全局Timestamp管理器 检测js执行时间
  • 原文地址:https://www.cnblogs.com/hit-zb/p/10401737.html
Copyright © 2011-2022 走看看