zoukankan      html  css  js  c++  java
  • Spark Standalone spark-env.sh

    export JAVA_HOME=/app/jdk
    export SPARK_MASTER_PORT=7077
    export SPARK_MASTER_WEBUI_PORT=8080
    export SPARK_WORKER_CORES=50
    export SPARK_WORKER_OPTS="
    -Dspark.worker.cleanup.enabled=true
    -Dspark.worker.cleanup.interval=1800
    -Dspark.worker.cleanup.appDataTtl=604800"
    export SPARK_EXECUTOR_INSTANCES=1
    export SPARK_WORKER_MEMORY=100g
    export HADOOP_HOME=/app/hadoop
    export HADOOP_CONF_DIR=/app/hadoop/etc/hadoop
    export SPARK_DAEMON_MEMORY=10G
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=VECS01459:2181,VECS01458:2181,VECS01457:2181,VECS01456:2181,VECS01455:2181 -Dspark.d
    eploy.zookeeper.dir=/spark"
    export SPARK_DAEMON_JAVA_OPTS="(SPARK_DAEMON_JAVA_OPTS -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+G1SummarizeConcMark -XX:MaxGCPauseMillis=150 -XX:-ResizePLAB -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -XX:ParallelGCThreads=24 -XX:ConcGCThreads=16 -XX:G1HeapWastePercent=3 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1Mi xedGCLiveThresholdPercent=85 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy" export JAVA_LIBRARY_PATH=)JAVA_LIBRARY_PATH:(HADOOP_HOME/lib/native export LD_LIBRARY_PATH=)LD_LIBRARY_PATH:(HADOOP_HOME/lib/native export SPARK_LIBRARY_PATH=)SPARK_LIBRARY_PATH:(HADOOP_HOME/lib/native export SPARK_CLASSPATH=)SPARK_CLASSPATH:$HADOOP_HOME/lib/*

    Generic options for the daemons used in the standalone deploy mode

    export SPARK_CONF_DIR=({SPARK_HOME}/conf export SPARK_LOG_DIR=){SPARK_HOME}/logs
    export SPARK_PID_DIR=({SPARK_HOME}/tmp export SPARK_LOCAL_DIRS=/data1/data/spark/local export SPARK_IDENT_STRING=)USER

    Spark Standalone 集群调优后的环境变量。

  • 相关阅读:
    C#多线程下更新UI的几种方法
    .net WebApi使用swagger 美化接口文档
    C#内存管理
    LINQ 推迟查询的执行
    C#多播委托详解
    泛型委托
    使用 ref 和 out 传递数组注意事项
    何时使用委托而不使用接口
    委托中的协变和逆变
    细说SQL Server数据类型
  • 原文地址:https://www.cnblogs.com/hit-zb/p/10401719.html
Copyright © 2011-2022 走看看