spark on yarn

说明

这篇文章记录下 spark提交左右在yarn上运行

hadoop配置

主要配置yarn-site.xml文件,我们目前使用mapreduce_shuffle,而有些公司也增加了spark_shuffle

• 只使用mapreduce_shuffle

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  
  <property>
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
    <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  </property>
  

• 使用mapreduce_shuffle & spark_shuffle

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,spark_shuffle</value>
  </property>
  
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  
  <property>
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
    <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  </property>
  

当提交hadoop MR 就启用,mapreduce_shuffle,当提交spark作业 就使用spark_shuffle,但个人感觉spark_shuffle 效率一般,shuffle是很大瓶颈,还有 如果你使用spark_shuffle 你需要把spark-yarn_2.10-1.4.1.jar 这个jar copy 到HADOOP_HOME/share/hadoop/lib下 ,否则 hadoop 运行报错 class not find exeception

spark配置

$SPARK_HOME/conf/spark-env.sh

export YARN_CONF_DIR=/home/cluster/apps/hadoop/etc/hadoop

export JAVA_HOME=/home/cluster/share/java1.7
export SCALA_HOME=/home/cluster/share/scala-2.10.5
export HADOOP_HOME=/home/cluster/apps/hadoop
export HADOOP_CONF_DIR=/home/cluster/apps/hadoop/etc/hadoop
export SPARK_MASTER_IP=master

export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/home/cluster/apps/hadoop/lib/native
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/home/cluster/apps/hadoop/share/hadoop/yarn/*:/home/cluster/apps/hadoop/share/hadoop/yarn/lib/*:/home/cluster/apps/hadoop/share/hadoop/common/*:/home/cluster/apps/hadoop/share/hadoop/common/lib/*:/home/cluster/apps/hadoop/share/hadoop/hdfs/*:/home/cluster/apps/hadoop/share/hadoop/hdfs/lib/*:/home/cluster/apps/hadoop/share/hadoop/mapreduce/*:/home/cluster/apps/hadoop/share/hadoop/mapreduce/lib/*:/home/cluster/apps/hadoop/share/hadoop/tools/lib/*:/home/cluster/apps/spark/spark-1.4.1/lib/*

SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://master:8020/var/log/spark"

参数解释:
YARN_CONF_DIR:指定yarn配置所在路径,如果不增加这行,在提交作业时候增加如下代码:

export YARN_CONF_DIR=/home/cluster/apps/hadoop/etc/hadoop

HADOOP_HOME:指定hadoop 根目录
HADOOP_CONF_DIR:hadoop配置文件,这个是在spark,如操作hdfs时候读取hadoop配置文件
SPARK_LIBRARY_PATH:告诉spark读取本地的.so文件
SPARK_CLASSPATH:spark加载各种需要的jar包
SPARK_HISTORY_OPTS:配置启动spark history 服务

前置条件

如果操作hdfs,需要启动namenode&datanode
还有yarn服务器,resourcemanger&nodemanager

 /home/cluster/apps$ jps
29368 MainGenericRunner
29510 Jps
22885 Main
29210 NodeManager
28952 NameNode
29158 ResourceManager
29023 DataNode

提交作业

1. PI:

• yarn-cluster模式:

  /home/cluster/apps/spark/spark-1.4.1/bin/spark-submit --master yarn-cluster --executor-memory 3g   --driver-memory 1g  --class org.apache.spark.examples.SparkPi /home/cluster/apps/spark/spark-1.4.1/examples/target/scala-2.10/spark-examples-1.4.1-hadoop2.3.0-cdh5.1.0.jar  10
  

• yarn-client模式:

  /home/cluster/apps/spark/spark-1.4.1/bin/spark-submit --master yarn-client --executor-memory 3g   --driver-memory 1g  --class org.apache.spark.examples.SparkPi /home/cluster/apps/spark/spark-1.4.1/examples/target/scala-2.10/spark-examples-1.4.1-hadoop2.3.0-cdh5.1.0.jar  10
  

2. wordcount:

• yarn-cluster模式:

  /home/cluster/apps/spark/spark-1.4.1/bin/spark-submit --master yarn-cluster --executor-memory 3g   --driver-memory 1g  --class org.apache.spark.examples.JavaWordCount /home/cluster/apps/spark/spark-1.4.1/examples/target/scala-2.10/spark-examples-1.4.1-hadoop2.3.0-cdh5.1.0.jar /data/hadoop/wordcount/
  

• yarn-client模式:

  /home/cluster/apps/spark/spark-1.4.1/bin/spark-submit --master yarn-client --executor-memory 3g   --driver-memory 1g  --class org.apache.spark.examples.JavaWordCount /home/cluster/apps/spark/spark-1.4.1/examples/target/scala-2.10/spark-examples-1.4.1-hadoop2.3.0-cdh5.1.0.jar /data/hadoop/wordcount/
  

结果截图

这四条记录从下往上看,分别是PI:yarn-cluster模式,PI:yarn-client模式,wordcount:yarn-cluster模式,wordcount:yarn-client模式

尊重原创,拒绝转载
http://blog.csdn.net/stark_summer/article/details/48661317