一、Scala安装
1.https://www.scala-lang.org/download/2.11.12.html下载并复制到/home/jun下解压
[jun@master ~]$ cd scala-2.12.6/ [jun@master scala-2.12.6]$ ls -l total 0 drwxrwxr-x. 2 jun jun 162 Apr 27 16:49 bin drwxrwxr-x. 4 jun jun 86 Apr 27 16:49 doc drwxrwxr-x. 2 jun jun 244 Apr 27 16:49 lib drwxrwxr-x. 3 jun jun 18 Apr 27 16:49 man
2.启动Scala并使用Scala Shell
[jun@master scala-2.12.6]$ bin/scala Welcome to Scala 2.12.6 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_171). Type in expressions for evaluation. Or try :help. scala> println("hello,world") hello,world scala> 5*9 res1: Int = 45 scala> 2*res1 res2: Int = 90 scala> :help All commands can be abbreviated, e.g., :he instead of :help. :completions <string> output completions for the given string :edit <id>|<line> edit history :help [command] print this summary or command-specific help :history [num] show the history (optional num is commands to show) :h? <string> search the history :imports [name name ...] show import history, identifying sources of names :implicits [-v] show the implicits in scope :javap <path|class> disassemble a file or class name :line <id>|<line> place line(s) at the end of history :load <path> interpret lines in a file :paste [-raw] [path] enter paste mode or paste a file :power enable power user mode :quit exit the interpreter :replay [options] reset the repl and replay all previous commands :require <path> add a jar to the classpath :reset [options] reset the repl to its initial state, forgetting all session entries :save <path> save replayable session to a file :sh <command line> run a shell command (result is implicitly => List[String]) :settings <options> update compiler options, if possible; see reset :silent disable/enable automatic printing of results :type [-v] <expr> display the type of an expression without evaluating it :kind [-v] <type> display the kind of a type. see also :help kind :warnings show the suppressed warnings from the most recent line which had any scala> :quit
3.将Scala安装包复制到slave节点
二、Spark集群的安装与配置
采用Hadoop Yarn模式安装Spark
1.http://spark.apache.org/downloads.html下载spark-2.3.1-bin-hadoop2.7.tgz.gz并赋值到/home/jun下解压
[jun@master ~]$ cd spark-2.3.1-bin-hadoop2.7/ [jun@master spark-2.3.1-bin-hadoop2.7]$ ls -l total 84 drwxrwxr-x. 2 jun jun 4096 Jun 2 04:49 bin drwxrwxr-x. 2 jun jun 230 Jun 2 04:49 conf drwxrwxr-x. 5 jun jun 50 Jun 2 04:49 data drwxrwxr-x. 4 jun jun 29 Jun 2 04:49 examples drwxrwxr-x. 2 jun jun 12288 Jun 2 04:49 jars drwxrwxr-x. 3 jun jun 25 Jun 2 04:49 kubernetes -rw-rw-r--. 1 jun jun 18045 Jun 2 04:49 LICENSE drwxrwxr-x. 2 jun jun 4096 Jun 2 04:49 licenses -rw-rw-r--. 1 jun jun 24913 Jun 2 04:49 NOTICE drwxrwxr-x. 8 jun jun 240 Jun 2 04:49 python drwxrwxr-x. 3 jun jun 17 Jun 2 04:49 R -rw-rw-r--. 1 jun jun 3809 Jun 2 04:49 README.md -rw-rw-r--. 1 jun jun 161 Jun 2 04:49 RELEASE drwxrwxr-x. 2 jun jun 4096 Jun 2 04:49 sbin drwxrwxr-x. 2 jun jun 42 Jun 2 04:49 yarn
2.配置Linux环境变量
#spark export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
3.配置spark-env.sh环境变量，注意三个计算机上都必须要这样配置才行
复制默认配置文件并使用gedit打开
[jun@master conf]$ cp ~/spark-2.3.1-bin-hadoop2.7/conf/spark-env.sh.template ~/spark-2.3.1-bin-hadoop2.7/conf/spark-env.sh [jun@master conf]$ gedit ~/spark-2.3.1-bin-hadoop2.7/conf/spark-env.sh
增加下面的配置
export SPARK_MASTER_IP=192.168.1.100 export JAVA_HOME=/usr/java/jdk1.8.0_171/ export SCALA_HOME=/home/jun/scala-2.12.6/
4.修改Spark的slaves文件
使用gedit打开文件
[jun@master conf]$ cp ~/spark-2.3.1-bin-hadoop2.7/conf/slaves.template slaves [jun@master conf]$ gedit ~/spark-2.3.1-bin-hadoop2.7/conf/slaves
删除默认的localhost并增加下面的配置
# A Spark Worker will be started on each of the machines listed below.
slave0
slave1
5.将Spark复制到Slave节点
三、Spark集群的启动与验证
1.启动Spark集群
首先确保Hadoop集群处于启动状态，然后执行启动脚本
[jun@master conf]$ /home/jun/spark-2.3.1-bin-hadoop2.7/sbin/start-all.sh starting org.apache.spark.deploy.master.Master, logging to /home/jun/spark-2.3.1-bin-hadoop2.7/logs/spark-jun-org.apache.spark.deploy.master.Master-1-master.out slave0: starting org.apache.spark.deploy.worker.Worker, logging to /home/jun/spark-2.3.1-bin-hadoop2.7/logs/spark-jun-org.apache.spark.deploy.worker.Worker-1-slave0.out slave1: starting org.apache.spark.deploy.worker.Worker, logging to /home/jun/spark-2.3.1-bin-hadoop2.7/logs/spark-jun-org.apache.spark.deploy.worker.Worker-1-slave1.out
2.验证启动状态
（1）通过jps查看进程，可以看到master节点上增加了Master进程，而slave节点上增加了Worker进程
[jun@master conf]$ jps 8480 ResourceManager 8273 SecondaryNameNode 8805 Master 8038 NameNode 8889 Jps [jun@slave0 ~]$ jps 7794 DataNode 8166 Worker 7961 NodeManager 8252 Jps [jun@slave1 ~]$ jps 5539 DataNode 5917 Worker 5711 NodeManager 5999 Jps
（2）通过Web查看系统状态，输入http://master:8080
（3）通过终端命令行向Spark集群提交计算程序
为了直接在目录下加载jar包，先将示例程序jar包复制到Spark主安装目录下
[jun@master conf]$ cp /home/jun/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar /home/jun/spark-2.3.1-bin-hadoop2.7/
执行SparkPi程序
[jun@master spark-2.3.1-bin-hadoop2.7]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 spark-examples_2.11-2.3.1.jar 10
这个时候报了一个错误，意思就是container要用2.3G内存，而实际的虚拟内存只有2.1G。Yarn默认的虚拟内存和物理内存比例是2.1，也就是说虚拟内存是2.1G，小于了需要的内存2.2G。解决的办法是把拟内存和物理内存比例增大，在yarn-site.xml中增加一个设置：
diagnostics: Application application_1532350446978_0001 failed 2 times due to AM Container for appattempt_1532350446978_0001_000002 exited with exitCode: -103 Failing this attempt.Diagnostics: Container [pid=6514,containerID=container_1532350446978_0001_02_000001] is running beyond virtual memory limits. Current usage: 289.7 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
关闭Yarn然后在配置文件中增加下面的配置，然后重启Yarn
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.5</value>
</property>
再次运行SparkPi程序，在final status可以看到运行成功！
2018-07-23 21:26:53 INFO Client:54 - client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.1.101 ApplicationMaster RPC port: 0 queue: default start time: 1532352353581 final status: SUCCEEDED tracking URL: http://master:18088/proxy/application_1532352327714_0001/ user: jun 2018-07-23 21:26:53 INFO ShutdownHookManager:54 - Shutdown hook called 2018-07-23 21:26:53 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-1ed5bee9-1aa7-4888-b3ec-80ff2b153192 2018-07-23 21:26:53 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-7349a4e3-9483-4d09-91ff-e1e48cb59b46
在tracking URL上右键然后选择open link即可在浏览器看到运行状态
点击logs，然后点击stdout，可以看到运行结果Pi is roughly 3.143951143951144
