Spark Standalone集群搭建

1. 环境说明
   • Win10
   • VMware 15.0.2
   • Ubuntu 18.10
2. 软件版本说明
   • hadoop-2.7.7.tar.gz
   • jdk-8u212-linux-x64.tar.gz
   • sbt-1.2.8.tgz
   • scala-2.12.8.tgz
   • spark-2.4.3-bin-hadoop2.7.tgz
3. 下载以上压缩包，解压缩后添加环境变量，vi ~./bashrc，在最后添加如下

   # set hadoop env
   export HARDOOP_HOME=/home/gemsuser/install/hadoop-2.7.7
   export PATH=${HARDOOP_HOME}/bin:${PATH}
   
   # set java env
   export JAVA_HOME=/home/gemsuser/install/jdk1.8.0_212
   export JRE_HOME=${JAVA_HOME}/jre
   export CLASS_PATH=${JAVA_HOME}/lib:${JRE_HOME}/lib
   export PATH=${JAVA_HOME}/bin:${PATH}
   
   # set scala env
   export SCALA_HOME=/home/gemsuser/install/scala-2.12.8
   export PATH=${SCALA_HOME}/bin:${PATH}
   
   # set sbt env
   export SBT_HOME=/home/gemsuser/install/sbt
   export PATH=${SBT_HOME}/bin:${PATH}
   
   # set spark env
   export SPARK_HOME=/home/gemsuser/install/spark-2.4.3-bin-hadoop2.7
   export PATH=${SPARK_HOME}/bin:${PATH}

4. 调整日志级别
   log4j.properties：复制template文件，可修改log4j.rootCategory=INFO, console，更改为WARN，修改日志打印级别
   • spark-env.sh：复制template文件，配置spark运行所需要的环境变量
5. 配置spark的config文件

   export JAVA_HOME=/home/gemsuser/install/jdk1.8.0_212
   export SCALA_HOME=/home/gemsuser/install/scala-2.12.8
   export SPARK_HOME=/home/gemsuser/install/spark-2.4.3-bin-hadoop2.7
   export HADOOP_CONF_DIR=/home/gemsuser/install/spark-2.4.3-bin-hadoop2.7/conf
   export SPARK_LOG_DIR=/home/gemsuser/install/spark-2.4.3-bin-hadoop2.7/logs
   export SPARK_PID_DIR=/home/gemsuser/install/spark-2.4.3-bin-hadoop2.7/pid
   export SPARK_MASTER_IP=localhost
   export SPARK_MASTER_HOST=localhost
   export SPARK_LOCAL_IP=localhost
   export SPARK_WORKER_MEMORY=1G

   maste
   slava1
   slave2

6. 配置maste无密登录slave
   • ssh-keygen -t rsa
   • cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   • 修改/etc/ssh/sshd_config：
7. 启动Spark进程：