Hadoop 单机搭建 Spark单机搭建

spark下载地址：http://spark.apache.org/downloads.html

spark版本：spark-3.0.0-preview2-bin-hadoop2.7

一、下载并上传解压安装包

tar -xvf spark-3.0.0-preview2-bin-hadoop2.7.tar -C /opt/modules/

二、修改配置文件,配置文件都在安装目录的conf文件夹中

/opt/modules/spark-3.0.0-preview2-bin-hadoop2.7/conf

1、修改spark-env.sh文件

[hadoop@centos01 conf]$ cp spark-env.sh.template spark-env.sh
[hadoop@centos01 conf]$ vi spark-env.sh

export JAVA_HOME=/opt/modules/jdk1.8.0_181
export SPARK_MASTER_IP=centos01
export SPARK_MASTER_PORT=7077

三、启动

sh /opt/modules/spark-3.0.0-preview2-bin-hadoop2.7/sbin/start-master.sh

[hadoop@centos04 conf]$ sh /opt/modules/spark-3.0.0-preview2-bin-hadoop2.7/sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/modules/spark-3.0.0-preview2-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-centos04.out
[hadoop@centos04 conf]$ 

四、查看节点

[hadoop@centos04 conf]$ jps
10579 Master
11427 Jps
[hadoop@centos04 conf]$ 

五、测试

http://192.168.0.181:8080/

连接spark-shell
sh /opt/modules/spark-3.0.0-preview2-bin-hadoop2.7/bin/spark-shell --master spark://192.168.0.181:7077

[hadoop@centos04 conf]$ sh /opt/modules/spark-3.0.0-preview2-bin-hadoop2.7/bin/spark-shell --master spark://192.168.0.181:7077
20/03/21 00:40:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://centos04:4040
Spark context available as 'sc' (master = spark://192.168.0.181:7077, app id = app-20200321004034-0000).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_   version 3.0.0-preview2
      /_/
         
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.


scala> :quit
[hadoop@centos04 conf]$ 

结束