VM+CentOS+Hadoop+Spark集群搭建

JDK   SCALA 

卸载jdk

rpm -qa | grep jdk

rpm -e --nodeps java-1.7.0-openjdk-1.7.0.221-2.6.18.1.el7.x86_64

rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.221-2.6.18.1.el7.x86_64

rpm -e --nodeps java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64

rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64

 

解压

tar -zxvf jdk-8u231-linux-x64.tar.gz

tar -zxvf scala-2.11.8.tgz

 

环境变量

vim /etc/profile

加

export JAVA_HOME=/home/z/app/jdk1.8.0_231

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

 

export SCALA_HOME=/home/z/app/scala-2.11.8

export PATH=$PATH:$SCALA_HOME/bin

 

最后

source /etc/profile

 

 

静态IP

vim /etc/sysconfig/network-scripts/ifcfg-ens33

改

BOOTPROTO=static

ONBOOT=yes

加(在虚拟网络编辑器里看，IPADDR设置本机IP)

IPADDR=192.168.179.1 

NETMAST=255.255.255.0

GATEWAY=192.168.179.2

DNS1=8.8.8.8

DNS2=8.8.8.4

最后

service network restart

 

关闭防火墙

systemctl stop firewalld.service

systemctl disable firewalld.service

 

关闭 selinux

vim /etc/selinux/config

其中selinux=enforcing 改为disabled

 

修改host

vim /etc/hosts

加(与设置的静态ip一致)

192.168.179.1 master

192.168.179.3 slave1

192.168.179.4 slave2

 

ssh免密登录

master上

ssh-keygen -t rsa

一直回车得到.ssh文件夹

cd .ssh

复制

cat id_rsa.pub>> authorized_keys

authorized_keys复制到slave节点

scp authorized_keys slave1:~/.ssh/

scp authorized_keys slave2:~/.ssh/

slave节点分别执行

ssh-keygen -t rsa

三个节点chmod 600 authorized_keys

测试免密

ssh slave1

ssh slave2

 

 

Hadoop

解压

tar -zxvf hadoop-2.7.2.tar.gz

环境变量

export HADOOP_HOME=/home/z/app/hadoop-2.7.2

export PATH=$HADOOP_HOME/bin:$PATH

source /etc/profile

配置hadoop-env.sh

vim hadoop-2.7.2/etc/hadoop/hadoop-env.sh

JAVA_HOME=/home/z/app/jdk1.8.0_231

配置yarn-env.sh

vim hadoop-2.7.2/etc/hadoop/yarn-env.sh

export JAVA_HOME=/home/z/app/jdk1.8.0_231

以下四个要修改的XML文件最后不要有空行或空格！！！

修改core-site.xml文件

vim hadoop-2.7.2/etc/hadoop/core-site.xml

替换

<configuration>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>/hadoop/tmp</value>

        <description>Abase for other temporary directories.</description>

    </property>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://master:9000</value>

    </property>

    <property>

        <name>io.file.buffer.size</name>

        <value>4096</value>

    </property>

</configuration>

 

创建子目录

mkdir -p hadoop-2.7.2/etc/hadoop/tmp

mkdir -p hadoop-2.7.2/etc/hadoop/hdfs/name

mkdir -p hadoop-2.7.2/etc/hadoop/hdfs/data

 

配置hdfs-site.xml

vim hadoop-2.7.2/etc/hadoop/hdfs-site.xml

替换

<configuration>

    <property>

            <name>dfs.namenode.name.dir</name>

            <value>file:/hadoop/hdfs/name</value>

    </property>

    <property>

            <name>dfs.datanode.data.dir</name>

            <value>file:/hadoop/hdfs/data</value>

    </property>

    <property>

            <name>dfs.replication</name>

            <value>2</value>

    </property>

    <property>

            <name>dfs.namenode.secondary.http-address</name>

            <value>master:9001</value>

    </property>

    <property>

            <name>dfs.webhdfs.enabled</name>

            <value>true</value>

    </property>

</configuration>

 

复制并修改

cd hadoop-2.7.2/etc/hadoop/

cp mapred-site.xml.template mapred-site.xml

vim mapred-site.xml

替换

<configuration>

 

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

        <final>true</final>

    </property>

    <property>

        <name>mapreduce.jobtracker.http.address</name>

        <value>master:50030</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>master:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>master:19888</value>

    </property>

    <property>

         <name>mapred.job.tracker</name>

         <value>http://master:9001</value>

    </property>

 

</configuration>

修改yarn-site.xml

vim yarn-site.xml

替换

<configuration>

 

    <property>

         <name>yarn.resourcemanager.hostname</name>

         <value>master</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>master:8032</value>

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>master:8030</value>

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>master:8031</value>

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>master:8033</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>master:8088</value>

    </property>

 

</configuration>

 

 

配置masters、slaves文件

vim slaves

替换

slave1

slave2

 

vim masters

替换

master

 

master执行格式化

hadoop-2.7.2/bin/hadoop namenode -format

 

master执行启动hadoop

hadoop-2.7.2/sbin/start-all.sh

启动单个

hadoop-2.7.2/sbin/start-yarn.sh

hadoop-2.7.2/sbin/start-hdfs.sh

 

SPARK

tar -zxvf spark-2.4.4-bin-hadoop2.7.tgz

环境变量

vim /etc/profile

export SPARK_HOME=/home/z/app/spark-2.4.4-bin-hadoop2.7

export PATH=PATH:SPARK_HOME/bin

source /etc/profile

验证

spark-shell

 

配置

cd spark-2.4.4-bin-hadoop2.7/conf

cp spark-env.sh.template spark-env.sh

vim spark-env.sh

追加

export SCALA_HOME=/home/z/app/jdk1.8.0_231

export JAVA_HOME=/home/z/app/jdk1.8.0_231

export HADOOP_HOME=/home/z/app/hadoop-2.7.2

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export SPARK_HOME=/home/z/app/spark-2.4.4-bin-hadoop2.7

export SPARK_MASTER_IP=master

export SPARK_EXECUTOR_MEMORY=1G

 

cp slaves.template slaves

vim slave

改为

master

slave1

slave2

 

复制到slave里

scp -r spark-2.4.4-bin-hadoop2.7 root@slave1:/home/z/app/

 

 

启动spark

hadoop-2.7.2/sbin/start-all.sh

spark-2.4.4-bin-hadoop2.7/sbin/start-all.sh

查看spark集群(masterIP)

http://192.168.179.1:8080/

 

修改了/etc/profile文件后命令失效

执行命令export PATH=/usr/bin:/usr/sbin:/bin:/sbin:/usr/X11R6/bin后可以暂时恢复，修改/etc/profile中的错误

设置共享文件夹

安装vmtools 设置里面设置共享，重启，文件夹在/mnt/hgfs/下