Hadoop HA架构搭建

Hadoop HA架构搭建

共七台服务器，节点角色分配如下：

192.168.133.21 （BFLN-01）：namenode  zookeeper  journalnade DFSZKFailoverController
192.168.133.23 （BFLN-02）：namenode resourcemanager zookeeper  journalnade DFSZKFailoverController
192.168.133.24 （BFLN-03）：resourcemanager zookeeper  journalnade DFSZKFailoverController
192.168.133.25 （BFLN-04）：datanode，nodemanager
192.168.133.26 （BFLN-05）：datanode，nodemanager
192.168.133.27 （BFLN-06）：datanode，nodemanager
192.168.133.28 （BFLN-07）：datanode，nodemanager

HA优势：双namedata和resourcemanager能防止hadoop核心组件单点故障导致集群不可用情况的发生。

配置步骤：

环境配置

1、集群间需实现时间同步：

 ntpdate

2、配置7台服务器的主机名解析/etc/hosts(每台都要配置):

192.168.133.21  BFLN-01
192.168.133.23  BFLN-02
192.168.133.24  BFLN-03
192.168.133.25  BFLN-04
192.168.133.26  BFLN-05
192.168.133.27  BFLN-06
192.168.133.28  BFLN-07

3、配置ssh服务/etc/ssh/sshd.conf

StrictHostKeyChecking no

UserKnownHostsFile /dev/null

不然启动hdfs服务的时候可能会异常：

Starting namenodes on [BFLN-01 BFLN-02]
The authenticity of host 'BFLN-02 (192.168.133.23)' can't be established.
ECDSA key fingerprint is 79:d1:ec:82:d3:1c:50:8a:17:c2:2d:f0:87:20:53:44.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'BFLN-01 (192.168.133.21)' can't be established.
ECDSA key fingerprint is 30:75:04:10:93:d2:57:d7:3d:b1:cc:31:92:30:1a:a1.
Are you sure you want to continue connecting (yes/no)? yes

4、每台服务器实现ssh无密钥认证，包括本机与本机的免密钥认证:

ssh-keygren ：生成一对密钥

ssh-copy-id : 把公钥发给对方服务器

5、配置安装JAVA环境并配置JAVA和hadoop环境变量：

export JAVA_HOME=/usr/java/jdk1.8.0_51/

export HADOOP_HOME=/opt/hadoop-spark/hadoop/hadoop-2.9.1

PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

安装zookeeper集群：

7、解压zookeeper压缩包。

8、修改zookeeper配置文件：

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/data/zookeeper

# the port at which the clients will connect

clientPort=2181

server.1=192.168.133.21:2888:3888

server.2=192.168.133.23:2888:3888

server.3=192.168.133.24:2888:3888

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60

#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

9、zookeeper数据路径文件下添加个代表zookeeper节点id的myid文件(本配置文件的数据路径为/data/zookeeper，节点id分别为1，2，3)

10、启动zookeeper集群

./zkServer.sh start

安装配置Hadoop-HA:

11、下载hadoop-spark压缩包，解压，尽量保持7台服务器的hadoop安装路径是一致的。

在192.168.133.21上配置：

cd $HADOOP_HOME/etc/hadoop/

vi core-site.xml

<configuration>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://BFLN</value>   <!--#BFLN为nodename集群的代理名字，此名字要和hdfs-site.xml配置的dfs.nameservices集群名字一致-->

    </property>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>/data/hadoop-spark/hadoop/tmp</value>    <!--#指定hdfs目录-->

    </property>

 <property>

      <name>ha.zookeeper.quorum</name>

      <value>BFLN-01:2181,BFLN-02:2181,BFLN-03:2181</value>  <！--配置zookeeper集群的地址-->

 </property>

</configuration>

 

vi hdfs-site.xml

<configuration>

    <!-- #BFLN为nodename集群的代理名字，此名字要和core-site.xml配置的fs.defaultFS集群名字一致 -->

    <property>

        <name>dfs.nameservices</name>

        <value>BFLN</value>

    </property>

    <!-- BFLN集群下有两个namenode节点，分别为BFLN1,BFLN2 -->

    <property>

       <name>dfs.ha.namenodes.BFLN</name>

       <value>BFLN1,BFLN2</value>

    </property>

    <!-- 配置namenode第一节点的rpc通信端口 -->

    <property>

       <name>dfs.namenode.rpc-address.BFLN.BFLN1</name>

       <value>BFLN-01:9000</value>

    </property>

    <!-- 配置namenode第一节点的http通信端口 -->

    <property>

        <name>dfs.namenode.http-address.BFLN.BFLN1</name>

        <value>BFLN-01:50070</value>

    </property>

    <!-- 配置namenode第二节点的rpc通信端口 -->

    <property>

        <name>dfs.namenode.rpc-address.BFLN.BFLN2</name>

        <value>BFLN-02:9000</value>

    </property>

    <!-- 配置namenode第二节点的http通信端口 -->

    <property>

        <name>dfs.namenode.http-address.BFLN.BFLN2</name>

        <value>BFLN-02:50070</value>

    </property>

    <!-- 配置journalnade互连的地址及端口，官网建议journalnade节点数为奇数 -->

    <property>

        <name>dfs.namenode.shared.edits.dir</name>

        <value>qjournal://BFLN-01:8485;BFLN-02:8485/BFLN</value>

    </property>

    <!-- 指定JournalNode在本地磁盘存放数据的位置 -->

    <property>

          <name>dfs.journalnode.edits.dir</name>

          <value>/data/hadoop-spark/hadoop/tmp/jn</value>

    </property>

    <!-- 开启NameNode故障时自动切换 -->

    <property>

          <name>dfs.ha.automatic-failover.enabled</name>

          <value>true</value>

    </property>

    <!--配置失败自动切换实现方式-->

    <property>

            <name>dfs.client.failover.proxy.provider.BFLN</name>

            <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

    </property>

    <!--配置当namenode出现脑裂时，hdfs对其处理的方式，sshfenc会自动通过ssh到old-active将其杀掉，将standby切换为active-->

    <property>

             <name>dfs.ha.fencing.methods</name>

             <value>sshfence</value>

    </property>

 

    <!--配置HA namenode通信公钥的地址-->

    <property>

            <name>dfs.ha.fencing.ssh.private-key-files</name>

            <value>/root/.ssh/id_rsa</value>

    </property>

 

    <!--配置启动集群代理，如果此选项没有配置，后期启动的时候hadoop会把集群名称BFLN当成主机名与之通信，导致报错-->

    <property> 

        <name>dfs.client.failover.proxy.provider.BFLN</name>

        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

    </property>

 

    <!--配置副本数-->

    <property>

        <name>dfs.replication</name>

        <value>3</value>

    </property> 

 

    <!--配置是否检查权限-->

    <property>

        <name>dfs.permissions</name>

        <value>false</value>

    </property>

</configuration>

vi yarn-site.conf

<configuration>

  <!-- 开启resourcemanager HA服务，默认是false -->

  <property>

    <name>yarn.resourcemanager.ha.enabled</name>

    <value>true</value>

  </property>

  <!-- 开启RM重启的功能，作用：当yarn中有任务在跑时，如果rm宕机，设置成ture，rm重启时会恢复原来没有跑完的application -->

  <property>

    <name>yarn.resourcemanager.recovery.enabled</name>

    <value>true</value>

  </property>

 

  <!--  配置RM集群ID  -->

  <property>

    <name>yarn.resourcemanager.cluster-id</name>

    <value>BFLN-yarn</value>

  </property>

 

 

  <!--RM集群下的两个RM节点名称  -->

  <property>

    <name>yarn.resourcemanager.ha.rm-ids</name>

    <value>BFLN-yarn1,BFLN-yarn2</value>

  </property>

 

 

  <!--  BFLN-yarn1节点的地址  -->

  <property>

    <name>yarn.resourcemanager.hostname.BFLN-yarn1</name>

    <value>BFLN-02</value>

  </property>

 

 

  <!--  BFLN-yarn2节点的地址  -->

  <property>

    <name>yarn.resourcemanager.hostname.BFLN-yarn2</name>

    <value>BFLN-03</value>

  </property>

 

 

  <!-- zookeeper集群的地址  -->

  <property>

    <name>yarn.resourcemanager.zk-address</name>

    <value>BFLN-01:2181,BFLN-02:2181,BFLN-03:2181</value>

  </property>

 

 

  <!-- 用于状态存储的类，默认是基于Hadoop 文件系统的实现（FileSystemStateStore）  -->

  <property>

    <name>yarn.resourcemanager.store.class</name>

    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

  </property>

 

  <!-- NodeManager上运行的附属服务。需配置成mapreduce_shuffle，才可运行MapReduce程序  -->

  <property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

    </property>

</configuration>

vi slaves:

(配置datanode节点)

192.168.133.25

192.168.133.26

192.168.133.27

192.168.133.28

 

vi hadoop-env.sh:

export JAVA_HOME=/usr/java/jdk1.8.0_51/ 

 

12：至此，所有的配置以及配置完成，需要将这几个文件复制发送给其他服务器。

启动HDFS服务：

注意：启动顺序很重要，顺序错了会导致后期频繁报错！

1、在BFLN-01上启动journalnode，命令：./sbin/hadoop-daemon.sh start journalnode # 启动 journalnode

2、在BFLN-01上格式化namenode，命令：./bin/hdfs namenode -format  # 格式化namemode路径

3、在BFLN-01上注册zookeeper，命令：./bin/hdfs zkfc -formatZK    # 向zookeeper集群注册hdfs

4、在BFLN-01上启动namenode，命令：./sbin/start-dfs.sh   # 启动hdfs服务，注意，此时只会启动BFLN-01上的namenode

5、在BFLN-02上同步namenode，命令：./bin/hdfs namenode -bootstrapStandby  # BFLN-02节点的namenode从BFLN-01上的namenode同步元数据。

6、在BFLN-02上启动namenode，命令：./sbin/hadoop-daemon.sh start namenode   # 在BFLN-02上启动namenode节点

7、在BFLN-02上启动resourcemanager，命令：./sbin/start-yarn.sh  #启动RM,NM服务

8、在0BFLN-02上启动resourcemanager，命令：./sbin/yarn-daemon.sh start resourcemanager  #启动备用RM服务。

 

测试：kill一个为active的namenode/resourcemanager节点，查看另外一个standby节点是否转化成active节点：

查看namenode节点状态的命令：

./bin/hdfs  haadmin -getServiceState BFLN1

./bin/hdfs  haadmin -getServiceState BFLN2

查看resourcemanager节点状态的命令:

./bin/yarn  rmadmin -getServiceState BFLN-yarn1

./bin/yarn  rmadmin -getServiceState BFLN-yarn2

如果kill active节点后standby节点无法切换成active节点，可能系统需要安装一个软件：

psmisc