HDFS NameNode HA 部署文档

简介：

HDFS High Availability Using the Quorum Journal Manager

Hadoop 2.x 中，HDFS 组件有三个角色：NameNode、DataNode、Secondary NameNode

1、NameNode

保存着 HDFS 的元数据信息，包括命名空间、块信息等。

维护着两种文件：fsimage、edit logs

2、DataNode

保存着 HDFS 的数据信息。

3、Secondary NameNode

负责合并 edit logs 到 fsimage。

# NameNode 会将元数据信息 fsimage 加载到内存中，供 DataNode 使用，将每次的修改保存在 edit logs 中。
# 每次 NameNode 重启时，将 edit logs 合并到 fsimage 中。
# 然而生产环境中 NameNode 很少重启，这就导致 edit logs 文件无限变大，当重启 NameNode 时，合并时间变长、启动时间变长。

# Secondary NameNode 负责定期将 edit logs 合并到 fsimage 然后同步给 NameNode，避免上述问题。

Hadoop 2.x 默认情况下，整个集群只有一台服务器充当 NameNode 角色，如果 NameNode 宕机，则 HDFS 及所有依赖 HDFS 的组件都无法访问。

由此可见 NameNode 的重要性，所以需要给 NameNode 做 HA ！

  角色分配如下：

  192.168.1.25 master.hadoop      (NameNode Active 、ZKFC)
  192.168.1.26 backup.hadoop      (NameNode Standby、ZKFC)
  192.168.1.27 datanode01.hadoop  (DataNode、ZooKeeper Server、JournalNode)
  192.168.1.28 datanode02.hadoop  (DataNode、ZooKeeper Server、JournalNode)
  192.168.1.29 datanode03.hadoop  (DataNode、ZooKeeper Server、JournalNode)

一、搭建 Hadoop 集群

文档链接：http://www.cnblogs.com/wangxiaoqiangs/p/6700717.html

# 停止集群 sh sbin/stop-all.sh

二、搭建 ZooKeeper 集群

文档链接：http://www.cnblogs.com/wangxiaoqiangs/p/6809065.html

shell > ansible datanode -m shell -a '/usr/local/zookeeper-3.4.10/bin/zkServer.sh status'
datanode02.hadoop | SUCCESS | rc=0 >>
Mode: followerZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

datanode03.hadoop | SUCCESS | rc=0 >>
Mode: leaderZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

datanode01.hadoop | SUCCESS | rc=0 >>
Mode: followerZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

shell > sh bin/zkCli.sh -server 192.168.1.27:2181,192.168.1.28:2181,192.168.1.29:2181

[zk: 192.168.1.27:2181,192.168.1.28:2181,192.168.1.29:2181(CONNECTED) 0] ls /
[zookeeper]

# 验证集群状态

三、配置 HDFS NameNode HA

1、修改 core-site.xml

shell > vim etc/hadoop/core-site.xml

<configuration>

    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://myhdfs</value>
    </property>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///data/hadoop/tmp</value>
    </property>

    <property>
        <name>fs.trash.interval</name>
        <value>1440</value>
    </property>

    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>

    <property>
        <name>ha.zookeeper.quorum</name>
        <value>192.168.1.27:2181,192.168.1.28:2181,192.168.1.29:2181</value>
    </property>

    <property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>5000</value>
    </property>

</configuration>

# fs.defaultFS 指定的不再是某台 NameNode 的地址，是一个逻辑名称 ( hdfs-site.xml 文件中定义 )
# ha.zookeeper.quorum 指定 ZooKeeper Server 地址
# ha.zookeeper.session-timeout NameNode 与 ZooKeeper Server 超时时间，超时会发生主备切换

2、修改 hdfs-site.xml

shell > vim etc/hadoop/hdfs-site.xml

<configuration>

    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///data/dfs/nn</value>
    </property>

    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>file:///data/dfs/sn</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///data/dfs/dn</value>
    </property>

    <property>
        <name>dfs.namenode.handler.count</name>
        <value>20</value>
    </property>

    <property>
        <name>dfs.nameservices</name>
        <value>myhdfs</value>
    </property>

    <property>
        <name>dfs.ha.namenodes.myhdfs</name>
        <value>nn1,nn2</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.myhdfs.nn1</name>
        <value>master.hadoop:8020</value>
    </property>
    
    <property>
        <name>dfs.namenode.rpc-address.myhdfs.nn2</name>
        <value>backup.hadoop:8020</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.myhdfs.nn1</name>
        <value>master.hadoop:50070</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.myhdfs.nn2</name>
        <value>backup.hadoop:50070</value>
    </property>

    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://192.168.1.27:8485;192.168.1.28:8485;192.168.1.29:8485/journal</value>
    </property>

    <property>
        <name>dfs.client.failover.proxy.provider.myhdfs</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>

    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/data/dfs/journal</value>
    </property>

    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hadoop/.ssh/id_rsa</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>5000</value>
    </property>

</configuration>

四、初始化 backup.hadoop

# 根据集群部署文档来做 ( 修改主机名、配置hosts、创建hadoop用户、密钥认证、java环境、修改/data目录权限等 )

# 密钥认证要与原 NameNode、DataNode 都做

五、同步配置文件

1、修改 HOSTS

shell > vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.25  master.hadoop
192.168.1.26  backup.hadoop
192.168.1.27  datanode01.hadoop
192.168.1.28  datanode02.hadoop
192.168.1.29  datanode03.hadoop

2、密钥认证

shell > ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 root@backup.hadoop"

shell > su - hadoop

hadoop shell > ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 hadoop@backup.hadoop"

3、同步配置文件

shell > ansible all -m synchronize -a 'src=/usr/local/hadoop-2.8.0 dest=/usr/local/'

六、初始化、启动 NameNode HA

1、初始化 zkfc

shell > su - hadoop

hadoop shell > hdfs zkfc -formatZK

INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/myhdfs in ZK.

2、启动 journalnode

hadoop shell > ansible datanode -m shell -a '/usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start journalnode'

3、初始化 namenode，启动 NameNode

hadoop shell > hadoop namenode -format myhdfs

hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start namenode

4、backup.hadoop 拷贝元数据、启动 NameNode

hadoop shell > hdfs namenode -bootstrapStandby  # 从 master.hadoop 拷贝 NameNode 元数据

hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start namenode

5、master.hadoop、backup.hadoop 启动 zkfc

hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start zkfc

6、启动 YARN

hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/start-yarn.sh

6、启动 DataNode

hadoop shell > ansible datanode -m shell -a '/usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start datanode'

# master.hadoop

8067 NameNode
8376 ResourceManager
8230 DFSZKFailoverController

# backup.hadoop

8255 NameNode
8541 DFSZKFailoverController

# datanode.hadoop

13227 DataNode
13000 NodeManager
12889 JournalNode

七、验证 NameNode HA

# 访问 http://192.168.1.25:50070 显示 active
# 访问 http://192.168.1.26:50070 显示 standby

hadoop shell > kill 8067  # 杀掉 master.hadoop 上的 NameNode

hadoop shell > tail -f /usr/local/hadoop-2.8.0/logs/hadoop-hadoop-zkfc-backup.hadoop.log  # backup.hadoop 监控 zkfc 日志

2017-05-22 14:08:50,035 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at backup.hadoop/192.168.1.26:8020 active...
2017-05-22 14:08:50,846 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at backup.hadoop/192.168.1.26:8020 to active state

# 有待进一步完善