.Hadoop NameNode单点问题解决方案之二 AvatarNode 部署

采用FACEBOOK的AvatarNode来实现Hadoop的双机热备功能具体如下

一.环境

机器

Primary:192.168.17.130               
Standby/NFS:192.168.17.131 
AvatarDataNode:192.168.17.132

软件

hadoop-release-0.20.2编译生成的hadoop-0.20.2-highavailability.jar,hadoop-0.20.3-dev-core.jar两个更新包

 

二.具体部署过程

1.在192.168.17.130,192.168.17.131,192.168.17.132上创建如下目录

 /home/hadoop/share

 /home/hadoop/name

 /home/hadoop/data

 /home/hadoop/tmp

2.在192.168.17.131 设置NFS 共享目录为 /home/hadoop/share/

3.192.168.17.130 挂载NFS   mount 192.168.17.131://home/hadoop/share/   /home/hadoop/share/

4.在NFS上创建  shared0,shared1两个目录

5.关闭所有机器上的防火墙   /etc/init.d/iptable stop

6.192.168.17.130与192.168.17.131相互之间能无密码SSH访问，并能无密码SSH访问数据节点

7.配置hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
       <name>dfs.name.dir</name>
       <value>/home/hadoop/name</value>
   </property>
   <property>
       <name>dfs.data.dir</name>
       <value>/home/hadoop/data</value>
   </property>
   <property>
       <name>dfs.http.address0</name>
       <value>192.168.17.130:50070</value>
   </property>
   <property>
      <name>dfs.http.address1</name>
      <value>192.168.17.131:50070</value>
   </property>
   <property>
      <name>dfs.name.dir.shared0</name>
      <value>/home/hadoop/share/shared0</value>
   </property>
   <property>
      <name>dfs.name.dir.shared1</name>
      <value>/home/hadoop/share/shared1</value>
   </property>
   <property>
      <name>dfs.name.edits.dir.shared0</name>
      <value>/home/hadoop/share/shared0</value>
   </property>
   <property>
      <name>dfs.name.edits.dir.shared1</name>
      <value>/home/hadoop/share/shared1</value>
   </property>
   <property>
      <name>dfs.replication</name>
      <value>1</value>
   </property>
</configuration>

8.配置core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
        <property>
               <name>hadoop.tmp.dir</name>
                <value>/home/hadoop/tmp</value>
       </property>
       <property>
              <name>fs.default.name</name>
              <value>hdfs://192.168.17.130:9000</value>
       </property> 
       <property>
             <name>fs.default.name0</name>
             <value>hdfs://192.168.17.130:9000</value>
       </property> 
       <property>
               <name>fs.default.name1</name>
               <value>hdfs://192.168.17.131:9000</value>
      </property>

      <property>
               <name>dfs.hosts.exclude</name>
               <value>excludes</value>
      </property>

</configuration>

9.节点配置

Primary:192.168.17.130 的 conf/master 为 192.168.17.130
Standby/NFS:192.168.17.131  的 conf/master 为 192.168.17.131

conf/slaves 为 192.168.17.132

10.分发文件至各服务器

    conf/core-site.xml

    conf/hdfs-site.xml

    {HADOOP_HOME}/hadoop-0.20.3-dev-core.jar

    {HADOOP_HOME}/lib/hadoop-0.20.2-highavailability.jar

11.格式化元数据

   <1. 在192.168.17.130上   {HADOOP_HOME}/bin/hadoop namenode -format

   <2. 查看是否格式化成功   /home/hadoop/name

    同步元数据

   <3. 将scp /home/hadoop/name/*  192.168.17.130://home/hadoop/name/ -R  

   <4. cp /home/hadoop/name/*  /home/hadoop/share/shared0/ -R

   <5. cp /home/hadoop/name/* /home/hadoop/share/shared1/ -R

12.启动

bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -zero        主节点

bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -one -standby 从节点

bin/hadoop org.apache.hadoop.hdfs.server.datanode.AvatarDataNode 数据节点