Hadoop配置机架感知

配置机架感知

core-site.xml

cat $HADOOP_HOME/etc/hadoop/core-site.xml

<property>

      <name>net.topology.script.file.name</name>

      <value>pathdir/RackAware.py</value>

</property>

RackAware.py

cat etc/hadoop/RackAware.py            

#!/usr/bin/python 

#-*-coding:UTF-8 -*- 

import sys 

rack = {"bs022.zx.nicx.cn":"rack1", 

        "bs035.zx.nicx.cn":"rack2", 

        "bs038.zx.nicx.cn":"rack2", 

        "bs042.zx.nicx.cn":"rack3",

        "192.168.1.22":"rack1", 

        "192.168.1.35":"rack2", 

        "192.168.1.38":"rack2", 

        "192.168.1.42":"rack3",

        } 

    

if __name__=="__main__": 

print "/" + rack.get(sys.argv[1])

机架感知验证

bin/hdfs dfsadmin -printTopology

Rack: /rack1

   192.168.1.22:50010 (bs022.zx.nicx.cn)

Rack: /rack2

   192.168.1.35:50010 (bs035.zx.nicx.cn)

   192.168.1.38:50010 (bs038.zx.nicx.cn)

Rack: /rack3

   192.168.1.42:50010 (bs042.zx.nicx.cn)

机架感知配置完成

注：

         在python脚本中配置的时候，需要兼容主机名和IP地址

         RackAware.py需要有可执行权限

         Namenode的日志中含有：

        

2016-07-04 14:42:22,198 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.38:50010
……
2016-07-04 14:42:22,232 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.35:50010
……
2016-07-04 14:42:22,263 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.42:50010
……
2016-07-04 14:42:22,295 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.22:50010
……
2016-07-04 14:42:52,503 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 4 datanodes

 参考自：http://blog.csdn.net/azhao_dn/article/details/7091258