原有环境
http://www.cnblogs.com/ilifeilong/p/7406944.html
| IP | host | JDK | linux | hadop | role |
| 172.16.101.55 | sht-sgmhadoopnn-01 | 1.8.0_111 | CentOS release 6.5 | hadoop-2.7.3 | NameNode,SecondaryNameNode,ResourceManager |
| 172.16.101.58 | sht-sgmhadoopdn-01 | 1.8.0_111 | CentOS release 6.5 | hadoop-2.7.3 | DataNode,NodeManager |
| 172.16.101.59 | sht-sgmhadoopdn-02 | 1.8.0_111 | CentOS release 6.5 | hadoop-2.7.3 | DataNode,NodeManager |
| 172.16.101.60 | sht-sgmhadoopdn-03 | 1.8.0_111 | CentOS release 6.5 | hadoop-2.7.3 | DataNode,NodeManager |
| 172.16.101.66 | sht-sgmhadoopdn-04 | 1.8.0_111 | CentOS release 6.5 | hadoop-2.7.3 | DataNode,NodeManager |
现计划向集群新增一台datanode,如表格所示
1. 配置系统环境
主机名,ssh互信,环境变量等
2. 修改namenode节点的slave文件,增加新节点信息
$ cat slaves sht-sgmhadoopdn-01 sht-sgmhadoopdn-02 sht-sgmhadoopdn-03 sht-sgmhadoopdn-04
3. 在namenode节点上,将hadoop-2.7.3复制到新节点上,并在新节点上删除data和logs目录中的文件
$ hostname sht-sgmhadoopnn-01 $ rsync -az --progress /usr/local/hadoop-2.7.3/* hduser@sht-sgmhadoopdn-04:/usr/local/hadoop-2.7.3/ $ hostname sht-sgmhadoopdn-04 $ rm -rf /usr/local/hadoop-2.7.3/logs/* $ rm -rf /usr/local/hadoop-2.7.3/data/*
4. 启动新datanode的datanode进程
$ hadoop-daemon.sh start datanode starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hduser-datanode-sht-sgmhadoopdn-04.out $ jps 31875 Jps 31821 DataNode
5. 在namenode查看当前集群情况,确认信节点已经正常加入
5.1 以命令行方式
$ hdfs dfsadmin -report Configured Capacity: 303324561408 (282.49 GB) Present Capacity: 83729309696 (77.98 GB) DFS Remaining: 83081265152 (77.38 GB) DFS Used: 648044544 (618.02 MB) DFS Used%: 0.77% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (4): Name: 172.16.101.66:50010 (sht-sgmhadoopdn-04) Hostname: sht-sgmhadoopdn-04 Decommission Status : Normal Configured Capacity: 75831140352 (70.62 GB) DFS Used: 24576 (24 KB) Non DFS Used: 35573932032 (33.13 GB) DFS Remaining: 40257183744 (37.49 GB) DFS Used%: 0.00% DFS Remaining%: 53.09% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Sep 01 22:50:16 CST 2017 Name: 172.16.101.60:50010 (sht-sgmhadoopdn-03) Hostname: sht-sgmhadoopdn-03 Decommission Status : Normal Configured Capacity: 75831140352 (70.62 GB) DFS Used: 216006656 (206 MB) Non DFS Used: 61714608128 (57.48 GB) DFS Remaining: 13900525568 (12.95 GB) DFS Used%: 0.28% DFS Remaining%: 18.33% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Sep 01 22:50:15 CST 2017 Name: 172.16.101.59:50010 (sht-sgmhadoopdn-02) Hostname: sht-sgmhadoopdn-02 Decommission Status : Normal Configured Capacity: 75831140352 (70.62 GB) DFS Used: 216006656 (206 MB) Non DFS Used: 62057410560 (57.80 GB) DFS Remaining: 13557723136 (12.63 GB) DFS Used%: 0.28% DFS Remaining%: 17.88% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Sep 01 22:50:14 CST 2017 Name: 172.16.101.58:50010 (sht-sgmhadoopdn-01) Hostname: sht-sgmhadoopdn-01 Decommission Status : Normal Configured Capacity: 75831140352 (70.62 GB) DFS Used: 216006656 (206 MB) Non DFS Used: 60249300992 (56.11 GB) DFS Remaining: 15365832704 (14.31 GB) DFS Used%: 0.28% DFS Remaining%: 20.26% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Sep 01 22:50:15 CST 2017
5.2 以web方式
6. 在namenoe上设置 hdfs 的负载均衡
$ hdfs dfsadmin -setBalancerBandwidth 67108864 Balancer bandwidth is set to 67108864 $ start-balancer.sh -threshold 5 starting balancer, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hduser-balancer-sht-sgmhadoopnn-01.out
7. 查看hdfs负载信息(有时候节点数据量较小,看出来数据量变化,可以上传大文件测试)
8. 启动新节点的nodemanager进程
$ yarn-daemon.sh start nodemanager starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hduser-nodemanager-sht-sgmhadoopdn-04.out $ jps 32562 NodeManager 32599 Jps 31821 DataNode
