HDFS重启集群导致数据损坏，使用fsck命令修复过程
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
我们先看一组输出
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new/2019/05/23/ Connecting to namenode via http://calculation111.aggrx:50070 FSCK started by root (auth:SIMPLE) from /10.1.2.112 for path /user/kuaikan/report_new/2019/05/23/ at Sat May 25 17:37:43 CST 2019 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ....................................Status: HEALTHY Total size: 48127214482 B (Total open files size: 72230205 B) Total dirs: 1 Total files: 5236 Total symlinks: 0 (Files currently being written: 26) Total blocks (validated): 5190 (avg. block size 9273066 B) (Total open file blocks (not validated): 26) #注意，这个26没有进行验证，说明有26个存在问题！ Minimally replicated blocks: 5190 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 18 Number of racks: 1 FSCK ended at Sat May 25 17:37:43 CST 2019 in 163 milliseconds The filesystem under path '/user/kuaikan/report_new/2019/05/23/' is HEALTHY [root@flume112 ~]#
官方关于fsck命令有对应的解释,我们查看更详细的信息，官方链接：https://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck。
[root@flume112 ~]# hdfs fsck /user/kuaikan/report_new/2019/05/23/ -openforwrite Connecting to namenode via http://calculation111.aggrx:50070 FSCK started by root (auth:SIMPLE) from /10.1.2.112 for path /user/kuaikan/report_new/2019/05/23/ at Sat May 25 17:40:23 CST 2019 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ................................................/user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-45.10-19-64-160.1558554302405.txt 1163794 bytes, 1 block(s), OPENFORWRITE: ../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-45.10-9-109-248.1558554301696.txt 1209239 bytes, 1 block(s), OPENFORWRITE: ....../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-45.10-9-91-71.1558554301576.txt 1276007 bytes, 1 block(s), OPENFORWRITE: ......................................... /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-55.10-19-169-208.1558554901562.txt 1117716 bytes, 1 block(s), OPENFORWRITE: .../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-55.10-9-109-248.1558554901949.txt 1137697 bytes, 1 block(s), OPENFORWRITE: ...../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-55.10-9-88-233.1558554901647.txt 1096156 bytes, 1 block(s), OPENFORWRITE: ......................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ................................................................/user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-09-30.10-19-169-208.1558575001741.txt 8126635 bytes, 1 block(s), OPENFORWRITE: ................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ......................./user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-12-55.10-19-147-93.1558587301545.txt 16778152 bytes, 1 block(s), OPENFORWRITE: ........../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-12-55.10-9-180-88.1558587301803.txt 16877467 bytes, 1 block(s), OPENFORWRITE: .........................../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-13-01.10-9-180-88.1558587709627.txt 15320896 bytes, 1 block(s), OPENFORWRITE: ..................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .........................................................................../user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-103-35.1558601101874.txt 314295 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-113-40.1558601102383.txt 363261 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-143-46.1558601101876.txt 1177059 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-145-58.1558601102255.txt 422652 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-147-93.1558601102225.txt 411212 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-157-38.1558601101874.txt 590968 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-169-208.1558601101557.txt 626874 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-64-160.1558601101782.txt 169801 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-95-33.1558601102283.txt 118028 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-19-98-150.1558601101668.txt 13544 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-9-109-248.1558601101427.txt 1112989 bytes, 1 block(s), OPENFORWRITE: ./user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-9-123-226.1558601101904.txt 1049811 bytes, 1 block(s), OPENFORWRITE: ./user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-9-176-96.1558601101582.txt 728802 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-9-180-88.1558601101302.txt 969764 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-9-88-233.1558601101531.txt 35615 bytes, 1 block(s), OPENFORWRITE: /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-16-45.10-9-91-71.1558601101720.txt 21771 bytes, 1 block(s), OPENFORWRITE: ....... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ..............................................................Status: HEALTHY Total size: 48199444687 B Total dirs: 1 Total files: 5262 Total symlinks: 0 Total blocks (validated): 5216 (avg. block size 9240691 B) Minimally replicated blocks: 5190 (99.50153 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.985046 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 18 Number of racks: 1 FSCK ended at Sat May 25 17:40:23 CST 2019 in 124 milliseconds The filesystem under path '/user/kuaikan/report_new/2019/05/23/' is HEALTHY [root@flume112 ~]#
找到损坏的文件后，我们手动进行回复，注意，我们输入的必须是一个文件，并不是一个目录哟！
[root@calculation111 ~]# hdfs debug recoverLease -path /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-45.10-9-91-71.1558554301576.txt -retries 3 recoverLease returned false. Retrying in 5000 ms... Retry #1 recoverLease SUCCEEDED on /user/kuaikan/report_new/2019/05/23/report.log.2019-05-23-03-45.10-9-91-71.1558554301576.txt [root@calculation111 ~]#
