环境:cdh集群5.15 ,redhat7
有一台zookeeper无法启动,查看日志报错如下
2020-07-04 14:24:23,880 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial. 2020-07-04 14:24:23,881 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial. 2020-07-04 14:24:23,881 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial. 2020-07-04 14:24:23,881 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial. 2020-07-04 14:24:23,882 ERROR org.apache.zookeeper.server.quorum.QuorumPeer: Unable to load database on disk java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:610) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:629) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:595) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:677) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:209) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:530) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:520) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:167) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79) 2020-07-04 14:24:23,891 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:578) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:520) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:167) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:610) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:629) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:595) at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:677) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:209) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:530)
报错解析: 磁盘写满导致无法写入数据导致
问题解决:进入到zookeeper的数据目录,删除文件大小为0的日志文件,然后重启节点,重启成功
[dip@dip007 zookeeper]$ cd version-2/ [dip@dip007 version-2]$ ls acceptedEpoch log.210069890e log.21006b70bc log.21006d6e04 log.21006f2bbc log.21006f3149 snapshot.21006a5af3 snapshot.21006c445a snapshot.21006ee9d6 currentEpoch log.21006a5af4 log.21006c445c log.21006ee9d8 log.21006f2d34 log.21006f317f snapshot.21006b70ba snapshot.21006d6e03 [dip@dip007 version-2]$ ll 总用量 342408 -rw-r--r-- 1 zookeeper zookeeper 2 4月 11 10:35 acceptedEpoch -rw-r--r-- 1 zookeeper zookeeper 2 5月 7 22:02 currentEpoch -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 6 15:29 log.210069890e -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 6 21:43 log.21006a5af4 -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 01:07 log.21006b70bc -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 02:41 log.21006c445c -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 15:48 log.21006d6e04 -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 21:23 log.21006ee9d8 -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 21:24 log.21006f2bbc -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 21:32 log.21006f2d34 -rw-r--r-- 1 zookeeper zookeeper 67108880 5月 7 22:01 log.21006f3149 -rw-r--r-- 1 zookeeper zookeeper 0 5月 7 22:02 log.21006f317f -rw-r--r-- 1 zookeeper zookeeper 33998220 5月 6 15:29 snapshot.21006a5af3 -rw-r--r-- 1 zookeeper zookeeper 35208598 5月 6 21:43 snapshot.21006b70ba -rw-r--r-- 1 zookeeper zookeeper 35673050 5月 7 01:07 snapshot.21006c445a -rw-r--r-- 1 zookeeper zookeeper 35637204 5月 7 02:41 snapshot.21006d6e03 -rw-r--r-- 1 zookeeper zookeeper 33990250 5月 7 15:48 snapshot.21006ee9d6
[root@dip007 version-2]# rm log.21006f317f