Hadoop搭建HA遇到的坑
重要报错信息
Zookeeper日志
- Cannot open channel to 2 at election address node03/172.17.0.4:3888
java.net.ConnectException: Connection refused (Connection refused)
Hadoop日志
- 2021-01-30 19:44:04,310 ERROR org.apache.hadoop.ha.NodeFencer: Unable to fence service by any configured method.
- 2021-01-30 19:44:04,310 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
java.lang.RuntimeException: Unable to fence NameNode at node01/172.17.0.2:8020 - java.io.EOFException: End of File Exception between local host is: "node02/172.17.0.3"; destination host is: "node02":8020; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException
- Caused by: java.io.EOFException
- 2021-01-30 19:44:06,535 WARN org.apache.hadoop.ha.FailoverController: Unable to gracefully make NameNode at node01/172.17.0.2:8020 standby (unable to connect)
java.net.ConnectException: Call From node02/172.17.0.3 to node01:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused - 2021-01-30 19:44:06,535 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to create SSH session
com.jcraft.jsch.JSchException: invalid privatekey: [B@1bfa0880
结构如下
| HOST | NN | NN | JNN | DN | ZKFC | ZK |
|---|---|---|---|---|---|---|
| node01 | * | * | * | |||
| node02 | * | * | * | * | * | |
| node03 | * | * | * | * | ||
| node04 | * | * |
问题
- 所有的节点角色均可正常运行，当我将node01中的Active状态的NameNode停止，node02中的NameNode并不会由StandBy状态转变为Active，而重新将node01中的NameNode启动后，node02中的NameNode转变为Active状态。
解决过程
- 尝试过将各个节点Zookeeper中的zoo.cfg各自的地址改为0.0.0.0，失败
- 时间已经设置同步
- 各个角色运行正常
- apt安装psmisc，失败
- 网络上说ssh-keygen版本过高高的时候生成秘钥会有问题，于是使用ssh-keygen -t rsa -m PEM，失败
- 最后发现所有的NameNode应当都对其余节点免密，而我只设置了node01，所以将node02的公钥发送给其余的节点后，故障得以解决
原因
- 如上结构图所示，当node02未配置与2，3，4免密时，相当于运行在node02的zkfc不能与Zookeeper集群通信