记ssh错误排查-ansible

背景

ansible 节点 到三个k8s节点，执行ansible UCA -m ping -o 只有 一个成功 ， 两个失败。
ansible到目标主机使用秘钥登录。
节点
ansible
k8s1
k8s2
k8s3

排查过程描述:

ansible默认通过ssh去连接目标主机，并执行制定命令脚本。
此次遇到问题在执行ansible UCA -m ping -o 时输出如下：

[root@ansible ~]# ansible UCA -m ping -o
[WARNING]: Unable to parse /root/hosts as an inventory source
Wednesday 10 June 2020  16:16:48 +0800 (0:00:00.062)       0:00:00.062 ******** 
[WARNING]: Unhandled error in Python interpreter discovery for host HD1-SHMY1-UCA-K8S-Node2: Failed to connect to the host via ssh: Warning: Permanently
added '100.66.0.2' (ECDSA) to the list of known hosts.  Permission denied (publickey).
[WARNING]: Unhandled error in Python interpreter discovery for host HD1-SHMY1-UCA-K8S-Node3: Failed to connect to the host via ssh: Warning: Permanently
added '100.66.0.3' (ECDSA) to the list of known hosts.  Permission denied (publickey).
HD1-SHMY1-UCA-K8S-Node2 | UNREACHABLE!: Data could not be sent to remote host "100.66.0.2". Make sure this host can be reached over ssh: Permission denied (publickey).
HD1-SHMY1-UCA-K8S-Node3 | UNREACHABLE!: Data could not be sent to remote host "100.66.0.3". Make sure this host can be reached over ssh: Permission denied (publickey).
HD1-SHMY1-UCA-K8S-Node1 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "ping": "pong"}
Wednesday 10 June 2020  16:16:49 +0800 (0:00:00.924)       0:00:00.986 ******** 
=============================================================================== 
ping ---------------------------------------------------------------------------------------------------------------------------------------------- 0.92s
Playbook run took 0 days, 0 hours, 0 minutes, 0 seconds

从输出里看是权限不允许, 于是尝试ssh连接

[root@ansible ~]# ssh 100.66.0.2 -vvv
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "100.66.0.2" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 100.66.0.2 [100.66.0.2] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
...中间输出省略
debug1: SSH2_MSG_SERVICE_ACCEPT received
__debug3: send packet: type 50__
__debug3: receive packet: type 51__
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
__debug3: send packet: type 50__
debug2: we sent a publickey packet, wait for reply
__debug3: receive packet: type 51__
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

看上边输出, 由于我使用了key登录， 故关注此处

debug1: Offering RSA public key: /root/.ssh/id_rsa
__debug3: send packet: type 50__
debug2: we sent a publickey packet, wait for reply
__debug3: receive packet: type 51__

debug1:提供了我的key文件路径之后， send了一个type 50的包，等待reply时收到了一个type 51的包, 谷歌一通后发现：

http://www.snailbook.com/docs/assigned-numbers.txt
4.1.2.  Initial Assignments
         Message ID                            Value    Reference
         -----------                           -----    ---------
         SSH_MSG_USERAUTH_REQUEST                50     [SSH-USERAUTH]
         SSH_MSG_USERAUTH_FAILURE                51     [SSH-USERAUTH]

http://www.snailbook.com/docs/userauth.txt
6.  Authentication Protocol Message Numbers
   These are the general authentication message codes:

      SSH_MSG_USERAUTH_REQUEST            50
      SSH_MSG_USERAUTH_FAILURE            51
      SSH_MSG_USERAUTH_SUCCESS            52
      SSH_MSG_USERAUTH_BANNER             53

7.  Public Key Authentication Method: "publickey"

   The following method-specific message numbers are used by the
   "publickey" authentication method.

      SSH_MSG_USERAUTH_PK_OK              60

也就是数发送了个ssh用户认证请求之后，返回了个用户验证错误，正确的返回应该是type 60的包。
于是我又ssh了k8s1（可以正常ssh的），输出如下：

debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60

果然正常的验证正确返回值是type 60的包。
常规的配置检查了一遍没有异常， 此时ls -l /root/.ssh ， 发现三个k8s节点的文件有些异常
k8s1

[root@k8s-node1 .ssh]# ll
total 12
-rw------- 1 root root 2569 Jun 10 16:37 authorized_keys
-rw------- 1 root root 1679 Apr 17 09:34 id_rsa
-rw-r--r-- 1 root root  410 Apr 17 09:34 id_rsa.pub

k8s2

[root@k8s-node2 .ssh]# ll
total 8
-rw------- 1 root root 2571 Jun 10 16:37 authorized_keys
-rw------- 1 root root 1679 Jun 10 16:05 id_rsa

k8s3

[root@k8s-node3 .ssh]# ll
total 8
-rw------- 1 root root 2571 Jun 10 16:37 authorized_keys
-rw------- 1 root root 1679 Jun 10 16:05 id_rsa

k8s1的authorized_keys大小和2、3的不一致，但是cat了一下内容完全一样。
于是将k8s1上的authorized_keys scp到2和3上，于是 都可以通过秘钥访问了。

结论

  ssh发送用户认证请求之后，验证key失败。 至于authorized_keys的大小为什么不一致，由于不能重现，就不得而知了。