[20210917]ssh: error while loading shared libraries: libcrypto.so.1.0.0.txt

[20210917]ssh: error while loading shared libraries: libcrypto.so.1.0.0.txt

--//以后写一些特殊文章,一定记录在那台服务器执行的命令,特别涉及多台服务器的情况.
--//一台服务器(192.168.xx.yyy)使用普通用户无法使用ssh登录别的机器.提示如下:
--//在192.168.xx.yyy 上执行,以grid,oracle用户:
$  which ssh
/usr/bin/ssh

$  ls -l /usr/bin/ssh
-rwxr-xr-x 1 root root 736616 2020-07-01 16:53:23 /usr/bin/ssh

$  ssh 192.168.100.78
ssh: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
--//实际上暴露做等保一些运维人员不熟悉oracle rac,至少没有严格测试,如果以后升级或者打patch,两台机器无法通过ssh相互认证,问
--//题马上暴露,给运维埋一个很大的坑,到时候再来解决这个问题,会手忙脚乱的.

$  ldd $(which ssh)
        linux-vdso.so.1 =>  (0x00007fff22710000)
        libcrypto.so.1.0.0 => not found
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        librt.so.1 => /lib64/librt.so.1 (0x00007fc76035f000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fc76015b000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007fc75ff58000)
        libz.so.1 => /usr/local/lib/libz.so.1 (0x00007fc75fd40000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fc75fb08000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc75f8f3000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc75f59a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc75f37e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc76082c000)
--//注意看下划线,libcrypto.so.1.0.0无法找到.

--//在192.168.xx.yyy 上执行,以root用户:
#  which ssh
/usr/bin/ssh

#  ldd $(which ssh)
        linux-vdso.so.1 =>  (0x00007fff1f1af000)
        libcrypto.so.1.0.0 => /usr/local/openssl/lib/libcrypto.so.1.0.0 (0x00007f28497e9000)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        librt.so.1 => /lib64/librt.so.1 (0x00007f28495df000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f28493db000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f28491d8000)
        libz.so.1 => /usr/local/lib/libz.so.1 (0x00007f2848fc1000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f2848d88000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f2848b73000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f284881b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f28485fe000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2849f36000)

--//以grid用户执行:
$  ls -ld /usr/local/openssl/
drwxr-x--- 8 root root 4096 2020-07-01 16:49:14 /usr/local/openssl/
--//其他组没有任何权限,这样既不能读也无法进入对应目录.

$  cd /usr/local/openssl/
-bash: cd: /usr/local/openssl/: Permission denied

#  stat  /usr/local/openssl/lib/libcrypto.so.1.0.0
  File: `/usr/local/openssl/lib/libcrypto.so.1.0.0'
  Size: 3028344         Blocks: 5928       IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 200386      Links: 1
Access: (0750/-rwxr-x---)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-09-17 11:08:33.000000000 +0800
Modify: 2020-07-01 16:50:34.000000000 +0800
Change: 2021-09-17 11:08:26.000000000 +0800
--//其他组对/usr/local/openssl/lib/libcrypto.so.1.0.0文件也没有任何权限,自然普通用户是无法访问打开这个文件.
--//有几种解决方法,第一种建立软链接在/lib64目录下.

#  cd /lib64
#  ln -s /usr/local/openssl/lib/libcrypto.so.1.0.0
#  chmod 755 /usr/local/openssl/lib/libcrypto.so.1.0.0

--//第二种直接拷贝文件/usr/local/openssl/lib/libcrypto.so.1.0.0到/lib64目录:

#  cd /lib64
#  cp /usr/local/openssl/lib/libcrypto.so.1.0.0 .
#  chmod 755 libcrypto.so.1.0.0

--//我选择第2种,主要原因我不想改动/usr/local/openssl/lib/libcrypto.so.1.0.0文件权限.
--//测试通过,这样普通用户也可以使用ssh,我记忆里以前肯定没有问题的,不然oracle rac安装不可能完成.
--//我估计是等保做了某些处理,具体看看一些细节.

# cd /lib64
# mv libcrypto.so.1.0.0 libcrypto.so.1.0.0_xxx
$ ls -l /lib64/libcrypto*
-rwxr-xr-x 1 root root 1365136 2013-03-05 05:52:53 /lib64/libcrypto.so.0.9.8e
-rwxr-xr-x 1 root root 3028344 2021-09-17 11:02:34 /lib64/libcrypto.so.1.0.0_xxx
lrwxrwxrwx 1 root root      19 2014-05-16 23:11:39 /lib64/libcrypto.so.6 -> libcrypto.so.0.9.8e
--//我估计原来的版本是0.9.8e,估计等保做了升级,版本是1.0.0.

#  rpm -qif /lib64/libcrypto.so.0.9.8e
Name        : openssl                      Relocations: (not relocatable)
Version     : 0.9.8e                            Vendor: Oracle America
Release     : 26.el5_9.1                    Build Date: Tue 05 Mar 2013 05:52:53 AM CST
Install Date: Fri 16 May 2014 11:11:39 PM CST      Build Host: ca-build56.us.oracle.com
Group       : System Environment/Libraries   Source RPM: openssl-0.9.8e-26.el5_9.1.src.rpm
Size        : 3649954                          License: BSDish
Signature   : DSA/SHA1, Tue 05 Mar 2013 05:55:45 AM CST, Key ID 66ced3de1e5e0159
URL         : http://www.openssl.org/
Summary     : The OpenSSL toolkit
Description :
The OpenSSL toolkit provides support for secure communications between
machines. OpenSSL includes a certificate management tool and shared
libraries which provide various cryptographic algorithms and
protocols.

#  rpm -qilf /usr/local/openssl/lib/libcrypto.so
file /usr/local/openssl/lib/libcrypto.so is not owned by any package

#  rpm -qilf /usr/local/openssl/lib/libcrypto.so.1.0.0
file /usr/local/openssl/lib/libcrypto.so.1.0.0 is not owned by any package

#  ls -ld /usr/local/openssl
drwxr-x--- 8 root root 4096 2020-07-01 16:49:14 /usr/local/openssl
--//日期也暴露了等保安装升级留下的痕迹,该目录下的文件都是2020-07-01 16:4X.而且还可以看出升级不是采用rpm包的形式升级,我估
--//计使用tar的方式拷贝升级的.

#  stat  /usr/local/openssl
  File: `/usr/local/openssl'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fc00h/64512d    Inode: 184243      Links: 8
Access: (0750/drwxr-x---)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-09-17 11:14:15.000000000 +0800
Modify: 2020-07-01 16:49:14.000000000 +0800
Change: 2020-07-01 16:49:14.000000000 +0800

#  stat  /usr/local/openssl/lib/libcrypto.so.1.0.0
  File: `/usr/local/openssl/lib/libcrypto.so.1.0.0'
  Size: 3028344         Blocks: 5928       IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 200386      Links: 1
Access: (0750/-rwxr-x---)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-09-17 11:08:33.000000000 +0800
Modify: 2020-07-01 16:50:34.000000000 +0800
Change: 2021-09-17 11:08:26.000000000 +0800

#  ls -l $(which ssh)
-rwxr-xr-x 1 root root 736616 2020-07-01 16:53:23 /usr/bin/ssh
--//日期暴露了做了升级的动作,执行ssh文件实际上被覆盖了,从另外的侧面可以看出不是rpm包的形式升级,这样旧的rpm相关文件还在.

#  rpm -qif $(which ssh)
Name        : openssh-clients              Relocations: (not relocatable)
Version     : 4.3p2                             Vendor: Oracle America
Release     : 82.el5                        Build Date: Thu 23 Feb 2012 07:01:22 AM CST
Install Date: Fri 16 May 2014 11:25:12 PM CST      Build Host: ca-build10.us.oracle.com
Group       : Applications/Internet         Source RPM: openssh-4.3p2-82.el5.src.rpm
Size        : 865836                           License: BSD
Signature   : DSA/SHA1, Fri 24 Feb 2012 07:44:57 AM CST, Key ID 66ced3de1e5e0159
URL         : http://www.openssh.com/portable.html
Summary     : The OpenSSH client applications
Description :
OpenSSH is a free version of SSH (Secure SHell), a program for logging
into and executing commands on a remote machine. This package includes
the clients necessary to make encrypted connections to SSH servers.
You'll also need to install the openssh package on OpenSSH clients.

#  ssh -V
OpenSSH_7.9p1, OpenSSL 1.0.2r-fips  26 Feb 2019

--//对比我的测试环境的情况(192.168.100.78):
$ ldd $(which ssh)
        linux-vdso.so.1 =>  (0x00007fff648e9000)
        libfipscheck.so.1 => /usr/lib64/libfipscheck.so.1 (0x00007ffdb4f45000)
        libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00007ffdb4bf3000)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        libutil.so.1 => /lib64/libutil.so.1 (0x00007ffdb49f0000)
        libz.so.1 => /lib64/libz.so.1 (0x00007ffdb47dc000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00007ffdb45c3000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ffdb438b000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007ffdb4176000)
        libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007ffdb3f47000)
        libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007ffdb3cb2000)
        libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007ffdb3a8d000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007ffdb388a000)
        libnss3.so => /usr/lib64/libnss3.so (0x00007ffdb355c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ffdb3203000)
        libplc4.so => /usr/lib64/libplc4.so (0x00007ffdb2ffe000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007ffdb2dfa000)
        libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007ffdb2bf2000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007ffdb29ef000)
        libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007ffdb27ca000)
        libplds4.so => /usr/lib64/libplds4.so (0x00007ffdb25c7000)
        libnspr4.so => /usr/lib64/libnspr4.so (0x00007ffdb238b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffdb216f000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003798c00000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007ffdb1f57000)
        libsepol.so.1 => /lib64/libsepol.so.1 (0x00007ffdb1d10000)

$ ls -l /lib64/libcrypto.so.6
lrwxrwxrwx 1 root root 19 2014-08-29 21:28:41 /lib64/libcrypto.so.6 -> libcrypto.so.0.9.8e

--//指向libcrypto.so.0.9.8e,也证明对方做了一些升级.

$ ls -l /lib64/libcrypto.so.*
-rwxr-xr-x 1 root root 1367232 2012-05-30 01:55:15 /lib64/libcrypto.so.0.9.8e
lrwxrwxrwx 1 root root      19 2014-08-29 21:28:41 /lib64/libcrypto.so.6 -> libcrypto.so.0.9.8e
--//原来的版本是0.9.8e.

$ ls -l $(which ssh)
-rwxr-xr-x 1 root root 306064 2012-02-23 07:01:22 /usr/bin/ssh

$ rpm -qif $(which ssh)
Name        : openssh-clients              Relocations: (not relocatable)
Version     : 4.3p2                        Vendor: Oracle America
Release     : 82.el5                       Build Date: Thu 23 Feb 2012 07:01:22 AM CST
Install Date: Fri 29 Aug 2014 09:30:48 PM CST      Build Host: ca-build10.us.oracle.com
Group       : Applications/Internet         Source RPM: openssh-4.3p2-82.el5.src.rpm
Size        : 865836                           License: BSD
Signature   : DSA/SHA1, Fri 24 Feb 2012 07:44:57 AM CST, Key ID 66ced3de1e5e0159
URL         : http://www.openssh.com/portable.html
Summary     : The OpenSSH client applications
Description :
OpenSSH is a free version of SSH (Secure SHell), a program for logging
into and executing commands on a remote machine. This package includes
the clients necessary to make encrypted connections to SSH servers.
You'll also need to install the openssh package on OpenSSH clients.
--//与上面的一样,我估计不是使用rpm包安装的,而是拷贝或者tar包安装的.

$ ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
--//对比前面版本完全不一致.

--//一旦生产系统上线,我个人很少在服务器安装升级软件包,除非存在安全漏洞,这台服务器我升级bash,即使升级我也选择rpm包模式,
--//也不会选择生产服务器安装编译软件,总而言之,做运维工作一定要小心再小心..