OEM报错"Failed to connect to ASM instance. The connection is closed: The connection is closed"处理
前言
秉着出现的报错就追根问底的原则，这次刚部署不久的OEM 13C有出现如下报警：
Host=xxxxx1 Target type=Automatic Storage Management Target name=+ASM1_xxxxx1 Categories=Availability Message=Failed to connect to ASM instance. The connection is closed: The connection is closed Severity=Fatal Event reported time=Aug 9, 2020 10:08:18 AM CST Operating System=Linux Platform=x86_64 Associated Incident Id=88 Associated Incident Status=New Associated Incident Owner= Associated Incident Acknowledged By Owner=No Associated Incident Priority=None Associated Incident Escalation Level=0 Event Type=Target Availability Event name=Status Availability status=Down Root Cause Analysis Status=Neither Cause Nor Symptom Causal analysis result=Neither a cause nor a symptom Rule Name=Incident management rule set for all targets,Incident creation rule for a Target Down availability status Rule Owner=System Generated Update Details: Failed to connect to ASM instance. The connection is closed: The connection is closed Incident created by rule (Name = Incident management rule set for all targets, Incident creation rule for a Target Down availability status [System generated rule]).
照例问度娘是没问出啥来......
MOS上搜的话就有结果了：
EM 13c: Enterprise Manager 13.2 Cloud Control ASM Incident Reported with Message=Failed To Connect To ASM Instance. The Connection Is Closed: The Connection Is Closed (Doc ID 2251591.1)
文档中提到，这个一个BUG。
验证
文档中提到，在gcagent.log日志会有如下报错（示例）：
| [65336:GC.Executor.126 (osm_instance:+ASM__host.company.com:ofs_performance_metrics) (osm_instance:+ASM__host.company.com:ofs_performance_metrics:Instance_Volume_Performance)] ERROR - The connection is closed: The connection is closed java.sql.SQLException: The connection is closed: The connection is closed at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:464) at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:448) at oracle.ucp.jdbc.proxy.JDBCConnectionProxyFactory.invoke(JDBCConnectionProxyFactory.java:307) at oracle.ucp.jdbc.proxy.ConnectionProxyFactory.invoke(ConnectionProxyFactory.java:50) at com.sun.proxy.$Proxy27.prepareCall(Unknown Source) |
该日志位于客户端如下位置：
[oracle@xxxxx1 log]$ ll $AGENT_HOME/sysman/log/gcagent.log -rw-r----- 1 oracle oinstall 960998 Aug 10 14:20 /u01/app/oem13c/agent/agent_inst/sysman/log/gcagent.log
查看日志可以发现，确实存在相似的日志信息：
2020-08-09 10:08:18,645 [99899:GC.Executor.23807 (osm_instance:+ASM1_xxxxx1:Response) (osm_instance:+ASM1_xxxxx1:Response:Response)] ERROR - The connection is closed: The connection is closed java.sql.SQLException: The connection is closed: The connection is closed at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:464) at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:448) at oracle.ucp.jdbc.proxy.JDBCConnectionProxyFactory.invoke(JDBCConnectionProxyFactory.java:307) at oracle.ucp.jdbc.proxy.ConnectionProxyFactory.invoke(ConnectionProxyFactory.java:50) at com.sun.proxy.$Proxy31.prepareCall(Unknown Source)
文档中还提到，
此外，如果在EM代理进程上进行了线程转储，则会观察到大量的"Timer-"线程（它们随着时间的推移而增加，并且从未关闭/结束）。例：
jstack <Agent PID>|grep "Timer-"|wc -l
983
注意：根据经验，"Timer-"线程的数量应随时间保持恒定，少于50，但这是一个近似值，因为它取决于目标数量，监视设置，执行的作业以及许多其他因素。关键因素是随着时间（天）的增加，此类线程的数量将保持恒定。
问题节点再次验证如下：
[oracle@xxxxx1 ~]# ps -ef | grep java ...省略部分内容... oracle 7687 7601 3 Aug05 ? 03:34:38 /u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/jdk/bin/java -Xmx128M -XX:MaxPermSize=128M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:-UseLargePages -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=7601 -cp /u01/app/oem13c/agent/agent_13.3.0.0.0/jdbc/lib/ojdbc7.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/ucp/lib/ucp.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/jsch-0.1.53.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/com.oracle.http_client.http_client_12.1.3.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.xdk_12.1.3/xmlparserv2.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.dms_12.1.3/dms.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.odl_12.1.3/ojdl.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.odl_12.1.3/ojdl2.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/lib/optic.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/sysman/jlib/log4j-core.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/jlib/gcagent_core.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/sysman/jlib/emagentSDK-intg.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain [oracle@xxxxx1 ~]$ /u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/jdk/bin/jstack 7687 | grep "Timer-" | wc -l 83
另外一个没报警的节点情况：
[oracle@xxxxx2 ~]$ ps -ef | grep 13.3 oracle 31845 31753 0 Aug05 ? 00:26:26 /u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/jdk/bin/java -Xmx128M -XX:MaxPermSize=128M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:-UseLargePages -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=31753 -cp /u01/app/oem13c/agent/agent_13.3.0.0.0/jdbc/lib/ojdbc7.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/ucp/lib/ucp.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/jsch-0.1.53.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/com.oracle.http_client.http_client_12.1.3.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.xdk_12.1.3/xmlparserv2.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.dms_12.1.3/dms.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.odl_12.1.3/ojdl.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/modules/oracle.odl_12.1.3/ojdl2.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/lib/optic.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/sysman/jlib/log4j-core.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/jlib/gcagent_core.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/sysman/jlib/emagentSDK-intg.jar:/u01/app/oem13c/agent/agent_13.3.0.0.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain [oracle@xxxxx2 ~]$ /u01/app/oem13c/agent/agent_13.3.0.0.0/oracle_common/jdk/bin/jstack 31845 | grep "Timer-" | wc -l 7
文档中给出经验值为<50，我在问题节点可以看出"Timer-"线程的数量为83，作为一个参考值，说明该节点很有可能出现了BUG。
处理
这是BUG导致的，13.2/13.3/13.4均存在此问题，不过对应BUG号不同，因此补丁也不同。
13.3的对应BUG为Bug 28406747，对应在agent段打上该补丁即可。
如何打补丁
首先是一个打补丁的目标的问题，之前给OMS打psu的时候虽然是第一次但是有给DB打PSU的经验倒是稍微折腾了下。
这次确是一个小补丁，根据readme提到的，其中一步是需要关闭Management Agent，这个地方纠结了好一会。
这个Management Agent指的是哪个？
正常来讲，出现问题的节点在于数据库服务器上的agent端，所以应该是打在数据库服务器上的agent上，但是，
这个management的单词让我觉得是oms上的agent端，并且如果是数据库服务器上的agent上那岂不是有很多台的agent都要关掉打上？
而且说实话，oms上的agent是否和数据库服务器上的agent是一样的我都不确定（后来确定是一样的）。
又是一阵度娘和mos，这次就找不出来啥了。
后来又想到，其实在oms刚刚搭建完成后，默认在网页管理的目标“主机”就有了oms服务器本身，那其实无论是oms的agent还是db服务器上的agent，
本质上应该是一个东西，于是尝试在oms上将agent停掉，
$AGENT_HOME/bin/emctl stop agent
果然，oem的网页还是可以登陆的，目标“主机”处oms本身的机器已经处于不健康的状态，看来确实是一样的。
也就是，全部的agent都需要一个一个打上补丁......
后边有想到一个问题，是否在oms上的agent打上补丁后，之后就算新推送到其他服务器上的agent估计就是带上了新打的补丁了呢？
话不多说，先给oms的agent打上补丁，在推一个新的agent到未监控的db服务器上看看情况就知道了。
首先，一定要先读补丁的readme，按照里边的要求一步一步来！！！
第一，需要给agent的OPatch版本升级，由于oms的agent之前打psu的时候已经升级过了，因此这一步不再需要做。
第二，设置环境变量，
[oracle@oem13c agent]$ export ORACLE_HOME=/u01/app/oem13c/agent/agent_13.3.0.0.0 [oracle@oem13c agent]$ /u01/app/oem13c/agent/agent_13.3.0.0.0/OPatch/opatch version OPatch Version: 13.9.3.3.0 OPatch succeeded.
这里扯点其他的，readme管这个目录/u01/app/oem13c/agent/agent_13.3.0.0.0叫agent core home，实际上，
环境变量AGENT_HOME设置的值为/u01/app/oem13c/agent/agent_inst，这个值在推送客户端的时候叫instance directory，
其中，/u01/app/oem13c/agent为agent的base目录，设置为AGENT_HOME=/u01/app/oem13c/agent/agent_inst原因是emctl命令在这个目录下的bin文件夹中。
实际上打小补丁的应用目录是agent core home。
继续回到打补丁这里，
第三，关闭agent，
[oracle@oem13c 28406747]$ export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH [oracle@oem13c 28406747]$ emctl stop agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Stopping agent ... stopped. [oracle@oem13c 28406747]$ opatch lspatches 25237184;One-off 24470104; OPatch succeeded.
第四，直接应用补丁即可，
[oracle@oem13c 28406747]$ opatch apply Oracle Interim Patch Installer version 13.9.3.3.0 Copyright (c) 2020, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oem13c/agent/agent_13.3.0.0.0 Central Inventory : /u01/app/oraInventory from : /u01/app/oem13c/agent/agent_13.3.0.0.0/oraInst.loc OPatch version : 13.9.3.3.0 OUI version : 13.9.1.0.0 Log file location : /u01/app/oem13c/agent/agent_13.3.0.0.0/cfgtoollogs/opatch/opatch2020-08-10_16-39-09PM_1.log OPatch detects the Middleware Home as "/u01/app/oem13c/agent" Verifying environment and performing prerequisite checks... OPatch continues with these patches: 28406747 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Backing up files... Applying interim patch '28406747' to OH '/u01/app/oem13c/agent/agent_13.3.0.0.0' Patching component oracle.sysman.agent.ic, 13.3.0.0.0... Patch 28406747 successfully applied. Log file location: /u01/app/oem13c/agent/agent_13.3.0.0.0/cfgtoollogs/opatch/opatch2020-08-10_16-39-09PM_1.log OPatch succeeded. [oracle@oem13c 28406747]$ opatch lspatches 28406747; 25237184;One-off 24470104; OPatch succeeded.
最后，开启agent，
[oracle@oem13c 28406747]$ emctl start agent
至此，小补丁成功打上。
后边推送新的agent到未监控的db服务器上，发现推送后，db上的agent是没有新的补丁的...
所以还是要手动全部打一遍。
一样的步骤，不是特别复杂。
后续再观察"Timer-"线程的数量是否会再次异常以及是否还有报警产生。