巨锅头顶罩，领导对我叫，开发说程序没问题，网络说数据库有问题
为什么我要做DBA，爱背锅爱口吃，我要找到到底谁的锅。。。
当你每次遇到问题有木有诚惶诚恐？反正我表面虽冷静但内心还是有点慌的。当公司只有一个DBA时必须要有一颗强大的心和善于沟通的嘴，否则早晚有背不完的锅
现在听起来像讲故事，今天下大雨心情本来就不好，家里窗户没关。当他们都把问题甩过来时候我只能证明数据库日志是正常的，由于甩锅的技能还没练到炉火纯青的地步，但看到一遇问题就肯定自己没问题使劲儿甩锅的人，有的还特么是领导，真是奇迹
【问题】数据库相关的网络防火墙设备迁移后两个重要系统时不时访问异常，登陆无响应，偶尔一些别的功能也点击无响应，还会自动断开，不完全统计一天大概出现个五六七八十来次。。。
【大概思路】
1.先检查DB日志，包括监听日志
2.如果日志没发现明显问题，再用SQL查下session及oracle进程，然后查下OS资源比如CPU、内存、存储
3.如果上面还没定位出明显问题，就要看网络防火墙了
当然如果监控做到位了就没这么复杂了，之前的Zabbix监控系统和ELK日志系统被人倒腾的一个相当于摆设一个不能用了
因为上述问题只是偶尔出现，分析了1、2分析不动了，alter日志确实有TNS-12535、TNS-00505的异常，不过DB层面最近没什么大的变更，所以就硬着头皮让网络清理了下防火墙，clear session all;原来部署代码一直失败的现在也ok了，此时网络BOSS蹦出来一句“这次算你厉害。。。”，我没有窃喜，就说三个字“我蒙的”，不过上述问题还有待观察，但我相信不会再出现了。虽然我不是很懂网络，但TCP/IP还多少懂点。
我当时想要一起分析解决问题，不能劈头盖脸发现一点跟数据库有关的异常就是数据库有问题，也不是说谁的声音大谁的人多谁的级别高谁就没问题，而不是互相指责推诿。其实我要甩锅也很简单：你怎么证明你那没问题？你怎么证明是DB的问题？
【结论】网络设备迁移后，有部分session未正常清理掉驻留在防火墙里面，而且这部分session可能是长连接，所以clear之后让其再重建就不会出现APP再次访问时访问到的是断头的session了
com.ibatis.common.jdbc.exception.NestedSQLException:
--- The error occurred in sqlmap/system/admin/SysUser.xml.
--- The error occurred while executing query.
--- Check the select col1,col2.... from 表0 where id=? .
--- Check the SQL Statement (preparation failed).
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 60, runningSqlCount 3 : SELECT 'x' from dual
runningSqlCount 1 : select col1,col2.... from 表1 where id=?
runningSqlCount 1 : select col1,col2.... from 表2 where 1=1 order by create_date desc,id desc
runningSqlCount 1 : select col1,col2.... from 表3
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1137)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:960)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4544)
at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:661)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)
at com.alibaba.druid.filter.logging.LogFilter.dataSource_getConnection(LogFilter.java:827)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:938)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:930)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:102)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:224)
at com.sun.proxy.$Proxy344.prepareStatement(Unknown Source)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.prepareStatement(SqlExecutor.java:497)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:175)
at com.nbtv.orm.dao.ibatis.executor.LimitSqlExecutor.executeQuery(LimitSqlExecutor.java:57)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
... 98 more
Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:148)
at com.alibaba.druid.filter.stat.StatFilter.connection_connect(StatFilter.java:211)
at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:142)
at com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:785)
at com.alibaba.druid.filter.FilterEventAdapter.connection_connect(FilterEventAdapter.java:38)
at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:142)
at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1377)
at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1431)
at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:1861)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection