当DataGuard升级CPU后。发现主备库不能同步了，日志报错，所以在主库的运行的升级的后脚本“@catbundle.sql cpu apply”不能同步到备库
现象：
broker状态：
[oracle@webdb2 ~]$ dgmgrl sys/oracle
DGMGRL for Linux: Version 11.2.0.3.0 -64bit Production
Copyright (c) 2000, 2009, Oracle. Allrights reserved.
欢迎使用 DGMGRL, 要获取有关信息请键入 "help"。
已连接。
DGMGRL> show configuration
配置 - webdb
保护模式: MaxAvailability
数据库:
webdb - 主数据库
webdb2 - 物理备用数据库
高速启动故障转移: DISABLED
配置状态:
ORA-16664: 无法从数据库接收结果
DGM-17017: 无法确定配置状态
--查看主库状态：
SQL> Select timestamp, message FROMV$DATAGUARD_STATUS Where Rownum < 5;
TIMESTAMP MESSAGE
-------------------------------------------------------------------------------------------
2014-10-23 Error 604 received logging on to the standby
2014-10-23 PING[ARC2]: Heartbeat failed to connect tostandby'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=webdb2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=webdb2)(SERVER=DEDICATED)))'.Error is 604.
2014-10-23 Error 604 received logging on to the standby
2014-10-23 PING[ARC2]: Heartbeat failed to connect to standby'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=webdb2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=webdb2)(SERVER=DEDICATED)))'.Error is 604.
--此时查看日志
主库日志：
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 29
Thu Oct 23 00:25:52 2014
Errors in file/u01/app/ora11g/diag/rdbms/webdb/webdb/trace/webdb_ora_18833.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 29
Thu Oct 23 00:25:52 2014
Errors in file/u01/app/ora11g/diag/rdbms/webdb/webdb/trace/webdb_ora_18842.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 29
Thu Oct 23 00:25:53 2014
Error 604 receivedlogging on to the standby
PING[ARC2]: Heartbeatfailed to connect to standby'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=webdb2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=webdb2)(SERVER=DEDICATED)))'.Error is 604.
Thu Oct 23 00:26:12 2014
Errors in file /u01/app/ora11g/diag/rdbms/webdb/webdb/trace/webdb_ora_18961.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 29
备库日志：
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-16000: 打开数据库以进行仅仅读訪问
ORA-06512: 在 line 3
Thu Oct 23 00:27:39 2014
Errors in file /u01/app/ora11g/diag/rdbms/webdb2/webdb2/trace/webdb2_ora_30130.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 29
Errors in file/u01/app/ora11g/diag/rdbms/webdb2/webdb2/trace/webdb2_ora_30130.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-16000: 打开数据库以进行仅仅读訪问
ORA-06512: 在 line 3
--进一步查看trace文件
主库trace：
[oracle@webdb1 trace]$ more/u01/app/ora11g/diag/rdbms/webdb/webdb/trace/webdb_ora_8589.trc
Trace file/u01/app/ora11g/diag/rdbms/webdb/webdb/trace/webdb_ora_8589.trc
Oracle Database 11g Enterprise EditionRelease 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic StorageManagement, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME =/u01/app/ora11g/product/11.2.0/db_1
System name: Linux
Node name: webdb1
Release: 2.6.32-200.13.1.el5uek
Version: #1 SMP Wed Jul 27 21:02:33 EDT 2011
Machine: x86_64
Instance name: webdb
Redo thread mounted by this instance: 1
Oracle process number: 176
Unix process pid: 8589, image:oracle@webdb1
*** 2014-10-14 13:19:18.860
*** SESSION ID:(21.12621) 2014-10-1413:19:18.860
*** CLIENT ID:() 2014-10-14 13:19:18.860
*** SERVICE NAME:(SYS$USERS) 2014-10-1413:19:18.860
*** MODULE NAME:(emagent@webdb1 (TNSV1-V3)) 2014-10-14 13:19:18.860
*** ACTION NAME:() 2014-10-14 13:19:18.860
Error in executing triggers on connectinternal
*** 2014-10-14 13:19:18.861
dbkedDefDump(): Starting a non-incidentdiagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 28
Trace file/u01/app/ora11g/diag/rdbms/webdb/webdb/trace/webdb_ora_8589.trc
Oracle Database 11g Enterprise EditionRelease 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic StorageManagement, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME =/u01/app/ora11g/product/11.2.0/db_1
System name: Linux
Node name: webdb1
Release: 2.6.32-200.13.1.el5uek
Version: #1 SMP Wed Jul 27 21:02:33 EDT 2011
Machine: x86_64
Instance name: webdb
Redo thread mounted by this instance: 1
Oracle process number: 163
Unix process pid: 8589, image:oracle@webdb1
备库trace:
[oracle@webdb2 trace]$ more/u01/app/ora11g/diag/rdbms/webdb2/webdb2/trace/webdb2_ora_29791.trc
Trace file/u01/app/ora11g/diag/rdbms/webdb2/webdb2/trace/webdb2_ora_29791.trc
Oracle Database 11g Enterprise EditionRelease 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic StorageManagement, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME =/u01/app/ora11g/product/11.2.0/db_1
System name: Linux
Node name: webdb2
Release: 2.6.32-200.13.1.el5uek
Version: #1 SMP Wed Jul 27 21:02:33 EDT 2011
Machine: x86_64
Instance name: webdb2
Redo thread mounted by this instance: 1
Oracle process number: 22
Unix process pid: 29791, image:oracle@webdb2
*** 2014-10-23 11:30:21.649
*** SESSION ID:(854.10603) 2014-10-2311:30:21.649
*** CLIENT ID:() 2014-10-23 11:30:21.649
*** SERVICE NAME:(SYS$USERS) 2014-10-2311:30:21.649
*** MODULE NAME:(emagent@webdb2 (TNSV1-V3)) 2014-10-23 11:30:21.649
*** ACTION NAME:() 2014-10-23 11:30:21.649
Error in executing triggers on connectinternal
*** 2014-10-23 11:30:21.649
dbkedDefDump(): Starting a non-incidentdiagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
ORA-06512: 在 line 29
Error in executing triggers on connectinternal
*** 2014-10-23 11:30:21.650
dbkedDefDump(): Starting a non-incidentdiagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-16000: 打开数据库以进行仅仅读訪问
ORA-06512: 在 line 3
分析：
能够从看到trace log中看到都与tiger运行有关。于是想到在主库是创建过的两个 logon on database tiger
解决的方法：
--禁用登录tiger
把一个登录IP限制的tiger的禁用后，主库的
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20154: TNS: 监听程序当前无法识别连接描写叙述符中请求的服务
报错消失。仅仅剩下以下的报错：
Error 604 received logging on to thestandby
PING[ARC2]: Heartbeat failed to connect tostandby'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=webdb2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=webdb2)(SERVER=DEDICATED)))'.Error is 604.
备库依旧报错
--启动备库到mount状态
SQL> shutdown immediate
SQL> startup mount
ORACLE 例程已经启动。
Total System Global Area 1.0456E+10 bytes
Fixed Size 2237488 bytes
Variable Size 5435821008 bytes
Database Buffers 4999610368 bytes
Redo Buffers 17973248 bytes
数据库装载完成。
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED
--查看日志
主库：
Thu Oct 23 13:45:15 2014
Archived Log entry 2225 added for thread 1sequence 1122 ID 0xc8d579d3 dest 1:
ARC3: Standby redo logfile selected forthread 1 sequence 1122 for destination LOG_ARCHIVE_DEST_2
Thu Oct 23 13:46:46 2014
ARC3: Archive log rejected (thread 1sequence 1120) at host'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=webdb2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=webdb2)(SERVER=DEDICATED)))'
Thu Oct 23 13:46:46 2014
ARC0: Archive log rejected (thread 1sequence 1121) at host'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=webdb2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=webdb2)(SERVER=DEDICATED)))'
FAL[server, ARC3]: FAL archive failed, seetrace file.
ARCH: FAL archive failed. Archivercontinuing
ORACLE Instance webdb - Archival Error.Archiver continuing.
FAL[server, ARC0]: FAL archive failed, seetrace file.
ARCH: FAL archive failed. Archivercontinuing
ORACLE Instance webdb - Archival Error.Archiver continuing.
Thu Oct 23 13:46:49 2014
Destination LOG_ARCHIVE_DEST_2 isSYNCHRONIZED
LGWR: Standby redo logfile selected toarchive thread 1 sequence 1124
LGWR: Standby redo logfile selected forthread 1 sequence 1124 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 1124(LGWR switch)
Current log# 2 seq# 1124 mem# 0:+DATA/webdb/onlinelog/group_2.262.819826783
Current log# 2 seq# 1124 mem# 1:+FRA/webdb/onlinelog/group_2.258.819826785
Archived Log entry 2244 added for thread 1sequence 1123 ID 0xc8d579d3 dest 1
备库：
Using STANDBY_ARCHIVE_DEST parameterdefault value as USE_DB_RECOVERY_FILE_DEST
Thu Oct 23 13:45:15 2014
RFS[1]: Assigned to RFS process 17293
RFS[1]: Opened log for thread 1 sequence1102 dbid -925543465 branch 819826777
Thu Oct 23 13:45:15 2014
Primary database is in MAXIMUM AVAILABILITYmode
Standby controlfile consistent with primary
Standby controlfile consistent with primary
RFS[2]: Assigned to RFS process 17297
RFS[2]: Selected log 7 for thread 1sequence 1123 dbid -925543465 branch 819826777
Deleted Oracle managed file+FRA/webdb2/archivelog/2014_10_13/thread_1_seq_1049.350.860808821
Archived Log entry 1104 added for thread 1sequence 1102 rlc 819826777 ID 0xc8d579d3 dest 2:
能够看到主备库恢复心跳。备库能恢复应用日志
打开备库操作：
SQL> alter database open;
数据库已更改。
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY
SQL> ALTER DATABASE RECOVER MANAGEDSTANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
ALTER DATABASE RECOVER MANAGED STANDBYDATABASE USING CURRENT LOGFILE DISCONNECT
*
第 1 行出现错误:
ORA-01153: 激活了不兼容的介质恢复
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY WITH APPLY
--主库切换日志
SQL> alter system switch logfile;
系统已更改。
--查看日志
主库：
Thu Oct 23 14:19:52 2014
LGWR: Standby redo logfile selected toarchive thread 1 sequence 1125
LGWR: Standby redo logfile selected forthread 1 sequence 1125 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 1125(LGWR switch)
Current log# 3 seq# 1125 mem# 0:+DATA/webdb/onlinelog/group_3.263.819826787
Current log# 3 seq# 1125 mem# 1:+FRA/webdb/onlinelog/group_3.259.819826789
Thu Oct 23 14:19:53 2014
Archived Log entry 2246 added for thread 1sequence 1124 ID 0xc8d579d3 dest 1:
备库：
Completed: ALTER DATABASE RECOVER MANAGEDSTANDBY DATABASE THROUGH ALL SWITCHOVERDISCONNECT USING CURRENT LOGFILE
Thu Oct 23 14:15:51 2014
ALTER DATABASE RECOVER MANAGED STANDBYDATABASE USING CURRENT LOGFILE DISCONNECT
ORA-1153 signalled during: ALTER DATABASERECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT...
Thu Oct 23 14:19:52 2014
Standby controlfile consistent with primary
RFS[2]: Selected log 7 for thread 1sequence 1125 dbid -925543465 branch 819826777
Thu Oct 23 14:19:52 2014
Archived Log entry 1123 added for thread 1sequence 1124 ID 0xc8d579d3 dest 1:
Thu Oct 23 14:19:52 2014
Media Recovery Waiting for thread 1sequence 1125 (in transit)
Recovery of Online Redo Log: Thread 1 Group7 Seq 1125 Reading mem 0
Mem# 0: +DATA/webdb2/onlinelog/group_7.282.819884421
Mem# 1: +FRA/webdb2/onlinelog/group_7.278.819884423
能够看到在仅仅读应用状态open备库后。日志应用仍然正常
--查看DataGuard状态
[oracle@webdb1 trace]$ dgmgrl sys/oracle
DGMGRL for Linux: Version 11.2.0.3.0 -64bit Production
Copyright (c) 2000, 2009, Oracle. Allrights reserved.
欢迎使用 DGMGRL, 要获取有关信息请键入 "help"。
已连接。
DGMGRL> show configuration
配置 - webdb
保护模式: MaxAvailability
数据库:
webdb - 主数据库
webdb2 - 物理备用数据库
高速启动故障转移: DISABLED
配置状态:
SUCCESS
总结：
通过报错分析看到，主备不同步跟CPU补丁升级没有直接关系，跟创建的logon on database 触发器有关。
假设在主数据创建一个logon on database tiger。可能会引起主备库不能同步传输日志的问题，
这时。我们要做是把备库又一次mount，然后再open到仅仅读应用状态（READ ONLY WITH APPLY）
由于当备库重新启动后，备库直接启动到仅仅读应用状态（READ ONLY WITH APPLY）。就会出现上面的问题。
參考：
http://www.cnblogs.com/Spiritual-journey/archive/2012/10/21/2732806.html