一套AIX 上的9.2.0.6 2节点RAC系统出现了ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []内部错误伴随有ROW CACHE ENQUEUE LOCK并引发clusterware split-brain resolution，详细的日志及ass.awk输出如下：
ALERT LOG
=============
Sun Jun 19 09:06:24 2011
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=24
Sun Jun 19 09:06:29 2011
Errors in file /s01/admin/prod/udump/prod2_ora_1061088.trc:
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Sun Jun 19 09:06:29 2011
Errors in file /s01/admin/prod/udump/prod2_ora_1061088.trc:
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Sun Jun 19 09:06:30 2011
Errors in file /s01/admin/prod/udump/prod2_ora_1061088.trc:
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Sun Jun 19 09:06:30 2011
Errors in file /s01/admin/prod/udump/prod2_ora_1061088.trc:
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Sun Jun 19 09:06:31 2011
Errors in file /s01/admin/prod/udump/prod2_ora_1061088.trc:
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Sun Jun 19 09:06:31 2011
Errors in file /s01/admin/prod/udump/prod2_ora_1061088.trc:
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Sun Jun 19 09:08:06 2011
Waiting for clusterware split-brain resolution
Sun Jun 19 09:13:17 2011
ALTER SYSTEM SET event='10511 trace name context forever, level 1' SCOPE=SPFILE SID='*';
Sun Jun 19 09:14:44 2011
Trace dumping is performing id=[cdmp_20110619091444]
Sun Jun 19 09:18:05 2011
Errors in file /s01/admin/prod/bdump/prod2_lmon_422072.trc:
ORA-29740: evicted by member 1, group incarnation 9
Sun Jun 19 09:18:05 2011
LMON: terminating instance due to error 29740
Sun Jun 19 09:18:05 2011
Errors in file /s01/admin/prod/bdump/prod2_lms2_725312.trc:
ORA-29740: evicted by member , group incarnation
Sun Jun 19 09:18:05 2011
Errors in file /s01/admin/prod/bdump/prod2_lms7_1008288.trc:
ORA-29740: evicted by member , group incarnation
Instance terminated by LMON, pid = 422072
Sun Jun 19 09:21:16 2011
Starting ORACLE instance (normal)
TRACE FILE
==============
prod2_ora_1061088.trc
Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production
ORACLE_HOME = /oracle/app/oracle/product/9.2
System name: AIX
Node name: tprod2
Release: 3
Version: 5
Machine: 00CE5E834C00
Instance name: prod2
*** 2011-06-19 09:06:28.931
================================
PROCESS DUMP FROM HANG ANALYZER:
================================
Current SQL statement for this session:
SELECT formatid, globalid, branchid FROM SYS.DBA_PENDING_TRANSACTIONS ORDER BY formatid, globalid, branchid
*** 2011-06-19 09:06:28.931
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedms+00dc bl ksedst 102905E64 ?
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl nsdo 1102A8098 ? 5500000055 ?
1102DFD20 ? 1102A8200 ?
FFFFFFFFFFFC4E0 ? 000000000 ?
300000003 ?
opikndf2+06a8 bl _ptrgl
opitsk+05fc bl _ptrgl
opiino+0798 bl opitsk 000000000 ? 000000000 ?
opiodr+08e8 bl _ptrgl
opidrv+032c bl opiodr 3C00000018 ? 4101F62A0 ?
FFFFFFFFFFFF8C0 ? 0A057DC60 ?
sou2o+0028 bl opidrv 3C0C000000 ? 4A0644B50 ?
FFFFFFFFFFFF8C0 ?
main+0138 bl 01FD7B5C
__start+0098 bl main 000000000 ? 000000000 ?
Repeat 2 times
----- End of Call Stack Trace -----
*** 2011-06-19 09:06:29.111
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedms+00dc bl ksedst 102905E64 ?
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl nsdo 1102A8098 ? 5500000055 ?
1102DFD20 ? 1102A8200 ?
FFFFFFFFFFFC4E0 ? 000000000 ?
300000003 ?
opikndf2+06a8 bl _ptrgl
opitsk+05fc bl _ptrgl
opiino+0798 bl opitsk 000000000 ? 000000000 ?
opiodr+08e8 bl _ptrgl
opidrv+032c bl opiodr 3C00000018 ? 4101F62A0 ?
FFFFFFFFFFFF8C0 ? 0A057DC60 ?
sou2o+0028 bl opidrv 3C0C000000 ? 4A0644B50 ?
FFFFFFFFFFFF8C0 ?
main+0138 bl 01FD7B5C
__start+0098 bl main 000000000 ? 000000000 ?
----- End of Call Stack Trace -----
*** 2011-06-19 09:06:29.133
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedms+00dc bl ksedst 102905E64 ?
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl nsdo 1102A8098 ? 5500000055 ?
1102DFD20 ? 1102A8200 ?
FFFFFFFFFFFC4E0 ? 000000000 ?
300000003 ?
opikndf2+06a8 bl _ptrgl
opitsk+05fc bl _ptrgl
opiino+0798 bl opitsk 000000000 ? 000000000 ?
opiodr+08e8 bl _ptrgl
opidrv+032c bl opiodr 3C00000018 ? 4101F62A0 ?
FFFFFFFFFFFF8C0 ? 0A057DC60 ?
sou2o+0028 bl opidrv 3C0C000000 ? 4A0644B50 ?
FFFFFFFFFFFF8C0 ?
main+0138 bl 01FD7B5C
__start+0098 bl main 000000000 ? 000000000 ?
----- End of Call Stack Trace -----
*** 2011-06-19 09:06:29.162
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedms+00dc bl ksedst 102905E64 ?
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl nsdo 1102A8098 ? 5500000055 ?
1102DFD20 ? 1102A8200 ?
FFFFFFFFFFFC4E0 ? 000000000 ?
300000003 ?
opikndf2+06a8 bl _ptrgl
opitsk+05fc bl _ptrgl
opiino+0798 bl opitsk 000000000 ? 000000000 ?
opiodr+08e8 bl _ptrgl
opidrv+032c bl opiodr 3C00000018 ? 4101F62A0 ?
FFFFFFFFFFFF8C0 ? 0A057DC60 ?
sou2o+0028 bl opidrv 3C0C000000 ? 4A0644B50 ?
FFFFFFFFFFFF8C0 ?
main+0138 bl 01FD7B5C
__start+0098 bl main 000000000 ? 000000000 ?
----- End of Call Stack Trace -----
*** 2011-06-19 09:06:29.175
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedms+00dc bl ksedst 102905E64 ?
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl nsdo 1102A8098 ? 5500000055 ?
1102DFD20 ? 1102A8200 ?
FFFFFFFFFFFC4E0 ? 000000000 ?
300000003 ?
opikndf2+06a8 bl _ptrgl
opitsk+05fc bl _ptrgl
opiino+0798 bl opitsk 000000000 ? 000000000 ?
opiodr+08e8 bl _ptrgl
opidrv+032c bl opiodr 3C00000018 ? 4101F62A0 ?
FFFFFFFFFFFF8C0 ? 0A057DC60 ?
sou2o+0028 bl opidrv 3C0C000000 ? 4A0644B50 ?
FFFFFFFFFFFF8C0 ?
main+0138 bl 01FD7B5C
__start+0098 bl main 000000000 ? 000000000 ?
----- End of Call Stack Trace -----
*** 2011-06-19 09:06:29.192
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedms+00dc bl ksedst 102905E64 ?
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl nsdo 1102A8098 ? 5500000055 ?
1102DFD20 ? 1102A8200 ?
FFFFFFFFFFFC4E0 ? 000000000 ?
300000003 ?
opikndf2+06a8 bl _ptrgl
opitsk+05fc bl _ptrgl
opiino+0798 bl opitsk 000000000 ? 000000000 ?
opiodr+08e8 bl _ptrgl
opidrv+032c bl opiodr 3C00000018 ? 4101F62A0 ?
FFFFFFFFFFFF8C0 ? 0A057DC60 ?
sou2o+0028 bl opidrv 3C0C000000 ? 4A0644B50 ?
FFFFFFFFFFFF8C0 ?
main+0138 bl 01FD7B5C
__start+0098 bl main 000000000 ? 000000000 ?
----- End of Call Stack Trace -----
Files currently opened by this process:
===================================================
PROCESS STATE
-------------
Process global information:
process: 700000676099520, call: 0, xact: 0, curses: 0, usrses: 700000673decd98
----------------------------------------
SO: 700000676099520, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=224, calls cur/top: 0/7000006c2ca3df8, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 0 0 50
last post received-location: kcbzww
last process to post me: 700000676119f00 7 0
last post sent: 0 0 21
last post sent-location: ksqrcl
last process posted by me: 700000676428258 1 0
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 700000676cc19b0
O/S info: user: oracle, term: UNKNOWN, ospid: 1061088
OSD pid info: Unix process pid: 1061088, image: oracle@tprod2 (TNS V1-V3)
----------------------------------------
END OF PROCESS STATE
******************** Cursor Dump ************************
Current cursor: 2, pgadep: 0
pgactx: 7000006f8bc2d40 ctxcbk: 0 ctxqbc: 0 ctxrws: 700000716aecfd0
Explain plan:
Plan Table
--------
-------------------------------------------------------------------------------------------------------------------------
| Operation | Name | Rows | Bytes | Cost | TQ |IN-OUT| PQ Distrib |Pstart| Pstop |
-------------------------------------------------------------------------------------------------------------------------
| SELECT STATEMENT | | 0 | 0 | 0 | | | | | |
| SORT ORDER BY | | 0 | 0 | 0 | | | | | |
| VIEW | | 0 | 0 | 0 | | | | | |
| SORT UNIQUE | | 0 | 0 | 0 | | | | | |
| UNION-ALL | | 0 | 0 | 0 | | | | | |
| MINUS | | 0 | 0 | 0 | | | | | |
| SORT UNIQUE | | 0 | 0 | 0 | | | | | |
| VIEW | | 0 | 0 | 0 | | | | | |
| FIXED TABLE FULL | X$K2GTE2 | 0 | 0 | 0 | | | | | |
| SORT UNIQUE | | 0 | 0 | 0 | | | | | |
| NESTED LOOPS | | 0 | 0 | 0 | | | | | |
*** 2011-06-19 09:06:29.376
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [1112], [], [], [], [], [], [], []
Current SQL statement for this session:
SELECT formatid, globalid, branchid FROM SYS.DBA_PENDING_TRANSACTIONS ORDER BY formatid, globalid, branchid
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+0148 bl ksedst 102905C84 ?
ksfdmp+0018 bl 01FD8148
kgeriv+0118 bl _ptrgl
kgesiv+0080 bl kgeriv 07FFFFFFC ? 800000000000000 ?
1000000000000000 ?
1800000000000000 ?
028828228 ?
ksesic0+005c bl kgesiv 7000006BE3BB328 ? 000010550 ?
7000006BE3AADD8 ? 10297D7E8 ?
FFFFFFFFFFF3A20 ?
kssadf_stage+0084 bl ksesic0 45800000458 ? 11007A2F8 ?
000000000 ? 000000000 ?
000000000 ? 70000000001DB80 ?
000000000 ? 700000703BBF040 ?
kqreqa+008c bl kssadf_stage 7000006BE3AADD8 ? 10297D7E8 ?
068A31055 ? 000006BB0 ?
000000001 ?
kqrpre1+06e4 bl kqreqa 000000001 ?
kqrpre+001c bl kqrpre1 BAC3F8E66 ? 000000001 ?
FFFFFFFFFFF4008 ? 1101F9A14 ?
1101F9A14 ? FFFFFFFFFFF4000 ?
07FFFFFFF ? 000000000 ?
kkdlobni+0058 bl kqrpre 100F29A04 ?
4222442400000000 ?
14DFD4B95 ?
166CCD19101F62A0 ?
000000002 ? 000000000 ?
FFFFFFFFFFF40C0 ?
xplObjnToName+0150 bl kkdlobni 9A0000009A ?
FFFFFFFFFFF4444 ? 000000000 ?
000000000 ?
xplPatchName+00a4 bl xplObjnToName 9AFFFF46F0 ?
FFFFFFFFFFF4444 ?
xplMakeRow+0190 bl xplPatchName 000000000 ? 000000000 ?
000000000 ?
xplFetchRow+00b4 bl _ptrgl
xplDumpRws+0604 bl xplFetchRow 1029CFB48 ? FFFFFFFFFFF4770 ?
1101F9A14 ?
curdmp+0164 bl xplDumpRws 102AE2A20 ?
ksedms+012c bl curdmp
ksdxfdmp+0200 bl _ptrgl
ksdxcb+02d8 bl _ptrgl
sspuser+0084 bl 01FD7CA8
000044C0 ? 00000000
snttread+0028 bl 00009CFC
nttrd+0118 bl snttread FFFFFFFFFFFBBB3 ?
FFFFFFFFFFFBBA8 ?
FFFFFFFFFFFB2C0 ?
nsprecv+0984 bl _ptrgl
nsrdr+01d0 bl nsprecv 000000000 ? 110299C00 ?
000000000 ?
nsdo+1818 bl nsrdr 000000000 ? 000000000 ?
nioqrc+05c4 bl
Blockers
~~~~~~~~
Above is a list of all the processes. If they are waiting for a resource
then it will be given in square brackets. Below is a summary of the
waited upon resources, together with the holder of that resource.
Notes:
~~~~~
o A process id of '???' implies that the holder was not found in the
systemstate. (The holder may have released the resource before we
dumped the state object tree of the blocking process).
o Lines with 'Enqueue conversion' below can be ignored *unless*
other sessions are waiting on that resource too. For more, see
http://dlsunuk11.uk.oracle.com/Public/TOOLS/Ass.html#enqcnv)
Resource Holder State
Latch 70000000000a4b8 115: Blocker
Latch 70000000000a4b8 210: Blocker
Latch 70000000000a4b8 270: Blocker
Latch 70000000000a4b8 406: Blocker
Latch 70000000000a4b8 614: Blocker
Latch 70000000000a4b8 626: Blocker
Latch 70000000000a4b8 882: Blocker
Latch 70000000000a4b8 1489: Blocker
Latch 70000000000a4b8 1617: Blocker
Latch 70000000000a4b8 1878: Blocker
Latch 70000000000a4b8 1916: Blocker
Latch 70000000000a4b8 1947: Blocker
Latch 70000000000a4b8 1963: Blocker
Latch 70000000000a4b8 2121: 2121: is waiting for Latch 700000675dae330
Latch 70000000000a4b8 2245: Blocker
Latch 70000000000a4b8 2351: Blocker
Latch 70000000000a4b8 2566: Blocker
Latch 70000000000a4b8 2585: Blocker
Latch 70000000000a4b8 2643: Blocker
Latch 70000000000a4b8 2773: 2773: is waiting for Latch 700000675daf3a8
Latch 70000000000a4b8 2791: Blocker
Latch 70000000000a4b8 2795: Blocker
Latch 70000000000a4b8 2966: Blocker
Latch 70000000000a4b8 2969: Blocker
Latch 700000675dadf50 ??? Blocker
Latch 700000675dadc68 ??? Blocker
Latch 700000675dadb70 ??? Blocker
Latch 7000006be3a6530 ??? Blocker
Latch 700000675dae808 ??? Blocker
Latch 700000675db0040 ??? Blocker
Latch 7000006d1d71138 ??? Blocker
Latch 700000675dad3b0 ??? Blocker
Latch 700000675dae330 ??? Blocker
Latch 7000006b2d4fd28 2211: Blocker
Latch 7000006b2d4fd28 2220: Blocker
Latch 7000006b2e5df68 2660: Blocker
Latch 7000006b2e5e3e8 2752: Blocker
Latch 7000006b2e5e3e8 2876: Blocker
Latch 7000006b2d06b28 ??? Blocker
Latch 7000006b2f9f928 ??? Blocker
Latch 7000006b2d4db68 ??? Blocker
Latch 7000006b2e5e868 ??? Blocker
Latch 7000006b2d4e6a8 ??? Blocker
Latch 7000006b2d4eb28 2434: Blocker
Latch 7000006b2d4eb28 2437: 2437: is waiting for 2434: 2437:
Latch 7000006b2d4f428 2925: Blocker
Latch 7000006b2d4f428 2948: Blocker
Latch 7000006b2d07428 ??? Blocker
Latch 7000006b2d4e588 ??? Blocker
Latch 7000006b2e5ece8 ??? Blocker
Latch 7000006b2d4efa8 ??? Blocker
Latch 7000006b2d07c08 ??? Blocker
Latch 7000006b2f9e968 ??? Blocker
Latch 700000675daf3a8 ??? Blocker
Latch 7000006b2a49f68 3198: Blocker
Latch 70000000001a968 ??? Blocker
Some of the above latches may be child latches. Please check the section
named 'Child Latch Report' below for further notes.
Blockers According to Tracefile Wait Info:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. This may not work for 64bit platforms. See bug 2902997 for details.
2. If the blocking process is shown as 0 then that session may no longer be
present.
3. If resources are held across code layers then sometimes the tracefile wait
info will not recognise the problem.
No blockers seen.
Object Names
~~~~~~~~~~~~
Latch 70000000000a4b8 enqueues
Latch 700000675dadf50 Child enqueue hash chains
Latch 700000675dadc68 Child enqueue hash chains
Latch 700000675dadb70 Child enqueue hash chains
Latch 7000006be3a6530 Child row cache objects
Latch 700000675dae808 Child enqueue hash chains
Latch 700000675db0040 Child enqueue hash chains
Latch 7000006d1d71138 Child library cache pin
Latch 700000675dad3b0 Child enqueue hash chains
Latch 700000675dae330 Child enqueue hash chains
Latch 7000006b2d4fd28 Child cache buffers chains
Latch 7000006b2e5df68 Child cache buffers chains
Latch 7000006b2e5e3e8 Child cache buffers chains
Latch 7000006b2d06b28 Child cache buffers chains
Latch 7000006b2f9f928 Child cache buffers chains
Latch 7000006b2d4db68 Child cache buffers chains
Latch 7000006b2e5e868 Child cache buffers chains
Latch 7000006b2d4e6a8 Child cache buffers chains
Latch 7000006b2d4eb28 Child cache buffers chains
Latch 7000006b2d4f428 Child cache buffers chains
Latch 7000006b2d07428 Child cache buffers chains
Latch 7000006b2d4e588 Child cache buffers chains
Latch 7000006b2e5ece8 Child cache buffers chains
Latch 7000006b2d4efa8 Child cache buffers chains
Latch 7000006b2d07c08 Child cache buffers chains
Latch 7000006b2f9e968 Child cache buffers chains
Latch 700000675daf3a8 Child enqueue hash chains
Latch 7000006b2a49f68 Child cache buffers chains
Latch 70000000001a968 Parent transaction allocation
Child Latch Report
~~~~~~~~~~~~~~~~~~
Some processes are being blocked waiting for child latches.
At the moment this script does not detect the blocker because the
child latch address differs to the parent latch address. To manually
detect the blocker please take the following steps :
1. Determine the TYPE of latch (Eg library cache) that is involved.
2. Search the source trace file for a target of :
holding.*Parent.*library cache
(Assuming we have a child library cache and have vi-like regular expressions)
If this shows nothing then the blocker may have released the resource
before we got to dump the state object tree of the blocked process.
A list of processes that hold parent latches is given below :
No processes found.
Summary of Wait Events Seen (count>10)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
No wait events seen more than 10 times
ORA-00600:[1112]内部错误的相关知识如下：
ERROR:
ORA-600 [1112] [a] [b] [c] [d] [e]
VERSIONS:
versions 7.3 to 9.2
DESCRIPTION:
ORA-600 [1112] is getting raised while trying to add a
row cache enqueue to a transaction state object during
lookup of the default tablespace number during table
creation.
FUNCTIONALITY:
STATE OBJECT MANAGEMENT
IMPACT:
PROCESS FAILURE
NON CORRUPTIVE - No underlying data corruption.
Bug 2489130 - OERI:1112 can occur while dumping PROCESSSTATE informatio (Doc ID 2489130.8)
Bug 4126973: ORA-600[504] AND ORA-600[1112] OCCURED WHEN GETTING "ERRORSTACK"
Base Bug 2489130
Bug 3954753: ORA-600 [1112] AND SESSION CRASH
经过诊断发现该ORA-00600:[1112]内部错误是由Bug 2489130所引起的，而触发该Bug的直接原因是WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!：
The cause for the ORA-00600 [1112] appears due to Bug 2489130
This error can occur on dumping of process state which is what occurred here.
The primary issue is the WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!
This then triggers a system state and process state to be dumped due to nature of the problem.
The ORA-00600 [1112] gets dumped out when process state is done.
Stack for trace very similar to Bug 2489130 and this is only known bug on 9.2 like this with a fix.
A fix for bug 2489130 is included in the 9.2.0.7 patchset.
Recommend applying 9.2.0.8 patchset to have this and other bug fixes.
This would only prevent the ORA-00600 [1112] from occurring on state dumps.
解决方案是 优化SQL性能以避免出现WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK!， 或者至少升级数据库版本到9.2.0.8 这个推荐的patchset。