巡检发现存在alert 日志存在ORA-600
1.0 查询alter 对应的Trace日志
/oracle/admin/fgsquery/udump/fgsquery_ora_21777.trc Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production Machine: x86_64 Redo thread mounted by this instance: 1 Oracle process number: 354 Unix process pid: 21777, image: oracle@pquerydb02 *** ACTION NAME:(SQL 窗口 - 新建) 2018-11-13 11:01:13.082 *** MODULE NAME:(PL/SQL Developer) 2018-11-13 11:01:13.082 *** SERVICE NAME:(shangdi) 2018-11-13 11:01:13.082 *** SESSION ID:(2125.22785) 2018-11-13 11:01:13.082 *** 2018-11-13 11:01:13.082 ksedmp: internal or fatal error ORA-00600: internal error code, arguments: [kcblasm_1], [103], [], [], [], [], [], [] Current SQL statement for this session: select * from ucontevent where eventtype= 'IBQ0010' and eventid in (select eventid from ulcinsureacctrace where confdate is null/* and cvalidate>date'2018-10-01'*/) ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- ssd_unwind_bp: unhandled instruction at 0x3d12d0e instr=f ksedst()+31 call ksedst1() 000000000 ? 000000001 ? 7FFF993CB020 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? ksedmp()+610 call ksedst() 000000000 ? 000000001 ? 7FFF993CB020 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? ksfdmp()+63 call ksedmp() 000000003 ? 000000001 ? 7FFF993CB020 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? kgerinv()+161 call ksfdmp() 006AE9A20 ? 000000003 ? 7FFF993CB020 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? kgeasnmierr()+163 call kgerinv() 006AE9A20 ? 2AC380BF1168 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? 000000000 ? kcblasm1()+113 call kgeasnmierr() 006AE9A20 ? 2AC380BF1168 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? 000000067 ? kcblasm()+52 call kcblasm1() 2AC380ED9D08 ? 7FFF993CAFC0 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? 000000067 ? kxhfFndFreeSlot()+2 call kcblasm() 2AC380ED9D08 ? 000001000 ? 65 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? 000000067 ? kxhfNewBuffer()+222 call kxhfFndFreeSlot() 2AC380ED7A78 ? 000001000 ? 7FFF993CB080 ? 7FFF993CAFC0 ? 000000000 ? 000000067 ? qerhjGetNewBuffer() call kxhfNewBuffer() 2AC380ED7A78 ? 000000038 ? +46 000000000 ? 7FFF993CC1F8 ? 000000000 ? 000000067 ? ksxb1bqb()+178 call qerhjGetNewBuffer() 7FFF993CC1F0 ? 000000038 ? 000000000 ? 7FFF993CC1F8 ? 000000000 ? 000000067 ? kxhrPack()+558 call ksxb1bqb() 7FFF993CC1F0 ? 143F7BA64B ? 000000000 ? 7FFF993CC1F8 ? 000000000 ? 000000067 ? qerhjSplitBuild()+3 call kxhrPack() 000000001 ? 143F7BA64B ? 03 000000001 ? 7FFF993CC1F0 ? 7FFF21F8B302 ? 000000067 ? qertbFetchByRowID() call qerhjSplitBuild() 7FFF993CC810 ? 143F7BA64B ? +1869 200D263A60 ? 000000004 ? 7FFF21F8B302 ? 000000067 ? rwsfcd()+88 call qertbFetchByRowID() 200D264800 ? 002E88560 ? 7FFF993CC810 ? 000007FFE ? 7FFF21F8B302 ? 000000067 ? qerhjFetch()+2549 call rwsfcd() 200D264800 ? 002E88560 ? 7FFF993CC810 ? 000007FFE ? 7FFF21F8B302 ? 000000067 ? opifch2()+2944 call qerhjFetch() 200D263588 ? 003230DC2 ? 7FFF993CCE80 ? 000000064 ? 7FFF21F8B302 ? 000000067 ? opifch()+64 call opifch2() 000000089 ? 000000005 ? 7FFF993CD060 ? 2AC380EDDAE0 ? 7FFF21F8B302 ? 000000067 ? opiodr()+1184 call opifch() 000000089 ? 000000005 ? 7FFF993CD060 ? 000000064 ? 000000000 ? 0013B0002 ? ttcpip()+1226 call opiodr() 000000005 ? 000000002 ? 7FFF993CFDE8 ? 000000001 ? 005BEAF88 ? 0013B0002 ? opitsk()+1310 call ttcpip() 006AF1FB0 ? 0054A51E0 ? 7FFF993CFDE8 ? 000000000 ? 7FFF993CF8E8 ? 7FFF993CFF50 ? opiino()+1024 call opitsk() 000000003 ? 000000000 ? 7FFF993CFDE8 ? 000000001 ? 000000000 ? 832011000000001 ? opiodr()+1184 call opiino() 00000003C ? 000000004 ? 7FFF993D0FE8 ? 000000000 ? 000000000 ? 832011000000001 ? opidrv()+548 call opiodr() 00000003C ? 000000004 ? 7FFF993D0FE8 ? 000000000 ? 005BEB820 ? 832011000000001 ? sou2o()+114 call opidrv() 00000003C ? 000000004 ? 7FFF993D0FE8 ? 000000000 ? 005BEB820 ? 832011000000001 ? opimai_real()+163 call sou2o() 7FFF993D0FC0 ? 00000003C ? 000000004 ? 7FFF993D0FE8 ? 005BEB820 ? 832011000000001 ? main()+116 call opimai_real() 000000002 ? 7FFF993D1050 ? 000000004 ? 7FFF993D0FE8 ? 005BEB820 ? 832011000000001 ? __libc_start_main() call main() 000000002 ? 7FFF993D1050 ? +244 000000004 ? 7FFF993D0FE8 ? 005BEB820 ? 832011000000001 ? _start()+41 call __libc_start_main() 00072D108 ? 000000002 ? 7FFF993D11A8 ? 000000000 ? 005BEB820 ? 000000002 ? --------------------- Binary Stack Dump ---------------------
2.0 匹配Mos Bug
ORA-600 [kcblasm_1] In 10.2.0.5. (文档 ID 1133845.1)
Oracle 内部错误，bug,解决方式安装小布丁，psu，大版本升级······
ORA-00600: internal error code, arguments: [kcblasm_1], [103], [], [], [], [], [], []
The call stack in the generated trace file looks similar to:
... kcblasm1 kcblasm kxhfFndFreeSlot kxhfNewBuffer qerhjGetNewBuffer ksxb1bqb kxhrPack ... 根据函数堆栈调用，发现顺序非常匹配
CAUSE
This is is a known and common problem hit in 10.2.0.5, investigated and corrected in unpublished Bug 7612454.
The problem was introduced in 10.2.0.5, by the fix of Bug:7523755.
This has been corrected in PSU 10.2.0.5.4 and 11.2.
Please refer to:
Note 7612454.8 - Bug 7612454 - More "direct path read" operations / OERI:kcblasm_1
CHANGES
This problem is hit only in 10.2.0.5, up to PSU 10.2.0.5.4 in which problem is fixed.
Problem is affecting 10.2.0.5 to 10.2.0.5.3.
SOLUTION
The quickest way to solve the problem is to apply PSU 10.2.0.5.4, Patch 12419392 or later.
All alternative solutions for this problem are listed below:
Upgrade the database to 11.2.
- OR -
Apply 10.2.0.5.4 Patch Set Update (Patch 12419392) or later PSUs where bug is fixed.
The available PSUs are mentioned in "10.2.0.5 Patch Set Updates - List of Fixes in each PSU" (Document 1337394.1)
- OR -
Apply interim Patch 7612454 on top of 10.2.0.5 (10.2.0.5.0-10.2.0.5.3):
For UNIX / Linux platforms apply Patch 7612454 available for download on MOS.
For Windows platforms apply Patch 3 or higher.
Please check Document 342443.1 for latest patches available for Windows on top of 10.2.0.5.
解决方案：1.升级数据库版本-11.2
2.patch 12419392 apply 10.2.0.5.4
3. patch 7612454 on top of 10.2.0.5 (10.2.0.5.0-10.2.0.5.3):
根据Mos的查询反馈结果，基本不会采纳，对问题sql进行查询
3.0 数据库查询信息
查询会话信息，ash已被刷出，使用awr视图查询
SQL>selelct sample_time,user_id,sql_id,event from v$active_session_history where sample_time between to_date('20181113 09','yyyymmdd hh24') and to_date('20181113 13','yyyymmdd hh24') and session_id=2125 and session_serial#=22785; SQL>select owner,object_type,status,object_name from dba_objects where object_name =upper('ucontevent') and object_type like '%TABLE%' OWNER OBJECT_TYPE STATUS OBJECT_NAME ------------------------------ ------------------- ------- ------------------------------ LISPRDD TABLE VALID UCONTEVENT
SQL> select sample_time,user_id,sql_id,event from dba_hist_active_sess_history where sample_time between to_date('20181113 11','yyyymmdd hh24') and to_date('20181113 12','yyyymmdd hh24') and session_id=2125 and session_serial#=22785;
SAMPLE_TIME USER_ID SQL_ID EVENT
------------------------------ ---------- ------------- ----------------------------------------------------------------
13-NOV-18 11.04.18.479 AM 249 cxp0p771cnx1f
13-NOV-18 11.02.18.293 AM 249 42wvj1k385aun
13-NOV-18 11.01.18.193 AM 249 4bbwc4gsfxu33
SQL> select username,sql_id,event,program,action,machine,port from dba_hist_active_sess_history a,dba_users b where a.user_id=b.user_id and sample_time between to_date('20181113 11','yyyymmdd hh24') and to_date('20181113 12','yyyymmdd hh24') and session_id=2125 and session_serial#=22785;
USERNAME SQL_ID EVENT PROGRAM ACTION MACHINE PORT
---------- ------------- ---------- ------------------------------ -------------------------------- ------------------------------ ----------
ZKRWANGYB 4bbwc4gsfxu33 plsqldev.exe SQL 窗口 - 新建 WORKGROUPPQZBLJVAPP1 59450
ZKRWANGYB 42wvj1k385aun plsqldev.exe SQL 窗口 - 新建 WORKGROUPPQZBLJVAPP1 59450
ZKRWANGYB cxp0p771cnx1f plsqldev.exe SQL 窗口 - 新建 WORKGROUPPQZBLJVAPP1 59450
USERNAME SQL_ID EVENT PROGRAM ACTION MACHINE PORT
---------- ------------- ---------- ------------------------------ -------------------------------- ------------------------------ ----------
ZKRWANGYB 4bbwc4gsfxu33 plsqldev.exe SQL 窗口 - 新建 WORKGROUPPQZBLJVAPP1 59450
ZKRWANGYB 42wvj1k385aun plsqldev.exe SQL 窗口 - 新建 WORKGROUPPQZBLJVAPP1 59450
ZKRWANGYB cxp0p771cnx1f plsqldev.exe SQL 窗口 - 新建 WORKGROUPPQZBLJVAPP1 59450
SQL> select * from table(dbms_xplan.display_awr('&sql_id'))
Enter value for sql_id: 42wvj1k385aun
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------
SQL_ID 42wvj1k385aun
--------------------
select * from ucontevent where eventtype = 'IBQ0010' and eventid in (select eventid
from ulcinsureacctrace where confdate is null )
Plan hash value: 3993290469
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | | 1218K(100)| |
| 1 | HASH JOIN SEMI | | 2282K| 298M| 274M| 1218K (2)| 04:03:46 |
| 2 | TABLE ACCESS BY INDEX ROWID| UCONTEVENT | 2282K| 248M| | 114K (1)| 00:22:50 |
| 3 | INDEX RANGE SCAN | UCONTEVENT_INDEX1 | 2282K| | | 6260 (1)| 00:01:16 |
| 4 | TABLE ACCESS FULL | ULCINSUREACCTRACE | 104M| 2289M| | 915K (2)| 03:03:09 |
----------------------------------------------------------------------------------------------------------
17 rows selected.
Enter value for sql_id: 42wvj1k385aun
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------
SQL_ID 42wvj1k385aun
--------------------
select * from ucontevent where eventtype = 'IBQ0010' and eventid in (select eventid
from ulcinsureacctrace where confdate is null )
Plan hash value: 3993290469
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | | 1218K(100)| |
| 1 | HASH JOIN SEMI | | 2282K| 298M| 274M| 1218K (2)| 04:03:46 |
| 2 | TABLE ACCESS BY INDEX ROWID| UCONTEVENT | 2282K| 248M| | 114K (1)| 00:22:50 |
| 3 | INDEX RANGE SCAN | UCONTEVENT_INDEX1 | 2282K| | | 6260 (1)| 00:01:16 |
| 4 | TABLE ACCESS FULL | ULCINSUREACCTRACE | 104M| 2289M| | 915K (2)| 03:03:09 |
----------------------------------------------------------------------------------------------------------
17 rows selected.
SQL>select owner,segment_name,sum(bytes)/1024/1024 from dba_segments where segment_name in('UCONTEVENT','ULCINSUREACCTRACE') group by segment_name,owner
OWNER SEGMENT_NAME SUM(BYTES)/1024/1024
------------------------------ ------------------------------ --------------------
LISPRDD ULCINSUREACCTRACE 33170
LISPRDD UCONTEVENT 8198
!SQL能否改写~全表扫描33G的数据量--只取部分数据
SQL> select count(*) from (select eventid from LISPRDD.ulcinsureacctrace where confdate is null);
COUNT(*)
----------
106790272
SQL> select num_rows from dba_tables where owner='LISPRDD' and table_name=upper('ulcinsureacctrace');
NUM_ROWS
----------
105814076
SQL> select count(*) from LISPRDD.ulcinsureacctrace sample block(10);
COUNT(*)
----------
10946783
SQL> select last_analyzed from dba_tables where owner='LISPRDD' and table_name=upper('ulcinsureacctrace');
LAST_ANALYZED
-------------------
2018-11-16 22:10:25
SQL> select column_name,data_type,num_distinct,num_nulls,last_analyzed from dba_tab_columns where owner='LISPRDD' and table_name=upper('ulcinsureacctrace') and column_name in('EVENTID','CONFDATE');
COLUMN_NAME DATA_TYPE NUM_DISTINCT NUM_NULLS LAST_ANALYZED
------------------------------ ------------------------------ ------------ ---------- -------------------
EVENTID VARCHAR2 62115722 0 2018-11-16 22:10:25
CONFDATE DATE 150 105578361 2018-11-16 22:10:25
COUNT(*)
----------
106790272
SQL> select num_rows from dba_tables where owner='LISPRDD' and table_name=upper('ulcinsureacctrace');
NUM_ROWS
----------
105814076
SQL> select count(*) from LISPRDD.ulcinsureacctrace sample block(10);
COUNT(*)
----------
10946783
SQL> select last_analyzed from dba_tables where owner='LISPRDD' and table_name=upper('ulcinsureacctrace');
LAST_ANALYZED
-------------------
2018-11-16 22:10:25
SQL> select column_name,data_type,num_distinct,num_nulls,last_analyzed from dba_tab_columns where owner='LISPRDD' and table_name=upper('ulcinsureacctrace') and column_name in('EVENTID','CONFDATE');
COLUMN_NAME DATA_TYPE NUM_DISTINCT NUM_NULLS LAST_ANALYZED
------------------------------ ------------------------------ ------------ ---------- -------------------
EVENTID VARCHAR2 62115722 0 2018-11-16 22:10:25
CONFDATE DATE 150 105578361 2018-11-16 22:10:25
4.0 分析结论
问题分析后，此ora-600为一个bug，遭遇此报错的是一个pl/sql 开发人员，且写的sql效率非常低，非业务sql，因此此报错可忽略。 如果是业务sql频繁报错，可能计划打补丁；
本次建议，让开发人员对此sql进行优化处理，sql的查询逻辑是否有改进的空间。