今天遇到一个奇怪问题，发现srvctl和crs_start命令无法启动Oracle RAC实例，但用sqlplus却可以正常启动。最终发现原因是在OCR中数据库的状态变成了disable，将此状态更改为enable后恢复正常。
以下是一个模拟示例：
oracle@his00mlcdrc02:~ $> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE his0...rc01
ora....01.lsnr application ONLINE ONLINE his0...rc01
ora....c01.gsd application ONLINE ONLINE his0...rc01
ora....c01.ons application ONLINE ONLINE his0...rc01
ora....c01.vip application ONLINE ONLINE his0...rc01
ora....SM2.asm application ONLINE ONLINE his0...rc02
ora....02.lsnr application ONLINE ONLINE his0...rc02
ora....c02.gsd application ONLINE ONLINE his0...rc02
ora....c02.ons application ONLINE ONLINE his0...rc02
ora....c02.vip application ONLINE ONLINE his0...rc02
ora....rcdb.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application OFFLINE OFFLINE
oracle@his00mlcdrc02:~ $> ps -ef | grep ora_
oracle 23001 16923 0 16:31:27 pts/1 0:00 grep ora_
oracle@his00mlcdrc02:~ $> crs_start ora.hisdrcdb.hisdrcdb2.inst
Attempting to start `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02`
Start of `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02` failed. (在节点his00mlcdrc02上启动失败）
his00mlcdrc01 : CRS-1019: Resource ora.hisdrcdb.hisdrcdb2.inst (application) cannot run on his00mlcdrc01
(对于象VIP这类可以failover到其它节点的资源，如果在原节点上启动不成功，则Oracle会试图在其它节点上启动。而实例是不能failover的资源，此处提示实例2不能在节点1上启动)
CRS-0215: Could not start resource 'ora.hisdrcdb.hisdrcdb2.inst'.
oracle@his00mlcdrc02:~ $> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE his0...rc01
ora....01.lsnr application ONLINE ONLINE his0...rc01
ora....c01.gsd application ONLINE ONLINE his0...rc01
ora....c01.ons application ONLINE ONLINE his0...rc01
ora....c01.vip application ONLINE ONLINE his0...rc01
ora....SM2.asm application ONLINE ONLINE his0...rc02
ora....02.lsnr application ONLINE ONLINE his0...rc02
ora....c02.gsd application ONLINE ONLINE his0...rc02
ora....c02.ons application ONLINE ONLINE his0...rc02
ora....c02.vip application ONLINE ONLINE his0...rc02
ora....rcdb.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application ONLINE OFFLINE
此时可以看到实例2启动并未成功。
oracle@his00mlcdrc02:~ $> srvctl start instance -d hisdrcdb -i hisdrcdb2
用srvctl命令尝试启动实例2，命令很快就返回了。
oracle@his00mlcdrc02:~ $> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE his0...rc01
ora....01.lsnr application ONLINE ONLINE his0...rc01
ora....c01.gsd application ONLINE ONLINE his0...rc01
ora....c01.ons application ONLINE ONLINE his0...rc01
ora....c01.vip application ONLINE ONLINE his0...rc01
ora....SM2.asm application ONLINE ONLINE his0...rc02
ora....02.lsnr application ONLINE ONLINE his0...rc02
ora....c02.gsd application ONLINE ONLINE his0...rc02
ora....c02.ons application ONLINE ONLINE his0...rc02
ora....c02.vip application ONLINE ONLINE his0...rc02
ora....rcdb.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application ONLINE OFFLINE
srvctl命令启动也不成功。
oracle@his00mlcdrc02:~ $> sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.4.0 - Production on Sat Oct 9 16:37:38 2010
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to an idle instance.
SQL> startup;
ORACLE instance started.
Total System Global Area 1610612736 bytes
Fixed Size 2139192 bytes
Variable Size 399465416 bytes
Database Buffers 1174405120 bytes
Redo Buffers 34603008 bytes
Database mounted.
Database opened.
SQL> show parameter instance_name
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
instance_name string hisdrcdb2
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ WRITE
用sqlplus启动实例成功了。
SQL> host crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE his0...rc01
ora....01.lsnr application ONLINE ONLINE his0...rc01
ora....c01.gsd application ONLINE ONLINE his0...rc01
ora....c01.ons application ONLINE ONLINE his0...rc01
ora....c01.vip application ONLINE ONLINE his0...rc01
ora....SM2.asm application ONLINE ONLINE his0...rc02
ora....02.lsnr application ONLINE ONLINE his0...rc02
ora....c02.gsd application ONLINE ONLINE his0...rc02
ora....c02.ons application ONLINE ONLINE his0...rc02
ora....c02.vip application ONLINE ONLINE his0...rc02
ora....rcdb.db application OFFLINE OFFLINE
ora....b1.inst application OFFLINE OFFLINE
ora....b2.inst application ONLINE OFFLINE
但crs_stat显示的状态仍然是OFFLINE.
先把实例2停止。
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
oracle@his00mlcdrc02:~ $> ocrdump
查看OCRDUMPFILE
......
[DATABASE.DATABASES.hisdrcdb.ENABLED]
ORATEXT : false
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}
[DATABASE.DATABASES.hisdrcdb.POLICY]
ORATEXT : MANUAL
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}
......
OCR中显示数据库DATABASE.DATABASES.hisdrcdb.ENABLED属性为false，DATABASE.DATABASES.hisdrcdb.POLICY为manual,而在别的数据库上可以看到这两个属性应为true和automatic.
oracle@his00mlcdrc02:~ $> srvctl status database -d hisdrcdb -f
Instance hisdrcdb1 is disabled
Instance hisdrcdb1 is not running on node his00mlcdrc01
Instance hisdrcdb2 is disabled
Instance hisdrcdb2 is not running on node his00mlcdrc02
srvctl状态表明数据库被disable了。
oracle@his00mlcdrc02:~ $> srvctl enable database -d hisdrcdb
oracle@his00mlcdrc02:~ $> srvctl status database -d hisdrcdb -f
Instance hisdrcdb1 is not running on node his00mlcdrc01
Instance hisdrcdb2 is not running on node his00mlcdrc02
用srvctl命令enable database
oracle@his00mlcdrc02:~ $> ocrdump ocr1
查看ocr1:
......
[DATABASE.DATABASES.hisdrcdb.ENABLED]
ORATEXT : true
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}
[DATABASE.DATABASES.hisdrcdb.POLICY]
ORATEXT : AUTOMATIC
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}
......
此时这两个属性恢复为true和automatic.
oracle@his00mlcdrc02:~ $> crs_start ora.hisdrcdb.hisdrcdb1.inst
Attempting to start `ora.hisdrcdb.hisdrcdb1.inst` on member `his00mlcdrc01`
Start of `ora.hisdrcdb.hisdrcdb1.inst` on member `his00mlcdrc01` succeeded.
oracle@his00mlcdrc02:~ $>
oracle@his00mlcdrc02:~ $> crs_start ora.hisdrcdb.hisdrcdb2.inst
Attempting to start `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02`
Start of `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02` succeeded.
oracle@his00mlcdrc02:~ $> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE his0...rc01
ora....01.lsnr application ONLINE ONLINE his0...rc01
ora....c01.gsd application ONLINE ONLINE his0...rc01
ora....c01.ons application ONLINE ONLINE his0...rc01
ora....c01.vip application ONLINE ONLINE his0...rc01
ora....SM2.asm application ONLINE ONLINE his0...rc02
ora....02.lsnr application ONLINE ONLINE his0...rc02
ora....c02.gsd application ONLINE ONLINE his0...rc02
ora....c02.ons application ONLINE ONLINE his0...rc02
ora....c02.vip application ONLINE ONLINE his0...rc02
ora....rcdb.db application ONLINE ONLINE his0...rc02
ora....b1.inst application ONLINE ONLINE his0...rc01
ora....b2.inst application ONLINE ONLINE his0...rc02
再次用crs_start命令则能够正常启动实例。
附：srvctl disable disables target, meaning CRS will not consider it for automatic startup, failover, or restart. This option is useful to ensure an object that is down for maintenance is not accidentally automatically restarted.
srvctl enable reenables the specified object.