ORACLE RAC OCR 不发找到引起的血案

在一次系统维护过程中，尝试启动RAC环境，结果RAC服务没有启动，在/tmp目录下发现了这个错误：

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such device or address] [6]

 

 

前两天检查备份日志时发现，在释放CHANNEL的时候报错，进一步详细的检查发现，带库有一个DRIVE DOWN掉了，备份只能在一个CHANNEL上进行，因此备份日志中出现了错误，错误信息如下：

bash-3.00$ more /data/backup/backup_tradedb_081101.out

Script. /data/backup/backup_tradedb.shITPUB个人空间A xdZ(pw'C Q'C8d'P1F
==== started on Sat Nov 1 23:00:00 CST 2008 ====

"Vh-ue.YIY
h7rL}0RMAN: /opt/oracle/product/10.2/database/bin/rman
{:UL.\K0ORACLE_SID: tradedb1
$R.|$`{6@0ORACLE_HOME: /opt/oracle/product/10.2/database
l-aJ? o1l$N4D0RMAN> 2> 3> 4> 5> 6> 7> 8> RMAN> 2> 3> 4> 5> 6> 7> 8> 9> RMAN> 2> 3> 4> RMAN>
X'b2H y]#{0Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: TRADEDB (DBID=4181457554)ITPUB个人空间6e Hc'R$c
using target database control file instead of recovery catalog

RMAN> 2> 3> 4> 5> 6> 7> 8>ITPUB个人空间"R;g6}$`T!?
L
allocated channel: C1
u:V0n[ L)M `0channel C1: sid=112 instance=tradedb1 devtype=SBT_TAPE
/GjHc+_?8M0channel C1: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: C2ITPUB个人空间$xx6O,R_G
channel C2: sid=146 instance=tradedb1 devtype=SBT_TAPEITPUB个人空间1j B9gkak
channel C2: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

Starting backup at 01-NOV-08
7gd._c:~F9O0input backupset count=842 stamp=669081253 creation_time=25-OCT-08
3r+{
Op*i jB0channel C2: starting piece 1 at 01-NOV-08ITPUB个人空间.a5j.mf&G'E;N#V
channel C2: backup piece /data/backup/tradedb/qaju2nl5_1_1ITPUB个人空间]#N"i*~J1r
d6u#{
input backupset count=840 stamp=669080836 creation_time=25-OCT-08
vX7Epc5]A0channel C1: starting piece 1 at 01-NOV-08ITPUB个人空间x`/`/j B k,f
channel C1: backup piece /data/backup/tradedb/q8ju2n84_1_1
!h#zv%g.N ?
ry0piece handle=qaju2nl5_1_2 comment=API Version 2.0,MMS Version 5.0.0.0ITPUB个人空间 {3}
b:M/ir'M7G
channel C2: finished piece 1 at 01-NOV-08ITPUB个人空间e_ B ^.l`}
u A}
channel C2: backup set complete, elapsed time: 00:03:35ITPUB个人空间-JM M H2w
deleted backup piece
'M9pvJ@0_0backup piece handle=/data/backup/tradedb/qaju2nl5_1_1 recid=1446 stamp=669081254ITPUB个人空间jr
M:N b2M*`9DH
input backupset count=841 stamp=669080836 creation_time=25-OCT-08
9F6EVl.oz,QO0@0channel C2: starting piece 1 at 01-NOV-08
6d[;F2A(a k0channel C2: backup piece /data/backup/tradedb/q9ju2n84_1_1ITPUB个人空间*To2nC-x
piece handle=q9ju2n84_1_2 comment=API Version 2.0,MMS Version 5.0.0.0
Kf0Jh#Z"K0channel C2: finished piece 1 at 01-NOV-08
'a],G R/_0channel C2: backup set complete, elapsed time: 00:03:15
yms0?9IC}0deleted backup piece
/]"\9r3v%?2m
b0backup piece handle=/data/backup/tradedb/q9ju2n84_1_1 recid=1447 stamp=669080837ITPUB个人空间_6])N6d0Gd
input backupset count=843 stamp=669081317 creation_time=25-OCT-08ITPUB个人空间u0w| IHie
channel C2: starting piece 1 at 01-NOV-08
XIy[;R,btl.]0channel C2: backup piece /data/backup/tradedb/qbju2nn5_1_1
'n u,J]Q7C9q0cz0piece handle=qbju2nn5_1_2 comment=API Version 2.0,MMS Version 5.0.0.0
;\ ^3]i O;_OkO2F0channel C2: finished piece 1 at 01-NOV-08
7y8` n*G4G#G:O*O0channel C2: backup set complete, elapsed time: 00:11:46ITPUB个人空间9y`F \"miz
deleted backup piece
2Dcp5p$mj![n0backup piece handle=/data/backup/tradedb/qbju2nn5_1_1 recid=1448 stamp=669081317ITPUB个人空间-\Ma
e_G4k
input backupset count=844 stamp=669081317 creation_time=25-OCT-08ITPUB个人空间^;ci Q[k8D-IX
channel C2: starting piece 1 at 01-NOV-08ITPUB个人空间,de*y A5pe(n*Ji
channel C2: backup piece /data/backup/tradedb/qcju2nn5_1_1ITPUB个人空间'C8[ ] t
q.Hj
RMAN-03009: failure of backup command on C1 channel at 11/01/2008 23:27:19
?7p0P2a'JA b@!]0ORA-19506: failed to create sequential file, name="q8ju2n84_1_2", parms=""ITPUB个人空间9X3O` fWa
ORA-27028: skgfqcre: sbtbackup returned errorITPUB个人空间qRZ2_Rn;x^
]
ORA-19511: Error received from media manager layer, error text:ITPUB个人空间+t9G j;Z3i
   VxBSACreateObject: Failed with error:ITPUB个人空间 Jju1h4x[8A[
v
   Server Status:  network connection timed outITPUB个人空间z:t&`1~HtisHY
ORA-19600: input file is backup piece  (/data/backup/tradedb/q8ju2n84_1_1)
1dN4gxQA.Va0ORA-19601: output file is backup piece  (q8ju2n84_1_2)ITPUB个人空间.]n'{1}A?.B}
channel C1 disabled, job failed on it will be run on another channelITPUB个人空间0I?%N9F8C5z#a&j4a5] jJ
piece handle=qcju2nn5_1_2 comment=API Version 2.0,MMS Version 5.0.0.0
jy4Eg/ko"MG)O0channel C2: finished piece 1 at 01-NOV-08
:z Md-M
n0channel C2: backup set complete, elapsed time: 00:21:41ITPUB个人空间*A#C-X;N+W7Bj
deleted backup pieceITPUB个人空间*sS"K!_2s
backup piece handle=/data/backup/tradedb/qcju2nn5_1_1 recid=1449 stamp=669081322ITPUB个人空间RCP S"j(uR5P y
input backupset count=840 stamp=669080836 creation_time=25-OCT-08
U5Jb1|_7AG0channel C2: starting piece 1 at 01-NOV-08
W&lz#GI0channel C2: backup piece /data/backup/tradedb/q8ju2n84_1_1
"f V+I-ks"t:d7eh0piece handle=q8ju2n84_1_2 comment=API Version 2.0,MMS Version 5.0.0.0
XU7ol-vE'[bS0channel C2: finished piece 1 at 01-NOV-08
Iq&P+yb&H0channel C2: backup set complete, elapsed time: 00:12:26
L.h2j6v,}5VB ]J0deleted backup piece
9|6oS?4_,R5Y4N6cd0backup piece handle=/data/backup/tradedb/q8ju2n84_1_1 recid=1445 stamp=669080837
+b:fh P9OI0input backupset count=846 stamp=669083380 creation_time=26-OCT-08ITPUB个人空间,FuKv7[Zm
.ITPUB个人空间*~ ^h%V-] R&J
.

MTRj_0.ITPUB个人空间6Oa3y,qO]
channel C2: starting piece 1 at 02-NOV-08
M)?(B1Nj*E)N.^$J:J0channel C2: backup piece /data/backup/tradedb/qhju2q9f_1_1ITPUB个人空间Ym4~(] xyV B\R}4^
piece handle=qhju2q9f_1_2 comment=API Version 2.0,MMS Version 5.0.0.0ITPUB个人空间D(Aa&{ u
channel C2: finished piece 1 at 02-NOV-08ITPUB个人空间+pk\"h*ag0T5G
channel C2: backup set complete, elapsed time: 00:08:56ITPUB个人空间1kOB*MI#t@
deleted backup pieceITPUB个人空间 t!D!S-jO
backup piece handle=/data/backup/tradedb/qhju2q9f_1_1 recid=1454 stamp=669083952
Qq7@mP ]7h9ff0Finished backup at 02-NOV-08

released channel: C1
N hZ?9g4cY7F0released channel: C2ITPUB个人空间3Q~'a-F7t
mS@(Nd
RMAN-00571: ===========================================================ITPUB个人空间h
gcLrU|
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============ITPUB个人空间A-}5?Q~R
\
RMAN-00571: ===========================================================ITPUB个人空间gx9~] yP8aZlP
RMAN-03002: failure of release command at 11/02/2008 00:44:39ITPUB个人空间y p.X6mb%zWuNp
RMAN-06012: channel: C1 not allocated

手头启动DRIVE，没有发现异常，但是一旦执行备份，这个DRIVE就DOWN掉了。尝试修改这个DRIVE的配置，发现DRIVE原本的路径对于NETBACKUP根本无法加载，看来可能是硬件问题导致了原因。

于是系统维护人员到现场解决问题，发现是光纤交换机出现了故障，于是重启了光纤交换机。由于RAC环境也依赖该光纤交换机，但是RAC环境配置了双路光纤交换机，因此重启光交的时候没有停RAC服务。

结果光纤交换机重启的结果导致RAC的一个节点服务器暂时无法启动，而另一个节点服务器也发生了重启。

由于RAC环境完全DOWN掉，于是尝试在目前可以启动的节点上启动RAC服务：

# /etc/init.d/init.crs start

Startup will be queued to init within 30 seconds.

服务启动后半天没有响应，检查后台经常没有任何的Oracle实例启动，感觉不太对劲，检查/tmp目录发现了上面的错误信息：

bash-3.00# cd /tmpITPUB个人空间+\X2I2h-hsz A.L
bash-3.00# lsITPUB个人空间#M,b&P/GQ A"SL
crsctl.4483          crsctl.4492          crsctl.4493          hsperfdata_noaccess  hsperfdata_root      ssh-sIvv2068ITPUB个人空间d`*iF"TT1d|
bash-3.00# ls -l
9qfw jpC0total 96
R[;t;U3k:xG%g-~0-rw-r--r--   1 oracle   oinstall     155 Nov  5 20:46 crsctl.4483
q }.}C?(v7B.q t]0-rw-r--r--   1 oracle   oinstall     155 Nov  5 20:46 crsctl.4492ITPUB个人空间
{3G%V0hsCj,d
-rw-r--r--   1 oracle   oinstall     155 Nov  5 20:46 crsctl.4493
m y f ~~e+X \0drwxr-xr-x   2 noaccess noaccess     178 Nov  5 19:53 hsperfdata_noaccessITPUB个人空间(W!Et/n6MQ4B%O*y7g+b
drwxr-xr-x   2 root     root         117 Nov  5 19:54 hsperfdata_rootITPUB个人空间"A~"L^ Sp
drwx------   2 root     root         184 Nov  5 19:57 ssh-sIvv2068
ln vr;w*gB:DJ0bash-3.00# more crsctl.4483ITPUB个人空间p1k,? ]5j{F)R4t
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such device or address]

Oracle的共享存储是通过VERITAS的VOLUMN CLUSTER MANAGER进行管理的，目前DOWN掉的节点是VOLUMN CLUSTER MANAGER的主节点，但是在当前节点上可以看到OCR裸设备、VOT裸设备以及所有的控制文件、日志文件、数据文件和参数文件的裸设备，这些裸设备的访问路径都是正常的，为什么还会导致这个错误呢。

查询了METALINK，发现可能是bug：Bug No. 3613622中描述的问题：

The problem here is that no node cannot rely on its perception of the network,since the network may be broken in an undetectable manner, so the node must have access to the voting disk.  When access to the voting disk is lost, or the I/O takes 'too long', the node must fail.

When Veritas CVM runs with Vendor Clusterware, then the Vendor Clusterware is the primary driver of node reconfiguration,@ not the miss count setting of CSS.  As John mentioned above,@ on Sun Cluster by default CSS tolerates up to almost 10 minutes@ of Veritas CVM I/O suspension.  It is Veritas's problem to fix.

看来问题很可能是由于VERITAS的CVM引起的，而且在一段时间后，这个节点上的RAC确实可以启动了，不过由于当时节点1恰好也可以正确启动了，所以不好确定是否是由于主节点的启动导致了问题消失，还是由于等待时间超过了10分钟，使得这个问题得以解决。

先记录这个问题，以后如果有机会的话，还要验证一下。