ORA00210 ORA00202 ORA27086

One oracle database crushed due to NFS issue. The database binary was mounted from another server A. So once the NFS was disconnected, the database was crushed. I tried to start it up today but hit below issue.

SQL> startup
ORACLE instance started.

Total System Global Area 2138521600 bytes
Fixed Size                  2161024 bytes
Variable Size            1761609344 bytes
Database Buffers          369098752 bytes
Redo Buffers                5652480 bytes
ORA-01102: cannot mount database in EXCLUSIVE mode

Then I try to check the alert log. But alert log for oracle 11g is stored in a different location. So I check the parameter background_dump_dest to find the alert log.

SQL> show parameter background

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
background_core_dump                 string                            partial
background_dump_dest                 string                            /opt/oracle/diag/rdbms/mrad/mr
                                                                       ad/trace

The alert log shows below error

ORACLE_BASE from environment = /opt/oracle
Tue May 28 16:28:44 2013
ALTER DATABASE   MOUNT
sculkget: failed to lock /opt/oracle/product/11.2/mrad/dbs/lkMRAD exclusive
sculkget: lock held by PID: 1078
ORA-09968: unable to lock file
SVR4 Error: 11: Resource temporarily unavailable
Additional information: 1078
ORA-1102 signalled during: ALTER DATABASE   MOUNT...
Tue May 28 16:35:08 2013
alter database mount

The file /opt/oracle/product/11.2/mrad/dbs/lkMRAD is an lock file so I renamed it. And try to restart again. Now the error is 

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-00205: error in identifying control file, check alert log for more info

Check the alert log find below

ALTER DATABASE   MOUNT
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/opt/oracle/oradata/mrad/sys03/control03.ctl'
ORA-27086: unable to lock file - already in use
SVR4 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 1084
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/opt/oracle/oradata/mrad/sys02/control02.ctl'
ORA-27086: unable to lock file - already in use
SVR4 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 1084
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/opt/oracle/oradata/mrad/sys01/control01.ctl'
ORA-27086: unable to lock file - already in use
SVR4 Error: 11: Resource temporarily unavailable
Additional information: 8
Additional information: 1084

It says there is lock on control file. This is because the NFS lock was not released properly due to last time crush. So I run below command to release the lock. 

-bash-4.1# clear_locks -s A
Clearing locks held for NFS client B on server A
clear of locks held for B on A returned success

　　

Then start the database. Everything is ok