环境： RHEL6.4 + Oracle 11.2.0.4 Primary RAC + Standby RAC
今天发现DG备库归档空间满，清理后发现备库出现GAP，需要从主库做基于SCN的增量备份进行恢复。
这个过程可以参考之前的文章《DG gap sequence修复一例》来完成。
不过本次在recover database noredo时碰到一个错误如下：
RMAN> recover database noredo;
Starting recover at 31-AUG-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=56 instance=jyzhao1 device type=DISK
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: +DATA/mynas/datafile/system.258.951608183
destination for restore of datafile 00002: +DATA/mynas/datafile/sysaux.257.951608183
destination for restore of datafile 00003: +DATA/mynas/datafile/undotbs1.259.951608185
destination for restore of datafile 00004: +DATA/mynas/datafile/users.265.951608205
destination for restore of datafile 00005: +DATA/mynas/datafile/undotbs2.261.951608185
destination for restore of datafile 00006: +DATA/mynas/datafile/dbs_d_jingyu.262.951608185
destination for restore of datafile 00007: +DATA/mynas/datafile/dbs_i_jingyu.263.951608185
destination for restore of datafile 00008: +DATA/mynas/datafile/test.264.951608185
destination for restore of datafile 00009: +DATA/mynas/datafile/test2.260.951608185
destination for restore of datafile 00010: +DATA/mynas/datafile/dbs_d_hank.274.951774467
destination for restore of datafile 00011: +DATA/mynas/datafile/dbadata.276.952933931
channel ORA_DISK_1: reading from backup piece /public/backup/incremental/inc26vsd9r18.bak
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/31/2017 15:11:05
ORA-19870: error while restoring backup piece /public/backup/incremental/inc26vsd9r18.bak
ORA-19573: cannot obtain exclusive enqueue for datafile 11
通过故障现象按关键字“recover database noredo with error ORA-19573”搜索MOS，匹配到文档：
ORA-19573 when trying to restore to standby with incremental backup From Primary (文档 ID 1646232.1)
文中描述原因：
It was failing even though MRP was stopped using sqlplus on standby:
SQL> alter database recover managed standby database cancel;
Below query was NOT returning any rows:
SQL> select process,status,sequence#,thread# from gv$managed_standby where process like 'MRP%';
Dataguard Broker is configured for this environment and MRP was not stopped using dataguard broker.
的确我这里的环境配置了DG Broker，的确我之前也停止过MRP进程，按文档的提示查询MRP进程：
SQL> select process,status,sequence#,thread# from gv$managed_standby where process like 'MRP%';
PROCESS STATUS SEQUENCE# THREAD#
--------- ------------ ---------- ----------
MRP0 WAIT_FOR_GAP 665 1
--再次尝试停止MRP
SQL> alter database recover managed standby database cancel;
Database altered.
--再次查询MRP进程
SQL> select process,status,sequence#,thread# from gv$managed_standby where process like 'MRP%';
no rows selected
我这里在再次停掉MRP进程成功后，直接尝试recover database noredo成功:
RMAN> recover database noredo;
Starting recover at 31-AUG-17
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: +DATA/mynas/datafile/system.258.951608183
destination for restore of datafile 00002: +DATA/mynas/datafile/sysaux.257.951608183
destination for restore of datafile 00003: +DATA/mynas/datafile/undotbs1.259.951608185
destination for restore of datafile 00004: +DATA/mynas/datafile/users.265.951608205
destination for restore of datafile 00005: +DATA/mynas/datafile/undotbs2.261.951608185
destination for restore of datafile 00006: +DATA/mynas/datafile/dbs_d_jingyu.262.951608185
destination for restore of datafile 00007: +DATA/mynas/datafile/dbs_i_jingyu.263.951608185
destination for restore of datafile 00008: +DATA/mynas/datafile/test.264.951608185
destination for restore of datafile 00009: +DATA/mynas/datafile/test2.260.951608185
destination for restore of datafile 00010: +DATA/mynas/datafile/dbs_d_hank.274.951774467
destination for restore of datafile 00011: +DATA/mynas/datafile/dbadata.276.952933931
channel ORA_DISK_1: reading from backup piece /public/backup/incremental/inc26vsd9r18.bak
channel ORA_DISK_1: piece handle=/public/backup/incremental/inc26vsd9r18.bak tag=TAG20170831T150303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:03:45
Finished recover at 31-AUG-17
如果这样不行，可以参考MOS给出的做法：
1. Stop MRP using dataguard broker by running below command:
DGMGRL> edit database '<Standby db_unique_name>' set STATE='APPLY-OFF' ;
2. Retry the restore
至此，本文完。