A dataguad exception of 10g RAC

When restarting a dataguad today, the MRP process could not be started. The error is as follows

Wed Mar 9 15:32:55 2016ALTER DATABASE recover managed standby database using current logfile disconnectWed Mar 9 15:32:55 2016Attempt to start background Managed Standby Recovery process (ORCL1)MRP0 started with pid=30, OS id=9267Wed Mar 9 15:32:55 2016MRP0: Background Managed Standby Recovery process started (ORCL1)Wed Mar 9 15:33:00 2016Managed Standby Recovery starting Real Time ApplyWed Mar 9 15:33:00 2016MRP0: Some datafile enqueues are still held! Retry recovery... Wed Mar 9 15:33:00 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:ORA-01124: Message 1124 not found; No message file for product=RDBMS, facility=ORA; arguments: [1]ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [1] [+DATA/orcl/system01.dbf]Wed Mar 9 15:33:00 2016Managed Standby Recovery not using Real Time ApplyWed Mar 9 15:33:00 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:ORA-01124: Message 1124 not found; No message file for product=RDBMS, facility=ORA; arguments: [1]ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [1] [+DATA/orcl/system01.dbf]Wed Mar 9 15:33:20 2016MRP0: Background Media Recovery terminated with error 1153Wed Mar 9 15:33:20 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:ORA-01153: Message 1153 not found; No message file for product=RDBMS, facility=ORAWed Mar 9 15:33:20 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:

Let's talk about the environment here.

The primary repository is 10g 2 nodes RAC, and the backup repository is also 2 nodes 10g RAC, but only node 1 does log recovery.

The accident occurred:

Yesterday, due to storage failure of the backup library (raid, no abnormal data), crs was stopped for maintenance by storage engineers. After maintenance, the new disk needs to synchronize data, so dg was started today, with an interval of about 24 hours.

The requested URL/etc/init.d/init.crs was not found on this server.

Everyone knows that this command will start instances of asm and db, but I ignored it

After node 1 gets up, look at the log, the database is open to readonly, then shutdown is executed, and mount is started to execute revoer, but here an error is reported, the information is as above.

In fact, it is now clear that node 1 starts the MRP process to prepare for recovery, but since node 2 is also in readonly mode, the data file has been occupied. So node 1's MRP can't recover the data file. was shut down

Solution:

shutdown 2 node db