Recover archive gaps in standby database from primary using RMAN incremental backups

A Physical Standby database synchs with Primary by continuous apply of archive logs from a Primary Database. In case of an archive log gone missing or corrupt, We have to bring back the standby to sync with the primary.

When the logs missing or corrupt is less in number (say below 15), we can ship the logs which were missing in the standby site from the primary site (scp/sftp/ftp) and then we can register the log file in the standby so that the gap can be resolved.

Find the archives which are missing by issuing the following command.

SQL> select * from v$archive_gap;

This would give the gap sequences. Or you can use the v$managed_standby view to find where the log apply stuck.

SQL> select sequence#,process,status from v$managed_standby;

Here u can see status as wait for log for say sequence# 100 but your primary would've proceeded to sequence# 110

At primary

SQL> select max(sequence#) from v$archived_log;

      The above command would show you 110

Copy the logs to the standby site from the primary site

$ scp log_file_name_n.arc oracle@standby:/log/file/location/log_file_name_n.arc

At standby site

SQL> alter database register logfile '/log/file/location/log_file_name_n.arc';

logfile registered

Do the log file registration at the standby site until all the missing log files are registered. Now apply would take place and your standby will become sync with the primary.

This is easy process if you have missing or corrupt logs in lesser number. But when the difference is huge (say around 500 logs) this method is very time consuming and not a proper approach. Else you have to rebuild the standby database from scratch.

As an enhancement from 10g, an incremental backup created with BACKUP INCREMENTAL... FROM SCN can be used to refresh the standby database with changes at the primary database since the last SCN at Standby and then managed recovery can resume i.e. Compensate for the missing archive logs.

Let us see the steps involved.

Step 1: On the primary:

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
144710998

On the standby:

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
130158742

Clearly there is a difference. But this by itself does not indicate a problem; since the standby is expected to lag behind the primary (this is an asynchronous non-real time apply setup). The real question is how much it is lagging in the terms of wall clock.

To know that use the scn_to_timestamp function to translate the SCN to a timestamp:

SQL> select scn_to_timestamp(144710998) from dual;

SCN_TO_TIMESTAMP(1447102)
-------------------------------
18-AUG-11 08.54.28.000000000 AM

Run the same query to know the timestamp associated with the SCN of the standby database as well

SQL> select scn_to_timestamp(130158742) from dual;

SCN_TO_TIMESTAMP(1301571)
-------------------------------
13-AUG-11 07.19.27.000000000 PM

Note: Run it on the primary database, since it will fail in the standby in a mounted mode

This shows that the standby is four and half days lagging!

Step 2: [Standby] Stop the managed standby apply process:

SQL> alter database recover managed standby database cancel;

Database altered.

Step 3: [Standby] Shutdown the standby database

SQL> shut immediate

Step 4: [Primary] On the primary, take an incremental backup from the SCN number where the standby has been stuck:

RMAN> run {
2> allocate channel c1 type disk format '/u01/backup/%U.bkp';
3> backup incremental from scn 130158740 database;
4> }

Step 5: [Primary] On the primary, create a new standby controlfile:

SQL> alter database create standby controlfile as '/u01/backup/for_standby.ctl';

Database altered.

or

RMAN> backup current controlfile for standby format '/backups/tempfol/_%U';

Step 6: [Primary] Copy these files to standby host:

oracle@dba1 /u01/backup]$ scp * oracle@dba2:/u01/backup

Step 7: [Standby] Bring up the instance in nomount mode:

SQL> startup nomount

Step 8: [Standby] Check the location of the controlfile:

SQL> show parameter control_files

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /u01/oradata/standby_cntfile.ctl

Step 9: [Standby] Replace the controlfile with the one you just created in primary.

$ cp /u01/backup/for_standby.ctl /u01/oradata/standby_cntfile.ctl

or

RMAN> RESTORE STANDBY CONTROLFILE FROM '/u01/backup/_v8o069ee_1_1';

Step 10: [Standby] Mount the standby database:

SQL> alter database mount standby database;

Step 11: [Standby] Connect to RMAN. RMAN does not know about these files yet; so you must let it know – by a process called cataloging. Catalog these files:

$ rman target=/

RMAN> catalog start with '/u01/backup';

Step 12: Recover these files:

RMAN> recover database;

Step 13: After some time, the recovery fails with the message:

archive log filename=/u01/oradata/1_18108_697108460.dbf thread=1 sequence=18109
ORA-00310: archived log contains sequence 18108; sequence 18109 required

This happens because we have come to the last of the archived logs. The expected archived log with sequence# 18108 has not been generated yet.

Step 14: At this point exit RMAN and start managed recovery process:

SQL> alter database recover managed standby database disconnect from session;

Database altered.

Step 15: Check the SCN’s in primary and standby:

[Standby]

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
144747125

[Primary]

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
144747111

Now they are very close to each other. The standby has now caught up.

Retrieve deleted rows from table using audit.

Firstly connect as scott from any other user.

Conn scott/tiger
select * from emp;
..
..
DELETE FROM EMP WHERE EMPNO = 7499;

Now lets retrieve this row by using audits and flashback.

I have mentioned using audits as the row may be accidentally deleted or even deleted by any malicious activity (some unauthorised person performing the delete operation)

If it's due to malicious activity, we can also find the culprit who performed the act.

Check what audit is recorded in the database.

NAME
----------------------------------------------------------------------
VALUE
----------------------------------------------------------------------
audit_sys_operations
FALSE
audit_file_dest
D:\ORACLE\PRODUCT\10.2.0\ADMIN\10gDB\ADUMP
audit_trail
DB

The audit is being done to the DB (Mostly DB audit would be done)

Now lets get some details from sys.aud$ table which is the table used for storing audit related things.

SELECT userid, action#, STATEMENT, OBJ$NAME, To_Char (timestamp#, 'mm/dd/yyyy hh24:mi:ss') 
FROM sys.aud$ ORDER BY timestamp# asc;

Output would look like this

Timeline from the database audit:

USERID	ACTION#	STATEMENT	OBJ$NAME	TIMESTAMP
SCOTT	101	1		04/30/2006 09:11:36
SCOTT	3	2	X$NLS_PARAMETERS	04/30/2006 09:29:07
SCOTT	3	2	GV$NLS_PARAMETERS	04/30/2006 09:29:07
SCOTT	3	2	V$NLS_PARAMETERS	04/30/2006 09:29:07
SCOTT	3	2	NLS_SESSION_PARAMETERS	04/30/2006 09:29:07
SCOTT	3	5	DUAL	04/30/2006 09:29:07
SCOTT	100	1		04/30/2006 09:29:41
SCOTT	3	22	OBJ$	04/30/2006 09:31:07
SCOTT	3	22	USER_OBJECTS	04/30/2006 09:31:07
SCOTT	3	28	EMP	04/30/2006 09:32:01
SCOTT	3	31	EMP	04/30/2006 09:32:20
SCOTT	7	37	EMP	04/30/2006 09:33:28
SCOTT	3	46	EMP	04/30/2006 09:35:24
SCOTT	7	52	EMP	04/30/2006 09:37:04
SCOTT	7	55	EMP	04/30/2006 09:37:13
SCOTT	3	61	EMP	04/30/2006 09:37:28

Now we have to find which action# refers to what operation. To find this issue the command below.

SELECT * FROM AUDIT_ACTIONS;

Now you can see action# 7 denotes delete operation.

Delete operation was performed around 9 37. So we want to flashback to before then so have to get the recorded timestamp. Oracle does not actually record a full timeline. Only takes the time every 5 minutes with the relevant SCN.

So now we will find the scn according to the timestamp from the below command which would be stored for maximum 5 days rolling figure.

SELECT To_Char(TIME_DP, 'dd/mm/yyyy hh24:mi:ss'), SCN_BAS 
FROM SYS.SMON_SCN_TIME order by scn_bas;

Now you can retrieve the deleted row by flashback the table to before the timestamp where the data is deleted.

CREATE TABLE EMPCOPY AS 
SELECT * FROM SCOTT.EMP 
AS OF TIMESTAMP (TO_TIMESTAMP('30/04/2006 09:31:10','DD-MM-YYYY:HH24:MI:SS'));

where 09:31:10 is the predecessor scn timestamp before the data deletion time of 09:37:13.

Now you can find the deleted row back in the EMPCOPY table.

Let's now find out did SCOTT do this deletion activity or any other malicious user did this on purpose to get SCOTT into trouble.

We can use the same sys.aud$ table to find this too..

SELECT userid, USERHOST, TERMINAL, SPARE1, action#, STATEMENT, OBJ$NAME, To_Char (timestamp#, 'mm/dd/yyyy hh24:mi:ss') 
FROM sys.aud$ ORDER BY timestamp# asc;

This additional data shows that SCOTT was coming from a different workstation from normal additionally the SPARE1 column shows that the Windows username was in fact SELVA and not SCOTT.

So take actions accordingly.