RMAN-00600: internal error, arguments [7038] [rec_cl_delarc]

Error:

{

   recover

   clone database

   noredo

    delete archivelog

   ;

}

executing Memory Script

Starting recover at 03-DEC-2017 21:20:54

RMAN-00571: ===============================================

RMAN-00569: ======== ERROR MESSAGE STACK FOLLOWS ==========

RMAN-00571: ===============================================

RMAN-03002: failure of Duplicate Db command at 12/03/2017 21:21:05

RMAN-05501: aborting duplication of target database

RMAN-00600: internal error, arguments [7038] [rec_cl_delarc][] [] []

Solution:

Source: LABPRD

Target : LABTEST

1. Take backup of pfile from spfile

SQL>create pfile=initLABTEST1_clone.ora from spfile;

2. optionally you may take current control file backup.

RMAN> backup as copy current controlfile format '/tmp/ctlbkpLABTPRD';

Starting backup at 04-DEC-17

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

channel ORA_DISK_1: starting datafile copy

copying current control file

output file name=/tmp/ctlbkpLABTPRD tag=TAG20171204T030618 RECID=1305 STAMP=961815980

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03

Finished backup at 04-DEC-17

Starting Control File Autobackup at 04-DEC-17

piece handle=/u01/app/oracle/product/11.2.0.2./dbs/c-3601421392-20171204-00 comment=NONE

Finished Control File Autobackup at 04-DEC-17

RMAN> exit

3. shut down database instance , make sure all other RAC instances are down

4. Update db_name as LABPRD (earlier LABTEST) in pfile created in step 1. 

This steps is to generate the control file command from trace , and need to be reverted back before starting up the instance post control file backup else it may generate following error

CREATE CONTROLFILE SET DATABASE "LABTEST" RESETLOGS FORCE LOGGING ARCHIVELOG

*

ERROR at line 1:

ORA-01503: CREATE CONTROLFILE failed

ORA-01504: database name 'LABTEST' does not match parameter db_name 'LABTPRD'

5. startup db in nomount using pfile

SQL> startup nomount pfile=initLABTEST1_clone.ora

ORA-32006: RESOURCE_MANAGER_CPU_ALLOCATION initialization parameter has been deprecated

ORACLE instance started.

Total System Global Area 3206836224 bytes

Fixed Size                  2257520 bytes

Variable Size             754978192 bytes

Database Buffers         2432696320 bytes

Redo Buffers               16904192 bytes

6.Take backup of control file to trace , will generate a trace in user dump destination

SQL> show parameter db_name

NAME                        TYPE        VALUE

-------------------------- ----------- ------------------------

db_name                              string      LABTPRD

SQL> show parameter db_uniq

NAME                        TYPE        VALUE

-------------------------- ----------- ------------------------

db_unique_name                       string      LABTEST

SQL> alter database backup controlfile to trace;

Database altered.

7. Prepare control file command from trace file generated in step 6

Make sure you use "SET" and target db_name as highlighted

CREATE CONTROLFILE SET DATABASE "LABTEST" RESETLOGS FORCE LOGGING ARCHIVELOG

..

..

8. revert back db_name as LABTEST (earlier LABPRD) in pfile created in step 1.  And startup db in nomount

SQL> startup nomount pfile=initLABTEST1_clone.ora

ORA-32006: RESOURCE_MANAGER_CPU_ALLOCATION initialization parameter has been deprecated

ORACLE instance started.

Total System Global Area 3206836224 bytes

Fixed Size                  2257520 bytes

Variable Size             754978192 bytes

Database Buffers         2432696320 bytes

Redo Buffers               16904192 bytes

SQL> show parameter db_name

NAME        TYPE          VALUE

--------- ------------ ------------------

db_name    string       LABTEST

SQL> show parameter db_uniq

NAME        TYPE         VALUE

--------- -----------  ------------------

db_unique_name   string  LABTEST         

9. create control file 

SQL> CREATE CONTROLFILE SET DATABASE "LABTEST" RESETLOGS FORCE LOGGING ARCHIVELOG

  2      MAXLOGFILES 32

  3      MAXLOGMEMBERS 5

  4      MAXDATAFILES 512

..

..

 77  CHARACTER SET AL32UTF8

 78  ;

Control file created.




10. Check datafile header status to check if any fuzzy file and need recovery



SQL>set line 900

SQL>set pages 900

SQL>set echo on

SQL>alter session set nls_date_format='dd-mon-yyyy hh24:mi:ss';

SQL>set numwidth 13

SQL>col CHECKPOINT_CHANGE# for 9999999999999999

SQL>col CHANGE# for 9999999999999999

SQL>column error format a10

SQL>column member format a60

SQL>show parameter db_recover

SQL>select file#, checkpoint_change#, checkpoint_time, error,fuzzy,recover from v$datafile_header;

FILE# CHECKPOINT_CHANGE# CHECKPOINT_TIME   ERROR   FUZ REC

----- -------------- ----------------- ---------- --- ---

1     10941273541716 03-dec-2017 21:01:50         YES

2     10941273541716 03-dec-2017 21:01:50         YES

3     10941273541716 03-dec-2017 21:01:50         NO

4     10941273541716 03-dec-2017 21:01:50         NO

5     10941273541716 03-dec-2017 21:01:50         NO

6     10941273541716 03-dec-2017 21:01:50         NO

..

..

11. if datafile status is fuzzy or checkpoint varies perform database recovery

SQL> recover database using backup controlfile until cancel;

ORA-00279: change 10941273541716 generated at 12/03/2017 21:01:50 needed for thread 2

ORA-00289: suggestion : +RECO/LABTEST/2_33685_806276183.dbf

ORA-00280: change 10941273541716 for thread 2 is in sequence #33685

Specify log: {=suggested | filename | AUTO | CANCEL}

+RECO/LABTEST/2_33685_806276183.dbf

ORA-00308: cannot open archived log '+RECO/LABTEST/2_33685_806276183.dbf'

ORA-17503: ksfdopn:2 Failed to open file +RECO/LABTEST/2_33685_806276183.dbf

ORA-15173: entry '2_33685_806276183.dbf' does not exist in directory 'LABTEST'

Specify log: {=suggested | filename | AUTO | CANCEL}

CANCEL

ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below

ORA-01194: file 1 needs more recovery to be consistent

ORA-01110: data file 1: '+DATA/LABTEST/datafile/system.7689.961795207'

ORA-01112: media recovery not started

On above error do not proceed with "open resetlogs" as it need more archive logs for recovery, if you "open resetlogs" db then it may result in corruption and need to re-restore complete database

SQL>  recover database using backup controlfile until cancel;
ORA-00279: change 10941273541716 generated at 12/03/2017 21:01:50 needed for
thread 2
ORA-00289: suggestion : +RECO/LABTEST/2_33685_806276183.dbf
ORA-00280: change 10941273541716 for thread 2 is in sequence #33685

Specify log: {=suggested | filename | AUTO | CANCEL}
/u09/arch_backup/archarch_D-LABTPRD_id-3601421392_S-33685_T-2_A-806276183_rgsl8asa
ORA-00279: change 10941273541716 generated at  needed for thread 1

Specify log: {=suggested | filename | AUTO | CANCEL}
/u09/arch_backup/archarch_D-LABTPRD_id-3601421392_S-61201_T-1_A-806276183_rqsl8b74
ORA-00279: change 10941273541751 generated at 12/03/2017 21:01:56 needed for
thread 1
ORA-00289: suggestion : +RECO/LABTEST/1_61202_806276183.dbf
ORA-00280: change 10941273541751 for thread 1 is in sequence #61202
ORA-00278: log file
'/u09/arch_backup/archarch_D-LABTPRD_id-3601421392_S-61201_T-1_A-806276183_rqsl
8b74' no longer needed for this recovery

Specify log: {=suggested | filename | AUTO | CANCEL}
/u09/arch_backup/archarch_D-LABTPRD_id-3601421392_S-61202_T-1_A-806276183_risl8b37
ORA-00279: change 10941357019367 generated at 12/03/2017 23:07:41 needed for
thread 1
ORA-00289: suggestion : +RECO/LABTEST/1_61203_806276183.dbf
ORA-00280: change 10941357019367 for thread 1 is in sequence #61203
ORA-00278: log file
'/u09/arch_backup/archarch_D-LABTPRD_id-3601421392_S-61202_T-1_A-806276183_risl
8b37' no longer needed for this recovery

Specify log: {=suggested | filename | AUTO | CANCEL}
CANCEL

Media recovery cancelled.

12. Check datafile header status and open database with resetlogs

SQL>select file#, checkpoint_change#, checkpoint_time, error,fuzzy,recover from v$datafile_header;

FILE# CHECKPOINT_CHANGE# CHECKPOINT_TIME  ERROR    FUZ REC
----- ------------------ --------------- ---------- --- ---
 1     10941357019367 03-dec-2017 23:07:41          NO
 2     10941357019367 03-dec-2017 23:07:41          NO
 3     10941357019367 03-dec-2017 23:07:41          NO
 4     10941357019367 03-dec-2017 23:07:41          NO
 5     10941357019367 03-dec-2017 23:07:41          NO



 6     10941357019367 03-dec-2017 23:07:41          NO
.
.

SQL> select status, checkpoint_change#, fuzzy,
  2  to_char(checkpoint_time, 'DD-MON-YYYY HH24:MI:SS') as checkpoint_time,
  3  count(*)
  4  from v$datafile_header
  5  group by status, checkpoint_change#, fuzzy, checkpoint_time
  6  order by status, checkpoint_change#, fuzzy, checkpoint_time;

STATUS   CHECKPOINT_CHANGE# FUZ CHECKPOINT_TIME        COUNT(*)
------ ----------------     --- -------------------- -----------

ONLINE   10941357019367     NO  03-DEC-2017 23:07:41    53

SQL> alter database open resetlogs;

Database altered.