A-A+
RAC_LOACL_BACKUPSET_VALIDATE_BUG ORA-19660 ORA-19661
问题描述：
在11gr2的2个节点的rac环境中，无论在哪个节点只要把备份片指到本地目录下进行备份，备份过程不会报错，但是验证时就会报错。
报错的相关文件我用dbv检查时是没有坏块的。
我把备份集默认在+ASM里的时候，就不会报错。
发现的过程：
本来想在原有的rac+dg的环境上延生出一个OGG的库进行灾备测试，其他ogg的参数都配好了，一开始备份主库就卡壳了，于是就纠结rman了一天，于是就有了这篇文档。
结论：oracle未知的BUG
相关环境：
[oracle@11grac1 tmp]$ uname -a Linux 11grac1 2.6.32-300.10.1.el5uek #1 SMP Wed Feb 22 17:22:40 EST 2012 i686 i686 i386 GNU/Linux
ORACLE信息
[oracle@11grac1 OPatch]$ ./opatch lsinventory -oh /u01/app/oracle/product/11.2.0/db_1 Invoking OPatch 11.1.0.6.6 Oracle Interim Patch Installer version 11.1.0.6.6 Copyright (c) 2009, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/11.2.0/db_1 Central Inventory : /u01/app/oraInventory from : /etc/oraInst.loc OPatch version : 11.1.0.6.6 OUI version : 11.2.0.1.0 OUI location : /u01/app/oracle/product/11.2.0/db_1/oui Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2013-05-06_15-40-42PM.log Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2013-05-06_15-40-42PM.txt -------------------------------------------------------------------------------- Installed Top-level Products (1): Oracle Database 11g 11.2.0.1.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. Rac system comprising of multiple nodes Local node = 11grac1 Remote node = 11grac2 -------------------------------------------------------------------------------- OPatch succeeded.
CRS信息
[grid@11grac1 OPatch]$ ./opatch lsinventory -oh /u01/11.2.0/grid Invoking OPatch 11.1.0.6.6 Oracle Interim Patch Installer version 11.1.0.6.6 Copyright (c) 2009, Oracle Corporation. All rights reserved. Oracle Home : /u01/11.2.0/grid Central Inventory : /u01/app/oraInventory from : /etc/oraInst.loc OPatch version : 11.1.0.6.6 OUI version : 11.2.0.1.0 OUI location : [grid@11grac1 OPatch]$ ./opatch lsinventory -oh /u01/11.2.0/grid Invoking OPatch 11.1.0.6.6 Oracle Interim Patch Installer version 11.1.0.6.6 Copyright (c) 2009, Oracle Corporation. All rights reserved. Oracle Home : /u01/11.2.0/grid Central Inventory : /u01/app/oraInventory from : /etc/oraInst.loc OPatch version : 11.1.0.6.6 OUI version : 11.2.0.1.0 OUI location : /u01/11.2.0/grid/oui Log file location : /u01/11.2.0/grid/cfgtoollogs/opatch/opatch2013-05-06_15-42-29PM.log Patch history file: /u01/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt Lsinventory Output file location : /u01/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2013-05-06_15-42-29PM.txt -------------------------------------------------------------------------------- Installed Top-level Products (1): Oracle Grid Infrastructure 11.2.0.1.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. Rac system comprising of multiple nodes Local node = 11grac1 Remote node = 11grac2 -------------------------------------------------------------------------------- OPatch succeeded.
过程如下：
首先默认备份到ASM中，验证是没有问题的。
RMAN> show all; using target database control file instead of recovery catalog RMAN configuration parameters for database with db_unique_name RACDB are: CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default CONFIGURE BACKUP OPTIMIZATION OFF; # default CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE MAXSETSIZE TO UNLIMITED; # default CONFIGURE ENCRYPTION FOR DATABASE OFF; # default CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+data/snapcf_racdb1.f'; CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DATA/snapcf_racdb1.f'; RMAN> backup full database; Starting backup at 06-MAY-13 using channel ORA_DISK_1 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00001 name=+DATA/racdb/system01.dbf input datafile file number=00002 name=+DATA/racdb/sysaux01.dbf input datafile file number=00008 name=+DATA/racdb/ogg01.dbf input datafile file number=00009 name=+DATA/racdb/test01.dbf input datafile file number=00003 name=+DATA/racdb/undotbs01.dbf input datafile file number=00005 name=+DATA/racdb/undotbs02.dbf input datafile file number=00004 name=+DATA/racdb/users01.dbf input datafile file number=00006 name=+DATA/racdb/datafile/yallonking.432.812066619 input datafile file number=00007 name=+DATA/racdb/yallonking_2.dbf channel ORA_DISK_1: starting piece 1 at 06-MAY-13 channel ORA_DISK_1: finished piece 1 at 06-MAY-13 piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357 tag=TAG20130506T092236 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:07:26 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set including current control file in backup set including current SPFILE in backup set channel ORA_DISK_1: starting piece 1 at 06-MAY-13 channel ORA_DISK_1: finished piece 1 at 06-MAY-13 piece handle=+DATA/racdb/backupset/2013_05_06/ncsnf0_tag20130506t092236_0.502.814699819 tag=TAG20130506T092236 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:16 Finished backup at 06-MAY-13 RMAN> list backupset; List of Backup Sets =================== BS Key Type LV Size Device Type Elapsed Time Completion Time ------- ---- -- ---------- ----------- ------------ --------------- 144 Full 994.09M DISK 00:07:23 06-MAY-13 BP Key: 144 Status: AVAILABLE Compressed: NO Tag: TAG20130506T092236 Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357 List of Datafiles in backup set 144 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 1 Full 1586465 06-MAY-13 +DATA/racdb/system01.dbf 2 Full 1586465 06-MAY-13 +DATA/racdb/sysaux01.dbf 3 Full 1586465 06-MAY-13 +DATA/racdb/undotbs01.dbf 4 Full 1586465 06-MAY-13 +DATA/racdb/users01.dbf 5 Full 1586465 06-MAY-13 +DATA/racdb/undotbs02.dbf 6 Full 1586465 06-MAY-13 +DATA/racdb/datafile/yallonking.432.812066619 7 Full 1586465 06-MAY-13 +DATA/racdb/yallonking_2.dbf 8 Full 1586465 06-MAY-13 +DATA/racdb/ogg01.dbf 9 Full 1586465 06-MAY-13 +DATA/racdb/test01.dbf BS Key Type LV Size Device Type Elapsed Time Completion Time ------- ---- -- ---------- ----------- ------------ --------------- 145 Full 19.80M DISK 00:00:23 06-MAY-13 BP Key: 145 Status: AVAILABLE Compressed: NO Tag: TAG20130506T092236 Piece Name: +DATA/racdb/backupset/2013_05_06/ncsnf0_tag20130506t092236_0.502.814699819 SPFILE Included: Modification time: 06-MAY-13 SPFILE db_unique_name: RACDB Control File Included: Ckp SCN: 1586888 Ckp time: 06-MAY-13 RMAN> restore database validate; Starting restore at 06-MAY-13 using channel ORA_DISK_1 channel ORA_DISK_1: starting validation of datafile backup set channel ORA_DISK_1: reading from backup piece +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357 channel ORA_DISK_1: piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357 tag=TAG20130506T092236 channel ORA_DISK_1: restored backup piece 1 channel ORA_DISK_1: validation complete, elapsed time: 00:00:35 Finished restore at 06-MAY-13
但是把备份片指定到本地时，备份过程没有报错，但是验证会报错。
RMAN> run{
2> allocate channel a1 type disk;
3> allocate channel a2 type disk;
4> allocate channel a3 type disk;
5> backup full database filesperset 3 format '/tmp/dbfullbak_%u_%s_%p_%T.bak';
6> sql 'ALTER SYSTEM switch logfile';
7> sql 'ALTER SYSTEM ARCHIVE LOG CURRENT';
8> backup archivelog all format '/tmp/arc_%T_%s.bak';
9> backup current controlfile format '/tmp/ctl0_%T_%s.bak';
10> release channel a1;
11> release channel a2;
12> release channel a3;
13> }
using target database control file instead of recovery catalog
allocated channel: a1
channel a1: SID=66 instance=racdb2 device type=DISK
allocated channel: a2
channel a2: SID=40 instance=racdb2 device type=DISK
allocated channel: a3
channel a3: SID=44 instance=racdb2 device type=DISK
Starting backup at 06-MAY-13
channel a1: starting full datafile backup set
channel a1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/racdb/system01.dbf
input datafile file number=00006 name=+DATA/racdb/datafile/yallonking.432.812066619
input datafile file number=00007 name=+DATA/racdb/yallonking_2.dbf
channel a1: starting piece 1 at 06-MAY-13
channel a2: starting full datafile backup set
channel a2: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATA/racdb/sysaux01.dbf
input datafile file number=00005 name=+DATA/racdb/undotbs02.dbf
input datafile file number=00004 name=+DATA/racdb/users01.dbf
... ...
Finished backup at 06-MAY-13
released channel: a1
released channel: a2
released channel: a3
RMAN> restore database validate;
Starting restore at 06-MAY-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=66 instance=racdb2 device type=DISK
channel ORA_DISK_1: starting validation of datafile backup set
channel ORA_DISK_1: reading from backup piece /tmp/dbfullbak_5do8unkp_173_1_20130506.bak
channel ORA_DISK_1: piece handle=/tmp/dbfullbak_5do8unkp_173_1_20130506.bak tag=TAG20130506T101029
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 05/06/2013 10:17:31
ORA-19660: some files in the backup set could not be verified
ORA-19661: datafile 3 could not be verified
ORA-19661: datafile 9 could not be verified
ORA-19661: datafile 8 could not be verified
RMAN> exit
Recovery Manager complete.
[oracle@11grac2 tmp]$ su -
Password:
[root@11grac2 ~]# mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@11grac2 ~]# df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda3 20638 11761 7812 61% /
/dev/sda1 190 42 140 23% /boot
tmpfs 1007 154 853 16% /dev/shm
[root@11grac2 ~]# ls -l /tmp/dbfullbak*
-rw-r----- 1 oracle asmadmin 614154240 May 6 10:14 /tmp/dbfullbak_5bo8unkm_171_1_20130506.bak
-rw-r----- 1 oracle asmadmin 422526976 May 6 10:13 /tmp/dbfullbak_5co8unkn_172_1_20130506.bak
-rw-r----- 1 oracle asmadmin 6012928 May 6 10:11 /tmp/dbfullbak_5do8unkp_173_1_20130506.bak
-rw-r----- 1 oracle asmadmin 20742144 May 6 10:12 /tmp/dbfullbak_5eo8unmn_174_1_20130506.bak
-rw-r----- 1 oracle asmadmin 98304 May 6 10:13 /tmp/dbfullbak_5fo8unpn_175_1_20130506.bak
单独对问题文件进行对比（ASM和LOACL）检查，文件是没有问题的。
RMAN> backup database datafile 3,8,9;
Starting backup at 06-MAY-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/racdb/system01.dbf
input datafile file number=00002 name=+DATA/racdb/sysaux01.dbf
input datafile file number=00008 name=+DATA/racdb/ogg01.dbf
input datafile file number=00009 name=+DATA/racdb/test01.dbf
input datafile file number=00003 name=+DATA/racdb/undotbs01.dbf
input datafile file number=00005 name=+DATA/racdb/undotbs02.dbf
input datafile file number=00004 name=+DATA/racdb/users01.dbf
input datafile file number=00006 name=+DATA/racdb/datafile/yallonking.432.812066619
input datafile file number=00007 name=+DATA/racdb/yallonking_2.dbf
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.501.814700017 tag=TAG20130506T093334 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:03:05
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00008 name=+DATA/racdb/ogg01.dbf
input datafile file number=00009 name=+DATA/racdb/test01.dbf
input datafile file number=00003 name=+DATA/racdb/undotbs01.dbf
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.500.814700201 tag=TAG20130506T093334 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/ncsnf0_tag20130506t093334_0.499.814700223 tag=TAG20130506T093334 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:08
Finished backup at 06-MAY-13
RMAN> exit
Recovery Manager complete.
[oracle@11grac1 tmp]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Mon May 6 09:37:31 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: RACDB (DBID=790139163)
RMAN> list backupset of datafile 3,8,9;
using target database control file instead of recovery catalog
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
144 Full 994.09M DISK 00:07:23 06-MAY-13
BP Key: 144 Status: AVAILABLE Compressed: NO Tag: TAG20130506T092236
Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357
List of Datafiles in backup set 144
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
3 Full 1586465 06-MAY-13 +DATA/racdb/undotbs01.dbf
8 Full 1586465 06-MAY-13 +DATA/racdb/ogg01.dbf
9 Full 1586465 06-MAY-13 +DATA/racdb/test01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
146 Full 994.12M DISK 00:03:00 06-MAY-13
BP Key: 146 Status: AVAILABLE Compressed: NO Tag: TAG20130506T093334
Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.501.814700017
List of Datafiles in backup set 146
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
3 Full 1587185 06-MAY-13 +DATA/racdb/undotbs01.dbf
8 Full 1587185 06-MAY-13 +DATA/racdb/ogg01.dbf
9 Full 1587185 06-MAY-13 +DATA/racdb/test01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
147 Full 6.09M DISK 00:00:05 06-MAY-13
BP Key: 147 Status: AVAILABLE Compressed: NO Tag: TAG20130506T093334
Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.500.814700201
List of Datafiles in backup set 147
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
3 Full 1587393 06-MAY-13 +DATA/racdb/undotbs01.dbf
8 Full 1587393 06-MAY-13 +DATA/racdb/ogg01.dbf
9 Full 1587393 06-MAY-13 +DATA/racdb/test01.dbf
RMAN> exit
Recovery Manager complete.
[oracle@11grac1 tmp]$ dbv file=+DATA/racdb/undotbs01.dbf userid=sys/oracle
DBVERIFY: Release 11.2.0.1.0 - Production on Mon May 6 09:38:44 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = +DATA/racdb/undotbs01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 4480
Total Pages Processed (Data) : 0
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing (Index): 0
Total Pages Processed (Other): 4479
Total Pages Processed (Seg) : 10
Total Pages Failing (Seg) : 0
Total Pages Empty : 1
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 0 (0.0)
[oracle@11grac1 tmp]$ dbv file=+DATA/racdb/ogg01.dbf userid=sys/oracle
DBVERIFY: Release 11.2.0.1.0 - Production on Mon May 6 09:39:23 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = +DATA/racdb/ogg01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 6400
Total Pages Processed (Data) : 36
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 13
Total Pages Failing (Index): 0
Total Pages Processed (Other): 178
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 6173
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 0 (0.0)
[oracle@11grac1 tmp]$ dbv file=+DATA/racdb/test01.dbf userid=sys/oracle
DBVERIFY: Release 11.2.0.1.0 - Production on Mon May 6 09:39:35 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = +DATA/racdb/test01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 6400
Total Pages Processed (Data) : 0
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing (Index): 0
Total Pages Processed (Other): 127
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 6273
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 0 (0.0)
结论：
和oracle官方前后沟通了近1天左右，也一共换了2个oracle工程师，最后的结论是：
——————————————————————————————————————————————————————————————————————————————
Dear customer,
感谢您的配合。
我将稍后整理一下问题，并提交新的BUG给研发部门。
如果有进一步的研发部门的回复，我将更新SR通知您。
感谢您的配合。
PS：
如果真的是oracle的未知bug，那俺也算是给oracle帮忙了，靠着oracle吃饭到现在，算是对oracle的一点点回报吧.
要是在13gc（Global cloud,自己猜的）版本中或者什么补丁里修补的话，以我的名字命名该bug就爽了..^_^，咱也是在oracle留名的淫了..^_^...
虽然每天在oracle软件园的斜对面上班... ...
后来在11.2.0.3版本中进行了测试，没有发现这个问题。
操作系统是x64