A-A+

RAC_LOACL_BACKUPSET_VALIDATE_BUG ORA-19660 ORA-19661

2013年05月06日 DG&RAC&OGG, TroubleShooting 评论 2 条 阅读 2,994 次

问题描述:
在11gr2的2个节点的rac环境中,无论在哪个节点只要把备份片指到本地目录下进行备份,备份过程不会报错,但是验证时就会报错。
报错的相关文件我用dbv检查时是没有坏块的。
我把备份集默认在+ASM里的时候,就不会报错。

发现的过程:
本来想在原有的rac+dg的环境上延生出一个OGG的库进行灾备测试,其他ogg的参数都配好了,一开始备份主库就卡壳了,于是就纠结rman了一天,于是就有了这篇文档。

结论:oracle未知的BUG

相关环境:

[oracle@11grac1 tmp]$ uname -a
Linux 11grac1 2.6.32-300.10.1.el5uek #1 SMP Wed Feb 22 17:22:40 EST 2012 i686 i686 i386 GNU/Linux

ORACLE信息

[oracle@11grac1 OPatch]$ ./opatch lsinventory -oh /u01/app/oracle/product/11.2.0/db_1
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0/db_1/oui
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2013-05-06_15-40-42PM.log

Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2013-05-06_15-40-42PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = 11grac1
  Remote node = 11grac2

--------------------------------------------------------------------------------

OPatch succeeded.

CRS信息

[grid@11grac1 OPatch]$ ./opatch lsinventory -oh /u01/11.2.0/grid
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      :

[grid@11grac1 OPatch]$ ./opatch lsinventory -oh /u01/11.2.0/grid
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/11.2.0/grid/oui
Log file location : /u01/11.2.0/grid/cfgtoollogs/opatch/opatch2013-05-06_15-42-29PM.log

Patch history file: /u01/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2013-05-06_15-42-29PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


Rac system comprising of multiple nodes
  Local node = 11grac1
  Remote node = 11grac2

--------------------------------------------------------------------------------

OPatch succeeded.

过程如下:
首先默认备份到ASM中,验证是没有问题的。

RMAN> show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name RACDB are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+data/snapcf_racdb1.f';
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DATA/snapcf_racdb1.f';

RMAN> backup full database;

Starting backup at 06-MAY-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/racdb/system01.dbf
input datafile file number=00002 name=+DATA/racdb/sysaux01.dbf
input datafile file number=00008 name=+DATA/racdb/ogg01.dbf
input datafile file number=00009 name=+DATA/racdb/test01.dbf
input datafile file number=00003 name=+DATA/racdb/undotbs01.dbf
input datafile file number=00005 name=+DATA/racdb/undotbs02.dbf
input datafile file number=00004 name=+DATA/racdb/users01.dbf
input datafile file number=00006 name=+DATA/racdb/datafile/yallonking.432.812066619
input datafile file number=00007 name=+DATA/racdb/yallonking_2.dbf
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357 tag=TAG20130506T092236 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:07:26
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/ncsnf0_tag20130506t092236_0.502.814699819 tag=TAG20130506T092236 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:16
Finished backup at 06-MAY-13

RMAN> list backupset;


List of Backup Sets
===================


BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
144 Full 994.09M DISK 00:07:23 06-MAY-13
BP Key: 144 Status: AVAILABLE Compressed: NO Tag: TAG20130506T092236
Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357
List of Datafiles in backup set 144
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 Full 1586465 06-MAY-13 +DATA/racdb/system01.dbf
2 Full 1586465 06-MAY-13 +DATA/racdb/sysaux01.dbf
3 Full 1586465 06-MAY-13 +DATA/racdb/undotbs01.dbf
4 Full 1586465 06-MAY-13 +DATA/racdb/users01.dbf
5 Full 1586465 06-MAY-13 +DATA/racdb/undotbs02.dbf
6 Full 1586465 06-MAY-13 +DATA/racdb/datafile/yallonking.432.812066619
7 Full 1586465 06-MAY-13 +DATA/racdb/yallonking_2.dbf
8 Full 1586465 06-MAY-13 +DATA/racdb/ogg01.dbf
9 Full 1586465 06-MAY-13 +DATA/racdb/test01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
145 Full 19.80M DISK 00:00:23 06-MAY-13
BP Key: 145 Status: AVAILABLE Compressed: NO Tag: TAG20130506T092236
Piece Name: +DATA/racdb/backupset/2013_05_06/ncsnf0_tag20130506t092236_0.502.814699819
SPFILE Included: Modification time: 06-MAY-13
SPFILE db_unique_name: RACDB
Control File Included: Ckp SCN: 1586888 Ckp time: 06-MAY-13

RMAN> restore database validate;

Starting restore at 06-MAY-13
using channel ORA_DISK_1

channel ORA_DISK_1: starting validation of datafile backup set
channel ORA_DISK_1: reading from backup piece +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357
channel ORA_DISK_1: piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357 tag=TAG20130506T092236
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:35
Finished restore at 06-MAY-13

但是把备份片指定到本地时,备份过程没有报错,但是验证会报错。

RMAN> run{
2> allocate channel a1 type disk;
3> allocate channel a2 type disk;
4> allocate channel a3 type disk;
5> backup full database filesperset 3 format '/tmp/dbfullbak_%u_%s_%p_%T.bak';
6> sql 'ALTER SYSTEM switch logfile';
7> sql 'ALTER SYSTEM ARCHIVE LOG CURRENT';
8> backup archivelog all format '/tmp/arc_%T_%s.bak';
9> backup current controlfile format '/tmp/ctl0_%T_%s.bak';
10> release channel a1;
11> release channel a2;
12> release channel a3;
13> }

using target database control file instead of recovery catalog
allocated channel: a1
channel a1: SID=66 instance=racdb2 device type=DISK

allocated channel: a2
channel a2: SID=40 instance=racdb2 device type=DISK

allocated channel: a3
channel a3: SID=44 instance=racdb2 device type=DISK

Starting backup at 06-MAY-13
channel a1: starting full datafile backup set
channel a1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/racdb/system01.dbf
input datafile file number=00006 name=+DATA/racdb/datafile/yallonking.432.812066619
input datafile file number=00007 name=+DATA/racdb/yallonking_2.dbf
channel a1: starting piece 1 at 06-MAY-13
channel a2: starting full datafile backup set
channel a2: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATA/racdb/sysaux01.dbf
input datafile file number=00005 name=+DATA/racdb/undotbs02.dbf
input datafile file number=00004 name=+DATA/racdb/users01.dbf
... ...
Finished backup at 06-MAY-13

released channel: a1

released channel: a2

released channel: a3

RMAN> restore database validate;

Starting restore at 06-MAY-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=66 instance=racdb2 device type=DISK

channel ORA_DISK_1: starting validation of datafile backup set
channel ORA_DISK_1: reading from backup piece /tmp/dbfullbak_5do8unkp_173_1_20130506.bak
channel ORA_DISK_1: piece handle=/tmp/dbfullbak_5do8unkp_173_1_20130506.bak tag=TAG20130506T101029
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 05/06/2013 10:17:31
ORA-19660: some files in the backup set could not be verified
ORA-19661: datafile 3 could not be verified
ORA-19661: datafile 9 could not be verified
ORA-19661: datafile 8 could not be verified

RMAN> exit


Recovery Manager complete.
[oracle@11grac2 tmp]$ su -
Password:
[root@11grac2 ~]# mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@11grac2 ~]# df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda3 20638 11761 7812 61% /
/dev/sda1 190 42 140 23% /boot
tmpfs 1007 154 853 16% /dev/shm
[root@11grac2 ~]# ls -l /tmp/dbfullbak*
-rw-r----- 1 oracle asmadmin 614154240 May 6 10:14 /tmp/dbfullbak_5bo8unkm_171_1_20130506.bak
-rw-r----- 1 oracle asmadmin 422526976 May 6 10:13 /tmp/dbfullbak_5co8unkn_172_1_20130506.bak
-rw-r----- 1 oracle asmadmin 6012928 May 6 10:11 /tmp/dbfullbak_5do8unkp_173_1_20130506.bak
-rw-r----- 1 oracle asmadmin 20742144 May 6 10:12 /tmp/dbfullbak_5eo8unmn_174_1_20130506.bak
-rw-r----- 1 oracle asmadmin 98304 May 6 10:13 /tmp/dbfullbak_5fo8unpn_175_1_20130506.bak

单独对问题文件进行对比(ASM和LOACL)检查,文件是没有问题的。

RMAN> backup database datafile 3,8,9;

Starting backup at 06-MAY-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/racdb/system01.dbf
input datafile file number=00002 name=+DATA/racdb/sysaux01.dbf
input datafile file number=00008 name=+DATA/racdb/ogg01.dbf
input datafile file number=00009 name=+DATA/racdb/test01.dbf
input datafile file number=00003 name=+DATA/racdb/undotbs01.dbf
input datafile file number=00005 name=+DATA/racdb/undotbs02.dbf
input datafile file number=00004 name=+DATA/racdb/users01.dbf
input datafile file number=00006 name=+DATA/racdb/datafile/yallonking.432.812066619
input datafile file number=00007 name=+DATA/racdb/yallonking_2.dbf
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.501.814700017 tag=TAG20130506T093334 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:03:05
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00008 name=+DATA/racdb/ogg01.dbf
input datafile file number=00009 name=+DATA/racdb/test01.dbf
input datafile file number=00003 name=+DATA/racdb/undotbs01.dbf
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.500.814700201 tag=TAG20130506T093334 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 06-MAY-13
channel ORA_DISK_1: finished piece 1 at 06-MAY-13
piece handle=+DATA/racdb/backupset/2013_05_06/ncsnf0_tag20130506t093334_0.499.814700223 tag=TAG20130506T093334 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:08
Finished backup at 06-MAY-13

RMAN> exit


Recovery Manager complete.
[oracle@11grac1 tmp]$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Mon May 6 09:37:31 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RACDB (DBID=790139163)

RMAN> list backupset of datafile 3,8,9;

using target database control file instead of recovery catalog

List of Backup Sets
===================


BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
144     Full    994.09M    DISK        00:07:23     06-MAY-13      
        BP Key: 144   Status: AVAILABLE  Compressed: NO  Tag: TAG20130506T092236
        Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t092236_0.506.814699357
  List of Datafiles in backup set 144
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  3       Full 1586465    06-MAY-13 +DATA/racdb/undotbs01.dbf
  8       Full 1586465    06-MAY-13 +DATA/racdb/ogg01.dbf
  9       Full 1586465    06-MAY-13 +DATA/racdb/test01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
146     Full    994.12M    DISK        00:03:00     06-MAY-13      
        BP Key: 146   Status: AVAILABLE  Compressed: NO  Tag: TAG20130506T093334
        Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.501.814700017
  List of Datafiles in backup set 146
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  3       Full 1587185    06-MAY-13 +DATA/racdb/undotbs01.dbf
  8       Full 1587185    06-MAY-13 +DATA/racdb/ogg01.dbf
  9       Full 1587185    06-MAY-13 +DATA/racdb/test01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
147     Full    6.09M      DISK        00:00:05     06-MAY-13      
        BP Key: 147   Status: AVAILABLE  Compressed: NO  Tag: TAG20130506T093334
        Piece Name: +DATA/racdb/backupset/2013_05_06/nnndf0_tag20130506t093334_0.500.814700201
  List of Datafiles in backup set 147
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  3       Full 1587393    06-MAY-13 +DATA/racdb/undotbs01.dbf
  8       Full 1587393    06-MAY-13 +DATA/racdb/ogg01.dbf
  9       Full 1587393    06-MAY-13 +DATA/racdb/test01.dbf

RMAN> exit


Recovery Manager complete.
[oracle@11grac1 tmp]$ dbv file=+DATA/racdb/undotbs01.dbf userid=sys/oracle

DBVERIFY: Release 11.2.0.1.0 - Production on Mon May 6 09:38:44 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = +DATA/racdb/undotbs01.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4480
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 4479
Total Pages Processed (Seg)  : 10
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 1
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)
[oracle@11grac1 tmp]$ dbv file=+DATA/racdb/ogg01.dbf userid=sys/oracle

DBVERIFY: Release 11.2.0.1.0 - Production on Mon May 6 09:39:23 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = +DATA/racdb/ogg01.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 6400
Total Pages Processed (Data) : 36
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 13
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 178
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 6173
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)
[oracle@11grac1 tmp]$ dbv file=+DATA/racdb/test01.dbf userid=sys/oracle

DBVERIFY: Release 11.2.0.1.0 - Production on Mon May 6 09:39:35 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = +DATA/racdb/test01.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 6400
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 127
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 6273
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)

结论:
和oracle官方前后沟通了近1天左右,也一共换了2个oracle工程师,最后的结论是:
——————————————————————————————————————————————————————————————————————————————

Dear customer,

感谢您的配合。

我将稍后整理一下问题,并提交新的BUG给研发部门。

如果有进一步的研发部门的回复,我将更新SR通知您。

感谢您的配合。

PS:
如果真的是oracle的未知bug,那俺也算是给oracle帮忙了,靠着oracle吃饭到现在,算是对oracle的一点点回报吧.
要是在13gc(Global cloud,自己猜的)版本中或者什么补丁里修补的话,以我的名字命名该bug就爽了..^_^,咱也是在oracle留名的淫了..^_^...
虽然每天在oracle软件园的斜对面上班... ...

2 条留言  访客:0 条  博主:0 条

  1. Yallon King

    后来在11.2.0.3版本中进行了测试,没有发现这个问题。

给我留言

Copyright © YallonKing 保留所有权利.   Theme  Ality

用户登录

分享到: