BANNER
——————————————————————————–
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
PL/SQL Release 11.2.0.3.0 – Production
CORE 11.2.0.3.0 Production
TNS for Linux: Version 11.2.0.3.0 – Production
NLSRTL Version 11.2.0.3.0 – Production
SQL> alter diskgroup datadg mount;
alter diskgroup datadg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk “5” is missing from group number “1”
ERROR: alter diskgroup datadg mount
Wed Mar 13 07:42:03 2013
SQL> alter diskgroup datadg mount
NOTE: cache registered group DATADG number=1 incarn=0xccb845cd
NOTE: cache began mount (first) of group DATADG number=1 incarn=0xccb845cd
NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)
NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)
NOTE: Assigning number (1,0) to disk (/dev/asm-diske)
Wed Mar 13 16:42:09 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 20 for pid 27, osid 5439
NOTE: Assigning number (1,5) to disk ()
GMON querying group 1 at 21 for pid 27, osid 5439
NOTE: cache dismounting (clean) group 1/0xCCB845CD (DATADG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 5439, image: oracle@vmac1 (TNS V1-V3)
NOTE: dbwr not being msg’d to dismount
NOTE: lgwr not being msg’d to dismount
NOTE: cache dismounted group 1/0xCCB845CD (DATADG)
NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xccb845cd
NOTE: cache deleting context for group DATADG 1/0xccb845cd
GMON dismounting group 1 at 22 for pid 27, osid 5439
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup DATADG was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk “5” is missing from group number “1”
ERROR: alter diskgroup datadg mount
Wed Mar 13 16:42:10 2013
ASM Health Checker found 1 new failures
[grid@vmac1 ~]$ kfed read /dev/asm-diskh
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
7FA1DA233400 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
col path for a20
set linesize 200 pagesize 1400
select path,header_status,state from v$asm_disk;
PATH HEADER_STATUS STATE
——————– ———————————— ————————
/dev/asm-diskh CANDIDATE NORMAL
/dev/asm-diskg MEMBER NORMAL
/dev/asm-diskf MEMBER NORMAL
/dev/asm-diske MEMBER NORMAL
/dev/asm-diskc MEMBER NORMAL
/dev/asm-diskd MEMBER NORMAL
/dev/asm-diskb MEMBER NORMAL
7 rows selected.
[grid@vmac1 ~]$ kfed repair /dev/asm-diskh
KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576
KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
关闭ASM实例
优先备份 问题ASM的header
dd if=<bad disk> of=<file> bs=4096 count=1
3. 检查现有磁盘找出拥有file 1 block 1的
[grid@vmac1 ~]$ kfed read /dev/asm-diske |grep f1b1
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
[grid@vmac1 ~]$
[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000
[grid@vmac1 ~]$ kfed read /dev/asm-diskg |grep f1b1
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000
这里asm-diske上出现了f1b1非零值。则其拥有file 1 block 1，能够通过检查第二个au的类型是否是KFBTYP_LISTHEAD来确认
[grid@vmac1 ~]$ kfed read /dev/asm-diske aun=2|grep kfbh.type
kfbh.type: 5 ; 0x002: KFBTYP_LISTHEAD
若丢失的磁盘包括了”file 1 block 1 F1B1″则扫描 该磁盘上全部的 AU直到找到KFBTYP_LISTHEAD ， 假设找不到LISTHEAD 那么别无选择仅仅能重建diskgroup。
从版本号11.1.0.7開始(10g是从10.2.0.5開始，所以尽量别用10.2.0.5之前的版本号上的ASM)。当每个I/O写提交向ASM disk header(AU 0 blocknum 0)，都会拷贝到 AU 1中，最后第二个块。基于不同的AU size，该块的位置不同
Allocation Unit Size Block Number on AU 1
1048576 254
4194304 1022
8388608 2046
16777216 4094
首先使用kfed 验证该位置是否有正确的disk header，否则手动找到合适的header。 表1如參考：
[grid@vmac1 ~]$ kfed read /dev/asm-diske ausz=1048576 aun=1 blkn=254|less
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 254 ; 0x004: blk=254
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 2086475720 ; 0x00c: 0x7c5d17c8
kfbh.fcn.base: 31322 ; 0x010: 0x00007a5a
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
之后使用kfed repair命令修复disk header
[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576
假设disk header的自己主动备份也丢失了。那么会报例如以下错误
KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
若上述kfed repair无效则须要通过 手动恢复disk header的办法了：
对于版本号10.2.0.5之前没有kred repair可用的情况：
找一个与问题disk 在同一个diskgroup中的，不包括f1b1 的好的disk header。 比如这里的asm-diskf
[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000
使用kfed read <device name> > fix.txt 命令保存其头部
[grid@vmac1 ~]$ kfed read /dev/asm-diskf > fix.txt
编辑fix.txt 改动kfdhdb.dsknum 、kfdhdb.dskname 、 kfdhdb.fgname 三个信息：
同一时候參考alert.log中的信息：
[grid@vmac1 trace]$ grep “cache opening” alert_+ASM1.log
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske
NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske
NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske
NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 0 of grp 2: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 2: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 2: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh
原fix.txt中的内容：
[grid@vmac1 ~]$ egrep “dsknum|grptyp|hdrsts|dskname|grpname|fgname” fix.txt
kfdhdb.dsknum: 1 ; 0x024: 0x0001
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATADG_0001 ; 0x028: length=11
kfdhdb.grpname: DATADG ; 0x048: length=6
kfdhdb.fgname: DATADG_0001 ; 0x068: length=11
改动后：
[grid@vmac1 ~]$ egrep “dsknum|grptyp|hdrsts|dskname|grpname|fgname” fix.txt
kfdhdb.dsknum: 5 ; 0x024: 0x0005
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATADG_0005 ; 0x028: length=11
kfdhdb.grpname: DATADG ; 0x048: length=6
kfdhdb.fgname: DATADG_0005 ; 0x068: length=11
还须要改动 kfbh.block.obj
[grid@vmac1 ~]$ grep kfbh.block.obj fix.txt
kfbh.block.obj: 2147483649 ; 0x008: disk=1
2147483649==》0x80000001
0x80000001 最后一位是ASM DISK NUMBER , 应当等于kfdhdb.dsknum ， 在这里为 0x80000005 ==》 2147483653
[grid@vmac1 ~]$ grep kfbh.block.obj fix.txt
kfbh.block.obj: 2147483653 ; 0x008: disk=5
若使用windows平台上的ASMLIB则还要麻烦地改动kfdhdb.driver.reserved[0]， 所幸非常少人会用Windows上的ASMLIB
接着查看aunum=2 blknum=2来查找 disk directory， kfed read 拥有 f1b1的disk的 aunum=2 blknum=位置:
kfed read <device name> aunum=2 blknum=2 | more
[grid@vmac1 ~]$ kfed read /dev/asm-diske|grep f1b1
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
[grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=2 blknum=2|more
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 4 ; 0x002: KFBTYP_FILEDIR
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 2 ; 0x004: blk=2
kfbh.block.obj: 1 ; 0x008: file=1
kfbh.check: 322527999 ; 0x00c: 0x133962ff
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfffdb.node.incarn: 1 ; 0x000: A=1 NUMM=0x0
kfffdb.node.frlist.number: 4294967295 ; 0x004: 0xffffffff
…………………….
kfffde[0].xptr.au: 3 ; 0x4a0: 0x00000003
kfffde[0].xptr.disk: 0 ; 0x4a4: 0x0000
kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0
kfffde[0].xptr.chk: 41 ; 0x4a7: 0x29
kfffde[1].xptr.au: 4294967295 ; 0x4a8: 0xffffffff
kfffde[1].xptr.disk: 65535 ; 0x4ac: 0xffff
kfffde[1].xptr.flags: 0 ; 0x4ae: L=0 E=0 D=0 S=0
kfffde[1].xptr.chk: 42 ; 0x4af: 0x2a
==》disk directory 位于 disk=0 的 aunum=3
NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske ==> 还是 asm-diske
[grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=3 blknum=0|more
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 6 ; 0x002: KFBTYP_DISKDIR
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2 ; 0x008: file=2
kfbh.check: 389127513 ; 0x00c: 0x17319d59
kfbh.fcn.base: 31299 ; 0x010: 0x00007a43
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn: 1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number: 4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn: 0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number: 4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn: 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number: 0 ; 0x014: 0x00000000
kffdnd.parent.incarn: 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number: 0 ; 0x01c: 0x00000000
kffdnd.fstblk.incarn: 1 ; 0x020: A=1 NUMM=0x0
kfddde[0].entry.incarn: 1 ; 0x024: A=1 NUMM=0x0
kfddde[0].entry.hash: 0 ; 0x028: 0x00000000
kfddde[0].entry.refer.number:4294967295 ; 0x02c: 0xffffffff
kfddde[0].entry.refer.incarn: 0 ; 0x030: A=0 NUMM=0x0
………………………………………………….
kfddde 结构为 disk directory结构，仅有kfddde[0].entry.incarn A=1的记录是已分配的记录，若A=0则说明该记录被删除。
[grid@vmac1 ~]$ grep “kfddde[5]” disk.txt
kfddde[5].entry.incarn: 1 ; 0x8e4: A=1 NUMM=0x0
kfddde[5].entry.hash: 5 ; 0x8e8: 0x00000005
kfddde[5].entry.refer.number:4294967295 ; 0x8ec: 0xffffffff
kfddde[5].entry.refer.incarn: 0 ; 0x8f0: A=0 NUMM=0x0
kfddde[5].dsknum: 5 ; 0x8f4: 0x0005
kfddde[5].state: 2 ; 0x8f6: KFDSTA_NORMAL
kfddde[5].ddchgfl: 132 ; 0x8f7: 0x84
kfddde[5].dskname: DATADG_0005 ; 0x8f8: length=11
kfddde[5].fgname: DATADG_0005 ; 0x918: length=11
kfddde[5].crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfddde[5].crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24
kfddde[5].failstmp.hi: 0 ; 0x940: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0
kfddde[5].failstmp.lo: 0 ; 0x944: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfddde[5].timer: 0 ; 0x948: 0x00000000
kfddde[5].size: 5120 ; 0x94c: 0x00001400
kfddde[5].srRloc.super.hiStart: 0 ; 0x950: 0x00000000
kfddde[5].srRloc.super.loStart: 0 ; 0x954: 0x00000000
kfddde[5].srRloc.super.length: 0 ; 0x958: 0x00000000
kfddde[5].srRloc.incarn: 0 ; 0x95c: 0x00000000
kfddde[5].dskrprtm: 0 ; 0x960: 0x00000000
kfddde[5].zones[0].start: 0 ; 0x964: 0x00000000
kfddde[5].zones[0].size: 5120 ; 0x968: 0x00001400
kfddde[5].zones[0].used: 2 ; 0x96c: 0x00000002
回到编辑fix.txt上来， 调整 crestmp.hi 和 crestmp.lo 匹配上面显示的信息，若已经匹配则无需改动。
原本
[grid@vmac1 ~]$ egrep “hi|lo” fix.txt
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483653 ; 0x008: disk=5
kfdhdb.crestmp.hi: 32983191 ; 0x0a8: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd
kfdhdb.crestmp.lo: 2328519680 ; 0x0ac: USEC=0x0 MSEC=0x299 SECS=0x2c MINS=0x22
kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000
kfdhdb.grpstmp.hi: 32983191 ; 0x0e4: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd
kfdhdb.grpstmp.lo: 2328331264 ; 0x0e8: USEC=0x0 MSEC=0x1e1 SECS=0x2c MINS=0x22
改动后
kfdhdb.crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfdhdb.crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24
kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12
之后使用kfed merge命令合并disk header
kfed merge <device name> text=fix.txt
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt
若使用ASMLIb，则使用例如以下命令修复header中的asmlib信息
/etc/init.d/oracleasm force-renamedisk /dev/sdbg1 <ASMLIB Disk Name>
/etc/init.d/oracleasm scandisks
/etc/init.d/oracleasm listdisks
之后启动ASM实例到nomount
SQL> startup nomount;
SQL> col path for a20
SQL> set linesize 200 pagesize 1400
SQL> select path,header_status,state from v$asm_disk;
PATH HEADER_STATUS STATE
——————– ———————————— ————————
/dev/asm-diskh MEMBER NORMAL
/dev/asm-diskg MEMBER NORMAL
/dev/asm-diskf MEMBER NORMAL
/dev/asm-diske MEMBER NORMAL
/dev/asm-diskc MEMBER NORMAL
/dev/asm-diskd MEMBER NORMAL
/dev/asm-diskb MEMBER NORMAL
7 rows selected.
检查头部信息是否为member
[grid@vmac1 ~]$ kfed read /dev/asm-diskh
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483653 ; 0x008: disk=5
kfbh.check: 3412972861 ; 0x00c: 0xcb6dd53d
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
之后 alter diskgroup 问题DG mount;
若上述步骤无问题则能够正常mount dg;
SQL>
SQL> alter diskgroup datadg mount;
Diskgroup altered.
NOTE: cache registered group DATADG number=1 incarn=0x01c845f0
NOTE: cache began mount (first) of group DATADG number=1 incarn=0x01c845f0
NOTE: Assigning number (1,5) to disk (/dev/asm-diskh)
NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)
NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)
NOTE: Assigning number (1,0) to disk (/dev/asm-diske)
Wed Mar 13 19:39:49 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 56 for pid 27, osid 8690
NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske
NOTE: F1X0 found on disk 0 au 2 fcn 0.31322
NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh
NOTE: cache mounting (first) external redundancy group 1/0x01C845F0 (DATADG)
Wed Mar 13 19:39:49 2013
* allocate domain 1, invalid = TRUE
kjbdomatt send to inst 2
Wed Mar 13 19:39:49 2013
NOTE: attached to recovery domain 1
NOTE: starting recovery of thread=1 ckpt=11.2351 group=1 (DATADG)
NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=11.2351
NOTE: cache recovered group 1 to fcn 0.33763
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Wed Mar 13 19:39:49 2013
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG)
NOTE: LGWR found thread 1 closed at ABA 11.2350
NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG)
NOTE: LGWR opening thread 1 at fcn 0.33763 ABA 12.2351
NOTE: cache mounting group 1/0x01C845F0 (DATADG) succeeded
NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x01c845f0
GMON querying group 1 at 57 for pid 18, osid 2911
Wed Mar 13 19:39:49 2013
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup DATADG was mounted
SUCCESS: alter diskgroup datadg mount
Wed Mar 13 19:39:49 2013
NOTE: diskgroup resource ora.DATADG.dg is online
NOTE: diskgroup resource ora.DATADG.dg is updated
Wed Mar 13 19:39:59 2013
NOTE: client PROD1:PROD registered, osid 10169, mbr 0x1
Wed Mar 13 19:40:11 2013
NOTE: ASM client PROD1:PROD disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_10169.trc
注意 以上修复仅针对 asm header丢失磁盘头大约20MB(不一定)空间的数据， 若很多其它则仍可能无法mount diskgroup
若file numbe=4 Continuing Operations Directory (COD) – ASM file number 4也丢失了 那么一般非常难mount了。在以下的样例中是前22MB丢失时即便kfed merge也无力回天了。
SELECT x.xnum_kffxp “Extent”,
x.au_kffxp “AU”,
x.disk_kffxp “Disk #”,
d.name “Disk name”
FROM x$kffxp x, v$asm_disk_stat d
WHERE x.group_kffxp=d.group_number
and x.disk_kffxp=d.disk_number
and x.group_kffxp=1
and x.number_kffxp=4
ORDER BY 1, 2;
Extent AU Disk # Disk name
———- ———- ———- ——————————————————————————————
0 21 5 DATADG_0005
1 16 1 DATADG_0001
2 33 2 DATADG_0002
3 34 0 DATADG_0000
4 22 5 DATADG_0005
5 34 2 DATADG_0002
6 35 0 DATADG_0000
7 33 1 DATADG_0001
SQL> alter diskgroup datadg dismount;
Diskgroup altered.
[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 0.0165823 s, 1.3 GB/s
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt
SQL> alter diskgroup datadg mount;
Diskgroup altered.
SQL> alter diskgroup datadg mount;
Diskgroup altered.
SQL> alter diskgroup datadg dismount;
Diskgroup altered.
[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=21
21+0 records in
21+0 records out
22020096 bytes (22 MB) copied, 0.0182842 s, 1.2 GB/s
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt
SQL> alter diskgroup datadg mount;
Diskgroup altered.
[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=22
22+0 records in
22+0 records out
23068672 bytes (23 MB) copied, 0.0312157 s, 739 MB/s
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt
SQL> alter diskgroup datadg mount;
alter diskgroup datadg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15130: diskgroup “DATADG” is being dismounted
ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
Errors in file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc:
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ERROR: cache failed to read group=1(DATADG) fn=4 blk=0 from disk(s): 5(DATADG_0005)
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
NOTE: cache initiating offline of disk 5 group DATADG
NOTE: process _user8690_+asm1 (8690) initiating offline of disk 5.3915953639 (DATADG_0005) with mask 0x7e in group 1
WARNING: Disk 5 (DATADG_0005) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 5/0xe968b5e7, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 108 for pid 27, osid 8690
ERROR: Disk 5 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 1)
WARNING: Offline of disk 5 (DATADG_0005) in group 1 and mode 0x7f failed on ASM inst 1
Wed Mar 13 20:00:56 2013
NOTE: halting all I/Os to diskgroup 1 (DATADG)
System State dumped to trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc
NOTE: AMDU dump of disk group DATADG created at /g01/orabase/diag/asm/+asm/+ASM1/trace
ERROR: ORA-15130 signalled during mount of diskgroup DATADG
NOTE: cache dismounting (clean) group 1/0xEB784617 (DATADG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 8690, image: oracle@vmac1 (TNS V1-V3)
NOTE: LGWR doing non-clean dismount of group 1 (DATADG)
NOTE: LGWR sync ABA=18.2360 last written ABA 18.2360
kjbdomdet send to inst 2
detach from dom 1, sending detach message to inst 2
Wed Mar 13 20:00:57 2013
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 12)
Global Resource Directory partially frozen for dirty detach
* dirty detach – domain 1 invalid = TRUE
0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
freeing rdom 1
WARNING: dirty detached from domain 1
NOTE: cache dismounted group 1/0xEB784617 (DATADG)
NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xeb784617
NOTE: cache deleting context for group DATADG 1/0xeb784617
GMON dismounting group 1 at 109 for pid 27, osid 8690
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup DATADG was not mounted
ORA-15032: not all alterations performed
ORA-15130: diskgroup “DATADG” is being dismounted
ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ERROR: alter diskgroup datadg mount
SQL> alter diskgroup datadg mount;
alter diskgroup datadg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15130: diskgroup “DATADG” is being dismounted
ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1]
[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=3 seek=3 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000617397 s, 6.6 MB/s
[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=4 seek=4 count=1
kfffde[0].xptr.au: 21 ; 0x4a0: 0x00000015
kfffde[0].xptr.disk: 5 ; 0x4a4: 0x0005
kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0
kfffde[0].xptr.chk: 58 ; 0x4a7: 0x3a
kfffde[1].xptr.au: 16 ; 0x4a8: 0x00000010
[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=1048576 skip=21 seek=21 count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00296742 s, 353 MB/s
ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4]
ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4]
ERROR: alter diskgroup datadg mount force