环境：虚拟机
[root@rac1 ~]# lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch
rinting-4.0-amd64rinting-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.10 (Final)
Release: 6.10
Codename: Final
[root@rac1 ~]# uname -r
2.6.32-754.el6.x86_64
SQL*Plus: Release 11.2.0.4.0 Production on Fri Feb 22 09:06:15 2019
在安装过程中，grid，asm，db软件安装都还比较顺利，但是在dbca的过程中，13%
[root@rac1 trace]# tail -n 200 alert_bol1.log
ALTER DATABASE DEFAULT TABLESPACE "USERS"
Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"
Fri Feb 22 12:42:13 2019
Thread 1 advanced to log sequence 2 (LGWR switch)
Current log# 2 seq# 2 mem# 0: +DATA/bol/onlinelog/group_2.267.1000903313
Current log# 2 seq# 2 mem# 1: +FRA/bol/onlinelog/group_2.261.1000903313
Fri Feb 22 12:42:31 2019
Thread 1 cannot allocate new log, sequence 3
Checkpoint not complete
Current log# 2 seq# 2 mem# 0: +DATA/bol/onlinelog/group_2.267.1000903313
Current log# 2 seq# 2 mem# 1: +FRA/bol/onlinelog/group_2.261.1000903313
Thread 1 advanced to log sequence 3 (LGWR switch)
Current log# 1 seq# 3 mem# 0: +DATA/bol/onlinelog/group_1.266.1000903311
Current log# 1 seq# 3 mem# 1: +FRA/bol/onlinelog/group_1.260.1000903311
Fri Feb 22 12:43:04 2019
Thread 1 cannot allocate new log, sequence 4
Checkpoint not complete
Current log# 1 seq# 3 mem# 0: +DATA/bol/onlinelog/group_1.266.1000903311
Current log# 1 seq# 3 mem# 1: +FRA/bol/onlinelog/group_1.260.1000903311
Thread 1 advanced to log sequence 4 (LGWR switch)
Current log# 2 seq# 4 mem# 0: +DATA/bol/onlinelog/group_2.267.1000903313
Current log# 2 seq# 4 mem# 1: +FRA/bol/onlinelog/group_2.261.1000903313
Fri Feb 22 12:43:15 2019
WARNING: Write Failed. group:3 disk:0 AU:315 offset:210432 size:1024
WARNING: Write Failed. group:2 disk:1 AU:1026 offset:210432 size:1024
Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1024
WARNING: failed to write mirror side 1 of virtual extent 12 logical extent 0 of file 267 in group 2 on disk 1 allocation unit 1026
Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1024
WARNING: failed to write mirror side 1 of virtual extent 12 logical extent 0 of file 261 in group 3 on disk 0 allocation unit 315
Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:
ORA-00345: redo log write error block 24987 count 2
ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1000903313'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00345: redo log write error block 24987 count 2
ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1000903313'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:
ORA-00340: IO error processing online log 2 of thread 1
ORA-00345: redo log write error block 24987 count 2
ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1000903313'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00345: redo log write error block 24987 count 2
ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1000903313'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
LGWR (ospid: 13834): terminating the instance due to error 340
Fri Feb 22 12:43:16 2019
System state dump requested by (instance=1, osid=13834 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_diag_13812_20190222124316.trc
Dumping diagnostic data in directory=[cdmp_20190222124316], requested by (instance=1, osid=13834 (LGWR)), summary=[abnormal instance termination].
Instance terminated by LGWR, pid = 13834
asm alert日志
Fri Feb 22 17:32:48 2019
NOTE: client bol1:bol registered, osid 30819, mbr 0x1
Fri Feb 22 17:32:51 2019
NOTE: Advanced to new COD format for group DATA
NOTE: Advanced to new COD format for group FRA
Fri Feb 22 17:34:30 2019
NOTE: ASM client bol1:bol disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_30819.trc
遇到此问题，一开始以为是缺少什么安装包或者共享磁盘的配置问题
在换一套安装依然如此
第一套使用udev，第二套使用asmlib
asm相关状态
[grid@rac2 ~]$ crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora.FRA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1 ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1 ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1 ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac1 ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1 ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac1 ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1 ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1 ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1 ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2 ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2 ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2 ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
[grid@rac1 .ssh]$ olsnodes -n rac1 1 rac2 2 [grid@rac1 .ssh]$ olsnodes -n -i -s -t rac1 1 rac1vip Active Unpinned rac2 2 rac2vip Active Unpinned [grid@rac1 .ssh]$ srvctl status asm -a ASM is running on rac2,rac1 ASM is enabled.
[grid@rac2 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 4c22914de7394f7abfd6c9635cb1c1fd (ORCL:OCR_VOTE1) [OCR] 2. ONLINE 0f6aab9aaa744faebfc22d0622804518 (ORCL:OCR_VOTE2) [OCR] 3. ONLINE 1d1fd2ecffcf4f76bf0ea356730a9c52 (ORCL:OCR_VOTE3) [OCR] Located 3 voting disk(s).
[grid@rac1 trace]$ ps -ef|grep lsnr|grep -v 'grep'|grep -v 'ocfs'|awk '{print$9}' LISTENER LISTENER_SCAN1
[grid@rac1 ~]$ asmcmd ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 163840 161875 0 161875 0 N DATA/ MOUNTED EXTERN N 512 4096 1048576 81920 81692 0 81692 0 N FRA/ MOUNTED EXTERN N 512 4096 1048576 6144 5744 0 5744 0 Y OCR/
在查找过程中，找到相似的文档和案例
https://community.oracle.com/thread/2477850?start=0&tstart=0
https://www.itcodemonkey.com/article/8825.html
听说这次用的存储是自定义？ 第一次使用的。。。目前还没有搞定此问题
--添加详细日志
[root@rac1 ~]# ls -l /dev/asm* brw-rw---- 1 grid asmadmin 8, 16 Feb 26 09:00 /dev/asm-diskb brw-rw---- 1 grid asmadmin 8, 32 Feb 25 18:30 /dev/asm-diskc brw-rw---- 1 grid asmadmin 8, 48 Feb 25 21:31 /dev/asm-diskd brw-rw---- 1 grid asmadmin 8, 64 Feb 25 23:11 /dev/asm-diske brw-rw---- 1 grid asmadmin 8, 80 Feb 25 18:22 /dev/asm-diskf brw-rw---- 1 grid asmadmin 8, 96 Feb 25 23:11 /dev/asm-diskg [root@rac1 dev]# ll /dev/sd* brw-rw---- 1 root disk 8, 0 Feb 25 16:46 /dev/sda brw-rw---- 1 root disk 8, 1 Feb 25 16:46 /dev/sda1 brw-rw---- 1 root disk 8, 2 Feb 25 16:46 /dev/sda2 brw-rw---- 1 root disk 8, 16 Feb 25 16:59 /dev/sdb brw-rw---- 1 root disk 8, 32 Feb 25 16:59 /dev/sdc brw-rw---- 1 root disk 8, 48 Feb 25 16:59 /dev/sdd brw-rw---- 1 root disk 8, 64 Feb 25 16:59 /dev/sde brw-rw---- 1 root disk 8, 80 Feb 25 16:59 /dev/sdf brw-rw---- 1 root disk 8, 96 Feb 25 16:59 /dev/sdg [root@rac1 ~]# ll /u01/app/oracle/ total 20 drwxr-x--- 3 oracle oinstall 4096 Feb 25 17:58 admin drwxr-x--- 4 oracle oinstall 4096 Feb 25 17:55 cfgtoollogs drwxr-xr-x 2 oracle oinstall 4096 Feb 25 17:52 checkpoints drwxrwxr-x 11 oracle oinstall 4096 Feb 25 17:45 diag drwxr-xr-x 3 oracle oinstall 4096 Feb 25 17:40 product [grid@rac1 ~]$ crs_stat -t -v CRS-0184: Cannot communicate with the CRS daemon. [root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl start crs CRS-4640: Oracle High Availability Services is already active CRS-4000: Command Start failed, or completed with errors. SQL> conn /as sysasm Connected. SQL> select name,state from v$asm_diskgroup; NAME STATE ------------------------------ ----------- OCR DISMOUNTED FRA DISMOUNTED DATA DISMOUNTED SQL> ALTER DISKGROUP ALL MOUNT; [root@rac1 ~]# tail -f -n 300 /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log ORA-15078: ASM diskgroup was forcibly dismounted Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc: ORA-15078: ASM diskgroup was forcibly dismounted Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc: ORA-15078: ASM diskgroup was forcibly dismounted Mon Feb 25 23:16:52 2019 NOTE: ASMB process exiting due to lack of ASM file activity for 305 seconds Mon Feb 25 23:16:52 2019 NOTE: client +ASM1:+ASM deregistered Mon Feb 25 23:18:24 2019 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc: ORA-15078: ASM diskgroup was forcibly dismounted Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc: Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc: ORA-15078: ASM diskgroup was forcibly dismounted WARNING: requested mirror side 1 of virtual extent 4 logical extent 0 offset 446464 is not allocated; I/O request failed Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc: ORA-15078: ASM diskgroup was forcibly dismounted Mon Feb 25 23:18:24 2019 * allocate domain 3, invalid = TRUE kjbdomatt send to inst 2 NOTE: attached to recovery domain 3 NOTE: starting recovery of thread=2 ckpt=2.0 group=3 (OCR) NOTE: advancing ckpt for group 3 (OCR) thread=2 ckpt=2.0 NOTE: cache recovered group 3 to fcn 0.531 NOTE: redo buffer size is 256 blocks (1053184 bytes) NOTE: LGWR attempting to mount thread 1 for diskgroup 3 (OCR) NOTE: LGWR found thread 1 closed at ABA 3.36 NOTE: LGWR mounted thread 1 for diskgroup 3 (OCR) NOTE: LGWR opening thread 1 at fcn 0.531 ABA 4.37 NOTE: cache mounting group 3/0xB2FAEF18 (OCR) succeeded NOTE: cache ending mount (success) of group OCR number=3 incarn=0xb2faef18 Tue Feb 26 09:07:00 2019 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATA was mounted NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2 SUCCESS: diskgroup FRA was mounted NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 3 SUCCESS: diskgroup OCR was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 1 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Successful voting file relocation on diskgroup OCR Tue Feb 26 09:07:02 2019 WARNING: failed to online diskgroup resource ora.DATA.dg (unable to communicate with CRSD/OHASD) WARNING: failed to online diskgroup resource ora.FRA.dg (unable to communicate with CRSD/OHASD) WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD) [root@rac1 ~]# vim /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc Trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options ORACLE_HOME = /u01/app/11.2.0/grid System name: Linux Node name: rac1 Release: 2.6.32-754.el6.x86_64 Version: #1 SMP Tue Jun 19 21:26:04 UTC 2018 Machine: x86_64 VM name: VMWare Version: 6 Instance name: +ASM1 Redo thread mounted by this instance: 0 <none> Oracle process number: 24 Unix process pid: 14673, image: oracle@rac1 (TNS V1-V3) *** 2019-02-25 17:17:34.002 *** SESSION ID:(13.3) 2019-02-25 17:17:34.002 *** CLIENT ID:() 2019-02-25 17:17:34.002 *** SERVICE NAME:() 2019-02-25 17:17:34.002 *** MODULE NAME:(crsd.bin@rac1 (TNS V1-V3)) 2019-02-25 17:17:34.002 *** ACTION NAME:() 2019-02-25 17:17:34.002 WARNING: Write Failed. group:1 disk:2 AU:23 offset:151552 size:4096 path:/dev/asm-diskd incarnation:0xe96a1e30 synchronous result:'I/O error' subsys:System iop:0x7f688dfdd000 bufp:0x7f688e3d4a00 osderr:0x0 osderr1:0x0 ORA-15080: synchronous I/O operation to a disk failed ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 47400 Additional information: -1 WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 1 on disk 2 allocation unit 23 ORA-15081: failed to submit an I/O operation to a disk ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 47400 Additional information: -1 WARNING: Write Failed. group:1 disk:2 AU:23 offset:118784 size:4096 path:/dev/asm-diskd incarnation:0xe96a1e30 synchronous result:'I/O error' subsys:System iop:0x7f688dfdd000 bufp:0x7f688e3d4a00 osderr:0x0 osderr1:0x0 ORA-15080: synchronous I/O operation to a disk failed ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 47400 Additional information: -1 WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 1 on disk 2 allocation unit 23 ORA-15081: failed to submit an I/O operation to a disk ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 47400 Additional information: -1 WARNING: Write Failed. group:1 disk:2 AU:23 offset:118784 size:4096 path:/dev/asm-diskd incarnation:0xe96a1e30 synchronous result:'I/O error' subsys:System iop:0x7f688dfdd000 bufp:0x7f688e3d4a00 osderr:0x0 osderr1:0x0 ORA-15080: synchronous I/O operation to a disk failed ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 47336 Additional information: -1 WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 1 on disk 2 allocation unit 23 ORA-15081: failed to submit an I/O operation to a disk ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 47336 Additional information: -1 *** 2019-02-25 17:28:27.478 Received ORADEBUG command (#1) 'CLEANUP_KFK_FD' from process 'Unix process pid: 18224, image: <none>' *** 2019-02-25 17:28:27.478 Finished processing ORADEBUG command (#1) 'CLEANUP_KFK_FD' *** 2019-02-25 23:11:47.745 WARNING:failed xlate 1 ORA-15078: ASM diskgroup was forcibly dismounted WARNING:failed xlate 1 *** 2019-02-25 23:11:48.098 WARNING:failed xlate 1 ORA-15078: ASM diskgroup was forcibly dismounted WARNING:failed xlate 1 ORA-15078: ASM diskgroup was forcibly dismounted *** 2019-02-25 23:11:58.466 Received ORADEBUG command (#2) 'CLEANUP_KFK_FD' from process 'Unix process pid: 29490, image: <none>' *** 2019-02-25 23:11:58.467 Finished processing ORADEBUG command (#2) 'CLEANUP_KFK_FD' *** 2019-02-25 23:12:13.475 Received ORADEBUG command (#3) 'CLEANUP_KFK_FD' from process 'Unix process pid: 29490, image: <none>' *** 2019-02-25 23:12:13.475 Finished processing ORADEBUG command (#3) 'CLEANUP_KFK_FD' ORA-15078: ASM diskgroup was forcibly dismounted ksfdrfms:Mirror Read file=+OCR.255.4294967295 fob=0xa0802c18 bufp=0x7f688e3d4a00 blkno=1133 nbytes=4096 WARNING:failed xlate 1 WARNING: requested mirror side 1 of virtual extent 4 logical extent 0 offset 446464 is not allocated; I/O request failed ksfdrfms:Read failed from mirror side=1 logical extent number=0 dskno=65535 ORA-15078: ASM diskgroup was forcibly dismounted [root@rac1 log]# tail -n 300 /var/log/messages Feb 25 16:47:34 rac1 rtkit-daemon[2828]: Sucessfully made thread 2826 of process 2826 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11. Feb 25 17:13:36 rac1 root: exec /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib /u01/app/11.2.0/grid/bin/crswrapexece.pl /u01/app/11.2.0/grid/crs/install/s_crsconfig_rac1_env.txt /u01/app/11.2.0/grid/bin/ohasd.bin "reboot" Feb 25 17:17:33 rac1 kernel: sd 3:0:0:0: reservation conflict Feb 25 17:17:33 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Feb 25 17:17:33 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 14 00 00 01 00 Feb 25 17:17:33 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262164 Feb 25 17:17:34 rac1 kernel: sd 3:0:2:0: reservation conflict Feb 25 17:17:34 rac1 kernel: sd 3:0:2:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Feb 25 17:17:34 rac1 kernel: sd 3:0:2:0: [sdd] CDB: Write(10): 2a 00 00 00 b9 28 00 00 08 00 Feb 25 17:17:34 rac1 kernel: end_request: critical nexus error, dev sdd, sector 47400 .... Feb 25 23:49:02 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262148 Feb 26 06:10:49 rac1 kernel: sd 3:0:0:0: reservation conflict Feb 26 06:10:49 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Feb 26 06:10:49 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 03 00 00 01 00 Feb 26 06:10:49 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262147 Feb 26 06:38:40 rac1 kernel: sd 3:0:0:0: reservation conflict Feb 26 06:38:40 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Feb 26 06:38:40 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 03 00 00 01 00 Feb 26 06:38:40 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262147 Feb 26 08:39:13 rac1 kernel: sd 3:0:0:0: reservation conflict Feb 26 08:39:13 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Feb 26 08:39:13 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 04 00 00 01 00 Feb 26 08:39:13 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262148 [root@rac1 ~]# tail -n 200 /u01/app/oracle/diag/rdbms/bol/bol1/trace/alert_bol1.log ALTER DATABASE DEFAULT TABLESPACE "USERS" Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS" Mon Feb 25 18:22:32 2019 Thread 1 advanced to log sequence 2 (LGWR switch) Current log# 2 seq# 2 mem# 0: +DATA/bol/onlinelog/group_2.267.1001182931 Current log# 2 seq# 2 mem# 1: +FRA/bol/onlinelog/group_2.261.1001182931 Mon Feb 25 18:22:49 2019 WARNING: Write Failed. group:3 disk:0 AU:333 offset:402432 size:1536 WARNING: Write Failed. group:2 disk:1 AU:1136 offset:402432 size:1536 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc: ORA-15080: synchronous I/O operation to a disk failed ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1 Additional information: 1536 WARNING: failed to write mirror side 1 of virtual extent 30 logical extent 0 of file 267 in group 2 on disk 1 allocation unit 1136 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc: ORA-15080: synchronous I/O operation to a disk failed ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1 Additional information: 1536 WARNING: failed to write mirror side 1 of virtual extent 30 logical extent 0 of file 261 in group 3 on disk 0 allocation unit 333 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc: ORA-00345: redo log write error block 62226 count 3 ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1001182931' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk ORA-00345: redo log write error block 62226 count 3 ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1001182931' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc: ORA-00340: IO error processing online log 2 of thread 1 ORA-00345: redo log write error block 62226 count 3 ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1001182931' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk ORA-00345: redo log write error block 62226 count 3 ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1001182931' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk LGWR (ospid: 32760): terminating the instance due to error 340 Mon Feb 25 18:22:49 2019 System state dump requested by (instance=1, osid=32760 (LGWR)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_diag_32738_20190225182249.trc Dumping diagnostic data in directory=[cdmp_20190