每年年底，系统管理员都要组织一次容灾方案的测试、演练。会在一个与生产环境网络隔离的DR环境中，启动各个“生产环境服务器”，然后让各路人员参与其中测试、演练容灾方案是否可靠。这次演练中，一台Oracle数据库服务器启动的时候遇到了问题。如下所示，启动的时候遇到ORA-03113: end-of-file on communication channel错误。
[oracle@mylnx6 ~]$ sqlplus / as sysdbaSQL*Plus: Release 10.2.0.5.0 - Production on Fri Dec 21 09:42:11 2018
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.Connected to an idle instance.SQL> startupORA-03113: end-of-file on communication channel
SQL>
检查告警日志，发现数据库在启动的时候，报“ORA-00471: DBWR process terminated with error”错误。如下所示：
PMON started with pid=2, OS id=25005
PSP0 started with pid=3, OS id=25007
MMAN started with pid=4, OS id=25009
DBW0 started with pid=5, OS id=25011
LGWR started with pid=6, OS id=25013
CKPT started with pid=7, OS id=25016
SMON started with pid=8, OS id=25018
RECO started with pid=9, OS id=25020
CJQ0 started with pid=10, OS id=25022
MMON started with pid=11, OS id=25024
Fri Dec 21 09:44:36 CST 2018
starting up 8 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=25026
Fri Dec 21 09:45:12 CST 2018
starting up 24 shared server(s) ...
Fri Dec 21 09:46:43 CST 2018
Errors in file /u01/app/oracle/admin/SCM2/bdump/scm2_pmon_25005.trc:
ORA-00471: DBWR process terminated with error
Fri Dec 21 09:46:43 CST 2018
PMON: terminating instance due to error 471
Instance terminated by PMON, pid = 25005
启动数据库实例的时候，报“ORA-00471: DBWR process terminated with error”这个错误，这个很蹊跷，很有可能是进程被系统给Kill掉了，检查操作系统的错误日志，发现出现了oom_kill_process,也就是说数据库实例启动的时候，由于系统内存资源紧张，DBWR进程被系统选作了牺牲品。具体错误日志如下所示：
Dec 21 09:46:39 mylnx6 kernel: oracle invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Dec 21 09:46:39 mylnx6 kernel: oracle cpuset=/ mems_allowed=0
Dec 21 09:46:39 mylnx6 kernel: Pid: 25026, comm: oracle Not tainted 2.6.32-200.13.1.el5uek #1Dec 21 09:46:39 mylnx6 kernel: Call Trace:
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810a0b66>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810d9ae6>] oom_kill_process+0x85/0x25b
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810d9fbc>] ? select_bad_process+0xbc/0x102
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810da03f>] __out_of_memory+0x3d/0x86
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810da30f>] out_of_memory+0xfc/0x195
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810dd75e>] __alloc_pages_nodemask+0x487/0x595
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff811075ac>] alloc_page_vma+0xb9/0xc8
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810ff0a7>] read_swap_cache_async+0x52/0xf1
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810ff1a3>] swapin_readahead+0x5d/0x9c
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810d725a>] ? find_get_page+0x22/0x69
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff810f1ea3>] handle_mm_fault+0x44b/0x80f
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff8106d7cd>] ? getrusage+0x2b1/0x2ce
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff8101270e>] ? common_interrupt+0xe/0x13
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff81043696>] ? should_resched+0xe/0x2f
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff81456006>] do_page_fault+0x210/0x299
Dec 21 09:46:39 mylnx6 kernel: [<ffffffff81453fd5>] page_fault+0x25/0x30
Dec 21 09:46:39 mylnx6 kernel: Mem-Info:
Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA per-cpu:
Dec 21 09:46:39 mylnx6 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 4: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 5: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 6: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 7: hi: 0, btch: 1 usd: 0
Dec 21 09:46:39 mylnx6 kernel: Node 0 DMA32 per-cpu:
Dec 21 09:46:39 mylnx6 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Dec 21 09:46:39 mylnx6 kernel: Node 0 Normal per-cpu:
Dec 21 09:46:39 mylnx6 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...
Dec 21 09:46:41 mylnx6 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Dec 21 09:46:40 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...
Dec 21 09:46:41 mylnx6 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Dec 21 09:46:41 mylnx6 kernel: active_anon:1764 inactive_anon:209 isolated_anon:64
Dec 21 09:46:41 mylnx6 kernel: active_file:349 inactive_file:1710 isolated_file:0
Dec 21 09:46:41 mylnx6 kernel: unevictable:5377 dirty:0 writeback:4 unstable:0
Dec 21 09:46:41 mylnx6 kernel: free:29838 slab_reclaimable:2400 slab_unreclaimable:119491
Dec 21 09:46:41 mylnx6 kernel: mapped:2703 shmem:830 pagetables:9849 bounce:0
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA free:15652kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15172kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 3000 24210 24210
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32 free:86296kB min:2464kB low:3080kB high:3696kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 21210 21210
Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal free:17404kB min:17440kB low:21800kB high:26160kB active_anon:7056kB inactive_anon:836kB active_file:1396kB inactive_file:6840kB unevictable:21508kB isolated(anon):256kB isolated(file):0kB present:21719040kB mlocked:21504kB dirty:0kB writeback:16kB mapped:10812kB shmem:3320kB slab_reclaimable:9600kB slab_unreclaimable:477964kB kernel_stack:2800kB pagetables:39396kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no
Dec 21 09:46:41 mylnx6 kernel: lowmem_reserve[]: 0 0 0 0
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA: 1*4kB 2*8kB 1*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15652kB
Dec 21 09:46:41 mylnx6 kernel: Node 0 DMA32: 12*4kB 13*8kB 2*16kB 5*32kB 5*64kB 11*128kB 3*256kB 7*512kB 6*1024kB 4*2048kB 16*4096kB = 86296kB
Dec 21 09:46:41 mylnx6 kernel: Node 0 Normal: 420*4kB 1917*8kB 49*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17800kB
Dec 21 09:46:41 mylnx6 kernel: 4722 total pagecache pages
Dec 21 09:46:41 mylnx6 kernel: 694 pages in swap cacheDec 21 09:46:41 mylnx6 kernel: Swap cache stats: add 589182, delete 588488, find 343370/443306
Dec 21 09:46:41 mylnx6 kernel: Free swap = 66723056kB
Dec 21 09:46:41 mylnx6 kernel: Total swap = 67108856kB
Dec 21 09:46:41 mylnx6 kernel: 6291440 pages RAM
Dec 21 09:46:41 mylnx6 kernel: 107316 pages reserved
Dec 21 09:46:41 mylnx6 kernel: 24060 pages shared
Dec 21 09:46:41 mylnx6 kernel: 77648 pages non-shared
Dec 21 09:46:41 mylnx6 kernel: Out of memory: kill process 25011 (oracle) score 8425150 or a childDec 21 09:46:41 mylnx6 kernel: Killed process 25011 (oracle)Dec 21 09:47:20 mylnx6 lvm[4702]: Another thread is handling an event. Waiting...
检查这个系统的内存，发现DR环境下，这个服务器只分配了24G内存，而实际生产环境的内存为64G（设置了Linux标准大页，而且SGA_MAX_SIZE大小为32G），而且这个环境是生产环境的“克隆体”，只是由于资源限制的缘故，系统管理员只分配24G内存。如下所示：
[root@mylnx6 ~]# free -m total used free shared buffers cachedMem: 24156 24033 123 0 0 6
-/+ buffers/cache: 24026 130
Swap: 65535 41 65494
[root@mylnx6 ~]# ps -ef | grep ora_
root 11759 11490 0 16:10 pts/1 00:00:00 grep ora_
[root@mylnx6 ~]# ipcs -m
------ Shared Memory Segments --------key shmid owner perms bytes nattch status 0x00000000 3080192 root 644 80 2
0x00000000 3112961 root 644 16384 2
0x00000000 3145730 root 644 280 2
0x00000000 4096003 gdm 600 393216 0
0x2cd12178 3866628 oracle 640 34361835520 0
0x00000000 5210117 gdm 600 393216 2 dest
如上所示，可以看到oracle用户的共享内存段为34361835520字节。所以引起这个错误的原因是因为在系统层面配置了标准大页的缘故（内存资源变化了，但是配置没有随之修改），为了快速解决问题，我们取消标准大页的相关设置。如下所示：
修改limits.conf参数，注释soft memlock和hard memlock参数。
vi /etc/security/limits.conf
然后修改sysctl.conf，将vm.nr_hugepages注释掉。然后重启一下（DR测试环境，可以随时重启）。然后启动Oracle数据库实例，一切正常，当然还需调整相关参数，继续后续测试~。