AIX 操作系统因 MTU 不一致导致主机和 RAC 数据库不断重启，事件就是发生在上周日。操作系统工程师因监控发现有一台主机不断重启，排查硬件后无问题，便将事件转至数据库工程师排查了。当时主机是一套 SVC 存储复制的灾备环境。
RAC 数据库是 11.2.0.4 版本的，操作系统是 AIX 7.1 版本，如下图：
MOS 上说该问题发生在 10.1.0.2 to 11.2.0.2 版本，但我这个库是 11.2.0.4，故记录一下排查过程。
Unable To Start ASM RAC Instances Due To ORA-27303: Remote Port MTU Does Not Match Local MTU. (Doc ID 947223.1)
Oracle Database - Enterprise Edition - Version 10.1.0.2 to 11.2.0.2 [Release 10.1 to 11.2]
MTU
MTU 是英文 Maximum Transmission Unit 的缩写，意为"最大传输单位"，也就是在连接的时候，所传输信息包最多可以有多少字节。我们必须找到不会返回 fragment (碎片)信息的最大 MTU。除了 ADSL PPPoE 的 MTU 是 1492 外，其余各种 DSL 的 MTU标准设置都是 1500。MaxMTU 是最大的 TCP/IP 传输单元，在 TCP/IP 协议中，将要传输的数据分成较小的组进行传输，每个组的大小为 576 字节。
AIX 系统错误日志存放路径：/var/adm/ras/errlog 。
errpt –a 列详细信息，详细使用方法可以参考 man。
AIX 系统启动错误日志存放路径：/var/adm/ras/bootlog。
该日志可以跟踪系统在 Boot 过程中发生的问题，包括服务器液晶板上的代码信息都有记载。可以使用 alog 命令监视这些问题, 存放在 /var/adm/ras/bootlog 中，可以使用 alog –o –t boot 命令查看该文件。
问题排查
RAC 的节点二是出现问题的机器，主机每隔五六分钟就会重启，而节点一数据库启动后过几分钟也就宕了，这就十分奇怪了，主机二不断宕机会导致实例一宕机吗？故让我很好奇不断查下去了，以下是节点一的 alert 日志。
Sun Mar 15 19:06:11 2020
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.XX.12)(PORT=1521))' SCOPE=MEMORY SID='test1';
ALTER DATABASE MOUNT /* db agent *//* {0:11:33} */
This instance was first to mount
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/test/control02.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/test/control02.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/test/control01.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/test/control01.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-205 signalled during: ALTER DATABASE MOUNT /* db agent *//* {0:11:33} */...
Sun Mar 15 19:06:12 2020
Shutting down instance (abort)
以下是节点二 alert 日志，
Thu Mar 12 23:33:16 2020NOTE: ASMB terminatingErrors in file /app/oracle/diag/rdbms/test/test2/trace/test2_asmb_31457440.trc:ORA-15064: communication failure with ASM instanceORA-03113: end-of-file on communication channelProcess ID: Session ID: 2 Serial number: 1Errors in file /app/oracle/diag/rdbms/test/test2/trace/test2_asmb_31457440.trc:ORA-15064: communication failure with ASM instanceORA-03113: end-of-file on communication channelProcess ID: Session ID: 2 Serial number: 1ASMB (ospid: 31457440): terminating the instance due to error 15064Instance terminated by ASMB, pid = 31457440Sun Mar 15 19:51:34 2020Starting ORACLE instance (normal)LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0
这里说一句，怎么查 alert 日志有很多小伙伴们不熟悉数据库便不知道日志位置，那么怎么查呢，登陆数据库查看 dump 参数指定的位置就是告警日志咯！
节点 1 ASM 日志，ORA-27303：additional information: Remote port MTU does not match local MTU. [local: 1500, remote: 9000] (169.254.1XX.XX8)。ORA-27301、ORA-27302、ORA-27303 这里看到主要问题了吧。
继续看一下节点二日志:
GMON started with pid=19, OS id=19857586
Sun Mar 15 18:18:10 2020
MMON started with pid=20, OS id=19398838
Sun Mar 15 18:18:10 2020
MMNL started with pid=21, OS id=24838354
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 10)
ASM instance
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
KSXP IPC protocol is incompatible with instance 2
Errors in file /app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_24707194.trc:
ORA-27300: OS system dependent operation:config_check failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvalpid
ORA-27303: additional information: Remote port MTU does not match local MTU. [local: 1500, remote: 9000] (169.254.1XX.XX8)
KSXP IPC protocol is incompatible with instance 2
Errors in file /app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_24707194.trc:
ORA-27300: OS system dependent operation:config_check failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvalpid
ORA-27303: additional information: Remote port MTU does not match local MTU. [local: 1500, remote: 9000] (169.254.1XX.XX8)
Communication channels reestablished
节点 2 ASM 日志，ASM 日志存放于 ASM 实例之下，RAC 除了这两种日志外还有集群日志以及集群下的各种进程日志等多种，这个有时间在说。ASM 日志位于 $ORACLE_BASE/diag/asm/+asm/+ASM2/trace 下。
cd $ORACLE_BASE/diag/asm/+asm/+ASM2/tracetail -999f /app/grid/diag/asm/+asm/+ASM2/trace/alert*Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options.ORACLE_HOME = /app/product/11.2.0/gridSystem name: AIXNode name: test2Release: 1Version: 7Machine: 00C0A6D84C00Using parameter settings in server-side spfile +OCR/test-cluster/asmparameterfile/registry.253.1004572351System parameters with non-default values: large_pool_size = 12M instance_type = "asm" remote_login_passwordfile= "EXCLUSIVE" asm_diskgroups = "OCR" asm_diskgroups = "ARCH" asm_diskgroups = "DATA" asm_power_limit = 1 diagnostic_dest = "/app/grid"Cluster communication is configured to use the following interface(s) for this instance 169.254.1XX.XX8cluster interconnect IPC version:Oracle UDP/IP (generic)IPC Vendor 1 proto 2Sun Mar 15 19:00:30 2020PMON started with pid=2, OS id=22216894 Sun Mar 15 19:00:30 2020PSP0 started with pid=3, OS id=30539926 Sun Mar 15 19:00:31 2020VKTM started with pid=4, OS id=28573718 at elevated priorityVKTM running at (10)millisec precision with DBRM quantum (100)msSun Mar 15 19:00:31 2020GEN0 started with pid=5, OS id=22282482 Sun Mar 15 19:00:31 2020DIAG started with pid=6, OS id=29360284 Sun Mar 15 19:00:31 2020PING started with pid=7, OS id=21889258 Sun Mar 15 19:00:31 2020DIA0 started with pid=8, OS id=21823656 Sun Mar 15 19:00:31 2020LMON started with pid=9, OS id=28180628 Sun Mar 15 19:00:31 2020LMD0 started with pid=10, OS id=29819036 * System load used for high load check * New Low - High Load Threshold Range = [55296 - 73728] Sun Mar 15 19:00:31 2020LMS0 started with pid=11, OS id=30998728 at elevated prioritySun Mar 15 19:00:31 2020LMHB started with pid=12, OS id=27328766 Sun Mar 15 19:00:31 2020MMAN started with pid=13, OS id=22741166 Sun Mar 15 19:00:31 2020DBW0 started with pid=14, OS id=21495954 Sun Mar 15 19:00:31 2020LGWR started with pid=15, OS id=21168264 Sun Mar 15 19:00:31 2020CKPT started with pid=16, OS id=21954804 Sun Mar 15 19:00:31 2020SMON started with pid=17, OS id=30081278 Sun Mar 15 19:00:31 2020RBAL started with pid=18, OS id=27787438 Sun Mar 15 19:00:31 2020GMON started with pid=19, OS id=29425830 Sun Mar 15 19:00:31 2020MMON started with pid=20, OS id=27721860 Sun Mar 15 19:00:31 2020MMNL started with pid=21, OS id=30146734 lmon registered with NM - instance number 2 (internal mem no 1)Reconfiguration started (old inc 0, new inc 4)ASM instance List of instances: 1 2 (myinst: 2) Global Resource Directory frozen* allocate domain 0, invalid = TRUE KSXP IPC protocol is incompatible with instance 1Errors in file /app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_28180628.trc:ORA-27300: OS system dependent operation:config_check failed with status: 0ORA-27301: OS failure message: Error 0ORA-27302: failure occurred at: skgxpvalpidORA-27303: additional information: Remote port MTU does not match local MTU. [local: 9000, remote: 1500] (169.254.1XX.XX)KSXP IPC protocol is incompatible with instance 1Errors in file /app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_28180628.trc:ORA-27300: OS system dependent operation:config_check failed with status: 0ORA-27301: OS failure message: Error 0ORA-27302: failure occurred at: skgxpvalpidORA-27303: additional information: Remote port MTU does not match local MTU. [local: 9000, remote: 1500] (169.254.1XX.XX) Communication channels reestablished
根据这些便可发现和 MOS （Doc ID 947223.1）所说一致了，MTU 不一致导致主机重启了。
当查看主机二的 MTU 时便惊奇的发现居然是 9000 ，而节点一均是 1500.这个就需要修改了，MTU 参数修改需要重启才会永久生效。所以需要把 en2 和 en3 改成 1500 才行。 其他类型的系统查看如下：
On AIX:# lsattr -El <adapter>Example:[celcaix4]/usr/sbin> lsattr -El en5alias4 IPv4 Alias including Subnet Mask Truealias6 IPv6 Alias including Prefix Length Truearp on Address Resolution Protocol (ARP) Trueauthority Authorized Users Truebroadcast Broadcast Address Truemtu 1500 Maximum IP Packet Size for This Device True <(== HEREnetaddr 10.xxx.xxx.91 Internet Address Truenetaddr6 IPv6 Internet Address Truenetmask 255.xxx.xxx.0 Subnet Mask Trueprefixlen Prefix Length for IPv6 Internet Address Trueremmtu 576 Maximum IP Packet Size for REMOTE Networks Truerfc1323 Enable/Disable TCP RFC 1323 Window Scaling Truesecurity none Security Level Truestate up Current Interface Status Truetcp_mssdflt Set TCP Maximum Segment Size Truetcp_nodelay Enable/Disable TCP_NODELAY Option Truetcp_recvspace Set Socket Buffer Space for Receiving Truetcp_sendspace Set Socket Buffer Space for Sending TrueOn Linux:[root@asmstorage ~]# ifconfig eth0eth0 Link encap:Ethernet HWaddr 00:13:xx:xx:xx:46inet addr:10.xxx.xxx.151 Bcast:10.xxx.xxx.255 Mask:255.xxx.xxx.0inet6 addr: fe80::213:xxxx:xxxx:a146/64 Scope:LinkUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 <(== HERERX packets:8357638 errors:0 dropped:0 overruns:0 frame:0TX packets:166322 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:776382023 (740.4 MiB) TX bytes:42281847 (40.3 MiB)Interrupt:169On Solaris:# ifconfig -alo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.xxx.xxx.1 netmask ff000000eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index <(== HEREinet 10.xxx.xxx.149 netmask fffff800 broadcast 10.xxx.xxx.255ether 0:3:xx:xx:xx:d5
解决问题
使用如下命令临时设置生效，重启方可永久生效。
ifconfig eth2 mtu 1500
ifconfig eth3 mtu 1500
当修改完之后，重启数据库监控了半小时，便再也没有发生过主机宕机数据库宕机了。至于为何不一致了由于已是晚上当时就没有深究了。
到这里就算解决了，如果本文对你有帮助，请点击右下角“在看”与转发，欢迎大家在评论区展开互动。
推荐阅读：144页！分享珍藏已久的数据库技术年刊
点击下图查看更多 ↓
云和恩墨大讲堂 | 一个分享交流的地方
长按，识别二维码，加入万人交流社群
请备注：云和恩墨大讲堂
点个“在看”
你的喜欢会被看到❤