最近某次进行压力测试时，需要安装最新的PSU补丁，现场的同事安装完PSU补丁后，结果数据库集群的其中一个实例无法启动。以下为远程处理的过程，简要记录之。
1、查看集群当前状态：
[root@nodedb01 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.LISTENER.lsnr
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.TEST.dg
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.asm
ONLINE ONLINE nodedb01 Started
ONLINE ONLINE nodedb02 Started
ora.gsd
OFFLINE OFFLINE nodedb01
OFFLINE OFFLINE nodedb02
ora.net1.network
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.ons
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE nodedb02
ora.cvu
1 ONLINE ONLINE nodedb02
ora.nodedb01.vip
1 ONLINE ONLINE nodedb01
ora.nodedb02.vip
1 ONLINE ONLINE nodedb02
ora.oc4j
1 ONLINE ONLINE nodedb02
ora.scan1.vip
1 ONLINE ONLINE nodedb02
ora.test.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE nodedb02 Open
可见，整个集群状态都是正常的，仅仅是节点1的数据库实例异常，无法启动。
2、尝试手动启动数据库实例，但查看alert日志：
SQL> startup
ORA-03113: end-of-file on communication channel
SQL>
手动启动实例时，报3113错误，同时数据库的alert日志提示：
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 96
Number of processor cores in the system is 48
Number of processor sockets in the system is 2
alert日志中只有这些信息，启动数据库实例的操作就已经失败。这说明还没有开始读取spfile文件就已经出故障了。
3、检查节点1的补丁信息：
[oracle@nodedb01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.21
Copyright (c) 2019, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version : 11.2.0.3.21
OUI version : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2019-09-06_17-48-51PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2019-09-06_17-48-51PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: nodedb01
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Database 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch 29141201 : applied on Fri Sep 06 16:08:23 CST 2019
Unique Patch ID: 22724549
Patch description: "OCW Patch Set Update : 11.2.0.4.190416 (29141201)"
Created on 4 Mar 2019, 09:09:03 hrs PST8PDT
Bugs fixed:
18328800, 19270660, 18691572, 18508710, 20038451, 21251192, 22162062
20365005, 21232394, 17336871, 17387214, 17750548, 17617807, 14497275
20219458, 17733927, 18180541, 23757020, 18962892, 17292250, 17378618
16759171, 20110156, 17843489, 17065496, 13991403, 21694632, 18419139
17273020, 22762046, 17336884, 17155238, 17336889, 18261183, 18053580
20012766, 21245437, 20218012, 17013634, 17886392, 20995001, 24422155
17039197, 16317771, 17947785, 10052729, 20340620, 22353346, 16237657
20317221, 18199185, 15917869, 18399991, 20186278, 18024089, 17374271
16849642, 20246071, 20746251, 14270845, 23621494, 20552947, 18414137
18882642, 17001914, 29323944, 17927970, 14378120, 16346413, 15986647
17336898, 18068871, 21222147, 16206997, 18143836, 21982225, 25490238
19168690, 18343490, 20235511, 21875360, 16613232, 19276791, 17722664
20440643, 12928658, 18226143, 18520351, 16249829, 18952577, 17172091
18229842, 16076412, 18265482, 20676340, 17818075, 20091753, 18231837
14373486, 17483479, 20136892, 20551654, 18120545, 18729166, 13843841
17405302, 21225209, 18709496, 18330979, 18744838, 14525998, 18187697
17087371, 20531190, 20598625, 14385860, 18348155, 19479503, 12928592
17516024, 18370031, 17764053, 19272663, 17551223, 14671408, 18272135
14207615, 21255373, 17500165, 18875012, 14769643, 18464784, 25656952
19558324, 18848125, 19241857, 17955615, 14851828, 14693336, 20315294
16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 17531342
21327402, 26546632, 19398098, 17159489, 17640316, 13823394, 16543190
22024217, 17983675, 20795241, 28973538, 17481314, 18346135, 17598201
16281493, 15986311, 17208793, 19601468, 18999857, 18700935, 14076173
18428146, 16709532, 17435488, 18352845, 18352846, 20408163, 17592037
19616601, 17387779, 17391726, 14777968, 15851860, 16206882, 20141091
15832129, 17305100, 20175174, 21113068, 19885321, 16901346, 17985714
25591658, 18536826, 17780903, 18752378, 18946768, 16876500, 16875342
17769597, 19955755, 16429265, 18336452, 23186035, 17273003, 17209968
25484507, 16988311, 19319357, 20094984, 17046460, 17059927, 18053631
16867761, 18774591, 21442094, 20235486, 19359787, 15869775, 17447588
19642566, 21152052, 16798862, 15920201
--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@nodedb01 ~]$
与正常的节点2对比发现，节点1的补丁不完整，只是显示了一部分补丁信息。
4、非常怀疑是补丁安装异常，接着检查了oracle二进制文件的权限：
[oracle@nodedb01 ~]$ cd $ORACLE_HOME/bin
[oracle@nodedb01 bin]$ ll oracle
-rwsr-s--x 1 oracle oinstall 239885712 Sep 6 17:54 oracle
[oracle@nodedb01 bin]$
oracle二进制文件的属组竟然为oinstall。 至此，故障的原因基本已经找到。
5、手动修改属组：
[root@nodedb01 etc]# su - grid
Last login: Fri Sep 6 17:57:38 CST 2019
[grid@nodedb01 ~]$ echo $ORACLE_HOME
/u01/app/11.2.0/grid
[grid@nodedb01 ~]$ /u01/app/11.2.0/grid/bin/setasmgidwrap -o /u01/app/oracle/product/11.2.0/db_1/bin/oracle
[grid@nodedb01 ~]$ exit
6、再次检查补丁情况：
[root@nodedb01 etc]# su - oracle
Last login: Fri Sep 6 18:02:47 CST 2019
[oracle@nodedb01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.21
Copyright (c) 2019, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version : 11.2.0.3.21
OUI version : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2019-09-06_18-03-06PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2019-09-06_18-03-06PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: nodedb01
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Database 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.
Interim patches (2) :
Patch 29141056 : applied on Fri Sep 06 17:53:33 CST 2019
Unique Patch ID: 22768427
Patch description: "Database Patch Set Update : 11.2.0.4.190416 (29141056)"
Created on 17 Feb 2019, 23:52:26 hrs PST8PDT
Sub-patch 28729262; "Database Patch Set Update : 11.2.0.4.190115 (28729262)"
Sub-patch 28204707; "Database Patch Set Update : 11.2.0.4.181016 (28204707)"
Sub-patch 27734982; "Database Patch Set Update : 11.2.0.4.180717 (27734982)"
Sub-patch 27338049; "Database Patch Set Update : 11.2.0.4.180417 (27338049)"
Sub-patch 26925576; "Database Patch Set Update : 11.2.0.4.180116 (26925576)"
Sub-patch 26392168; "Database Patch Set Update : 11.2.0.4.171017 (26392168)"
Sub-patch 26609445; "Database Patch Set Update : 11.2.0.4.170814 (26609445)"
Sub-patch 25869727; "Database Patch Set Update : 11.2.0.4.170718 (25869727)"
Sub-patch 24732075; "Database Patch Set Update : 11.2.0.4.170418 (24732075)"
Sub-patch 24006111; "Database Patch Set Update : 11.2.0.4.161018 (24006111)"
Sub-patch 23054359; "Database Patch Set Update : 11.2.0.4.160719 (23054359)"
Sub-patch 22502456; "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
Sub-patch 21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch 21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
17184721, 21174504, 21538558, 16091637, 18092127, 17381384, 15979965
20671094, 16731148, 16314254, 13837378, 18441944, 17291347, 23105538
17835048, 28254374, 13558557, 21656630, 21842740, 17008068, 18382302
17201159, 25427662, 17853498, 21197626, 20717359, 17246576, 18356166
18681862, 18440047, 20569094, 20031873, 16875449, 20387265, 19788842
17296856, 21330264, 14010183, 17648596, 18886413, 17025461, 17551063
24719736, 17267114, 21063322, 22507210, 17912217, 17889583, 18202441
17040764, 17478145, 16524926, 25655390, 19358317, 22730454, 22148226
18747196, 26544823, 18641419, 17036973, 18948177, 17811789, 16542886
14285317, 18009564, 17359610, 16618694, 8322815, 16832076, 18247991
16692232, 22507234, 17570240, 13871092, 24624166, 26631046, 19429927
24348685, 17848897, 17441661, 14034426, 17465741, 20273319, 19207156
16596890, 18510194, 17437634, 21343897, 20506706, 28849751, 18339044
21453153, 17951233, 21795111, 22321741, 18430495, 21787056, 22380919
19469538, 19692824, 20506715, 17811429, 17903598, 19721304, 11786053
18230522, 19554106, 19458377, 21281607, 17612828, 6599380, 22092979
17040527, 22321756, 17811438, 18641461, 18682983, 14657740, 25635149
13364795, 19490948, 21387964, 17346671, 17588480, 22351572, 18235390
18849970, 26474853, 17889549, 19309466, 20627866, 23008056, 16472716
24908321, 20134113, 25775213, 20596234, 18331850, 18641451, 20882568
17019356, 17344412, 19461270, 21179898, 17546761, 24842886, 17231779
14521849, 18203835, 18203838, 18964939, 18203837, 17313525, 22195457
18139690, 16837842, 14106803, 22296366, 17842825, 22657942, 21352646
16360112, 22594718, 20657441, 22195441, 17389192, 26198926, 14565184
19781326, 17019345, 17205719, 18740837, 18440095, 14764829, 14354737
22195448, 17019086, 13944971, 16571443, 21868720, 17186905, 17080436
18673342, 22905130, 17027426, 27374796, 19972569, 19972568, 20144308
19972566, 17282229, 19972564, 16870214, 16410570, 21629064, 19615136
21354456, 26039623, 17390431, 18762750, 23007241, 16613964, 17957017
18098207, 17484762, 18471685, 19730508, 18264060, 21538485, 17323222
17754782, 17600719, 18317531, 17852463, 17596908, 17655634, 18166013
16228604, 20074391, 27053456, 24790914, 19972570, 20856766, 18090142
19891090, 18996843, 16042673, 19854503, 17835627, 22901797, 20334344
17393683, 20861693, 18000422, 17551709, 26575788, 23315889, 20506699
19006849, 18277454, 18456514, 19174430, 17258090, 17174582, 25654936
17242746, 27097854, 16399083, 17824637, 21132297, 22465352, 17762296
22168163, 18604692, 17397545, 16450169, 12364061, 20067212, 18856999
19211724, 19463893, 27351628, 19463897, 21343775, 17853456, 18373438
18673304, 20004021, 28000269, 26030218, 21668627, 16194160, 17477958
23140259, 16538760, 12982566, 24570598, 20828947, 27255377, 18259031
20296213, 21425496, 28855981, 18293054, 17610798, 19699191, 23065323
17311728, 18135678, 18774543, 23294548, 16785708, 10136473, 24560906
19777862, 22551446, 17786518, 25879984, 18315328, 18334586, 12747740
18096714, 19032867, 21641760, 17390160, 18899974, 17232014, 20598042
26679352, 26245237, 16354467, 17484731, 18673325, 16422541, 18155762
19827973, 14015842, 22683225, 17726838, 18554871, 23177648, 18051556
20803583, 18282562, 21972320, 15990359, 17922254, 16855292, 16668584
21343838, 20299015, 17446237, 18043064, 18093615, 23713236, 17694209
17288409, 20475845, 18308268, 17274537, 13955826, 16934803, 17634921
17501491, 16315398, 23725036, 22683212, 17006183, 13829543, 18191164
26746894, 22809871, 17655240, 28819280, 18384391, 19393542, 21538567
16198143, 21847223, 25823754, 17892268, 20142975, 19584068, 17165204
25165496, 18604493, 21756699, 18508861, 18554763, 16901385, 21532755
18189036, 17443671, 17385178, 14829250, 17936109, 20925795, 20509482
17478514, 27441326, 16850630, 13951456, 16595641, 14054676, 15861775
21142837, 16912439, 17299889, 17297939, 23003979, 16833527, 18619917
17798953, 19697993, 17816865, 25914276, 18607546, 17571306, 21286665
16898135, 16819962, 17341326, 26910644, 17851160, 17586955, 20558005
19049453, 21051840, 17587063, 16956380, 18328509, 25423453, 14133975
29033139, 19718981, 18061914, 18522509, 17518652, 21051833, 20294666
18765602, 20860659, 18272672, 20324049, 18199537, 17332800, 13609098
22502493, 18384537, 14338435, 17945983, 16392068, 21067387, 17752995
21051862, 16863422, 17237521, 25505382, 18244962, 19544839, 24433711
24717859, 17156148, 18973907, 23026585, 17449815, 17877323, 18180390
17088068, 17037130, 20004087, 21422580, 19466309, 11733603, 25505371
18610915, 21051858, 18084625, 29027694, 18674024, 26243698, 21051852
18091059, 18306996, 25369547, 16306373, 17787259, 18193833, 19915271
20513399, 20631274, 26439748, 16344544, 25879656, 14692762, 18614015
22782647, 17346091, 18228645, 17721717, 13960236, 18685892, 18436307
11883252, 19888853, 21756677, 17891943, 19475971, 22353199, 16384983
19121551, 25634317, 27825893, 12816846, 17982555, 17761775, 17227073
22243719, 17265217, 25505394, 17071721, 16721594, 18262334, 21756661
17891946, 15913355, 17672719, 17602269, 17239687, 17042658, 25555252
17238511, 21059919, 17811456, 17284817, 17752121, 20879889, 28806384
21380789, 17394950, 17011832, 16579084, 22195465, 14602788, 28790634
18325460, 27567477, 26569225, 24476265, 24476274, 12611721, 18674465
16903536, 17006570, 19689979, 28076295, 16043574, 18783224, 24662775
16494615, 21526048, 17392698, 19197175, 16069901, 17811447, 27870645
28876684, 17308789, 24835538, 22195477, 17865671, 17343514, 19013183
17325413, 18316692, 16180763, 17348614, 14368995, 21983325, 17393915
16285691, 19211433, 20331945, 17883081, 17705023, 24316947, 17614227
19578350, 22195485, 14084247, 24975421, 26078387, 23115139, 13645875
16777840, 19727057, 14852021, 18744139, 18674047, 17716305, 19285025
18482502, 17622427, 19289642, 27534509, 25947799, 22195492, 14458214
20869721, 21172913, 17767676, 18723434, 25505407, 17786278, 19258504
17082983, 17365043, 21351877, 13498382, 18331812, 16065166, 25489607
16685417, 18031668, 22893153, 17551674, 16943711, 19272701, 21517440
25897615, 17649265, 13866822, 18094246, 24528741, 17783588, 14245531
17082359, 26007010, 18280813, 20448824, 23330119, 16268425, 19487147
25600421, 18018515, 17302277, 17215560, 24411921, 19271443, 25764020
14176370, 17016369, 20777150, 16756406, 23330124, 22977256, 20441797
19769489, 28100487, 17545847, 25093656, 18260550, 13853126, 17227277
17551699, 23536835, 25957038, 24652769, 20725343, 19207117, 9756271
18868646, 17614134, 26667023, 17546973, 19680952, 18704244, 26667015
17050888, 18273830, 18828868, 17360606, 16992075, 24563422, 17375354
12905058, 18362222, 21429602, 17254374, 26667032, 28364007, 27086138
17571039, 17468141, 18436647, 17570606, 17235750, 21168487, 17279227
16220077, 16929165
Patch 29141201 : applied on Fri Sep 06 16:08:23 CST 2019
Unique Patch ID: 22724549
Patch description: "OCW Patch Set Update : 11.2.0.4.190416 (29141201)"
Created on 4 Mar 2019, 09:09:03 hrs PST8PDT
Bugs fixed:
18328800, 19270660, 18691572, 18508710, 20038451, 21251192, 22162062
20365005, 21232394, 17336871, 17387214, 17750548, 17617807, 14497275
20219458, 17733927, 18180541, 23757020, 18962892, 17292250, 17378618
16759171, 20110156, 17843489, 17065496, 13991403, 21694632, 18419139
17273020, 22762046, 17336884, 17155238, 17336889, 18261183, 18053580
20012766, 21245437, 20218012, 17013634, 17886392, 20995001, 24422155
17039197, 16317771, 17947785, 10052729, 20340620, 22353346, 16237657
20317221, 18199185, 15917869, 18399991, 20186278, 18024089, 17374271
16849642, 20246071, 20746251, 14270845, 23621494, 20552947, 18414137
18882642, 17001914, 29323944, 17927970, 14378120, 16346413, 15986647
17336898, 18068871, 21222147, 16206997, 18143836, 21982225, 25490238
19168690, 18343490, 20235511, 21875360, 16613232, 19276791, 17722664
20440643, 12928658, 18226143, 18520351, 16249829, 18952577, 17172091
18229842, 16076412, 18265482, 20676340, 17818075, 20091753, 18231837
14373486, 17483479, 20136892, 20551654, 18120545, 18729166, 13843841
17405302, 21225209, 18709496, 18330979, 18744838, 14525998, 18187697
17087371, 20531190, 20598625, 14385860, 18348155, 19479503, 12928592
17516024, 18370031, 17764053, 19272663, 17551223, 14671408, 18272135
14207615, 21255373, 17500165, 18875012, 14769643, 18464784, 25656952
19558324, 18848125, 19241857, 17955615, 14851828, 14693336, 20315294
16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 17531342
21327402, 26546632, 19398098, 17159489, 17640316, 13823394, 16543190
22024217, 17983675, 20795241, 28973538, 17481314, 18346135, 17598201
16281493, 15986311, 17208793, 19601468, 18999857, 18700935, 14076173
18428146, 16709532, 17435488, 18352845, 18352846, 20408163, 17592037
19616601, 17387779, 17391726, 14777968, 15851860, 16206882, 20141091
15832129, 17305100, 20175174, 21113068, 19885321, 16901346, 17985714
25591658, 18536826, 17780903, 18752378, 18946768, 16876500, 16875342
17769597, 19955755, 16429265, 18336452, 23186035, 17273003, 17209968
25484507, 16988311, 19319357, 20094984, 17046460, 17059927, 18053631
16867761, 18774591, 21442094, 20235486, 19359787, 15869775, 17447588
19642566, 21152052, 16798862, 15920201
--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@nodedb01 ~]$
此时补丁信息显示正常。
7、再次手动启动节点1的数据库实例，一切恢复正常。
[root@nodedb01 etc]# crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.LISTENER.lsnr
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.TEST.dg
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.asm
ONLINE ONLINE nodedb01 Started
ONLINE ONLINE nodedb02 Started
ora.gsd
OFFLINE OFFLINE nodedb01
OFFLINE OFFLINE nodedb02
ora.net1.network
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
ora.ons
ONLINE ONLINE nodedb01
ONLINE ONLINE nodedb02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE nodedb02
ora.cvu
1 ONLINE ONLINE nodedb02
ora.nodedb01.vip
1 ONLINE ONLINE nodedb01
ora.nodedb02.vip
1 ONLINE ONLINE nodedb02
ora.oc4j
1 ONLINE ONLINE nodedb02
ora.scan1.vip
1 ONLINE ONLINE nodedb02
ora.test.db
1 ONLINE ONLINE nodedb01 Open
2 ONLINE ONLINE nodedb02 Open
[root@nodedb01 etc]#