Oracle 19c RAC 升级RU 从19.3 升级到19.6 操作手册（使用nonrolling 方式）
2020-03-14 16:09 935 0 原创 Oracle 19c
在之前的博客我们了解了Oracle 19c RAC的安装，以及单实例的RU升级，如下：
Linux 7.6 平台 Oracle 19c(19.3) RAC 安装手册 详细截图版
https://www.cndba.cn/dave/article/4073
Oracle 19c 单实例 RU 从19.3.0 升级到19.6.0 操作手册
https://www.cndba.cn/dave/article/4065
Linux 7.7 平台 Oracle 19c Data Guard 环境 升级RU 19.3 到19.6 操作手册
https://www.cndba.cn/dave/article/4068
Oracle 12.2 家族的补丁 RU 和 RUR 说明
https://www.cndba.cn/dave/article/4063
本篇我们看下19c RAC 平台如何升级RU.
1升级说明
从MOS上我们可以看到19.6的RU 有GI和DB 2个版本，但从GI Patch的readme上看，GI 的Patch 已经包含了DB HOME和GI HOME的补丁，只需要用root用户执行opatchauto命令打GI的补丁即可。
根据官方手册的说明，对于RU的升级，可以采用滚动升级的方式进行。依次在每个节点执行，不能同时在2个节点执行。 但是实际操作有问题，目前可行的操作是采用nonrolling的方式升级RU。
2具体升级过程
2.1 当前环境
[grid@www.cndba.cn_1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rac1] is [19.0.0.0.0]
[grid@www.cndba.cn_1 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]
[grid@www.cndba.cn_1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@www.cndba.cn_1 ~]$
[grid@www.cndba.cn_1 OPatch]$ ./opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
OPatch succeeded.
[grid@www.cndba.cn_1 OPatch]$
2.2 检查OPatch工具版本
安装19.6 的RU，OPatch的版本必须大于12.2.0.1.17. 最新版本的Opatch可以从MOS 6880880上下载。
[grid@www.cndba.cn_1 ~]$ cd $ORACLE_HOME/OPatch
[grid@www.cndba.cn_1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.17
OPatch succeeded.
[grid@www.cndba.cn_1 OPatch]$
因为opatch低版本打补丁的时候有bug，所以我这里直接将OPatch 升级到最新的12.2.0.1.19。
[oracle@www.cndba.cn_2 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.19
OPatch succeeded.
[oracle@www.cndba.cn_2 OPatch]$
12c opatchauto: Prerequisite check “CheckApplicable” failed (Doc ID 1937982.1)
OPatch升级可以参考我的博客：
Oracle 更新 OPatch 工具版本 的方法 说明
https://www.cndba.cn/cndba/dave/article/1353
2.3 验证Oracle Inventory的有效性
GI HOME 和DB HOME 都需要验证，分别使用grid和oracle用户执行如下命令，确保返回SUCCESS：
$ /OPatch/opatch lsinventory -detail -oh
GRID 用户：
[grid@www.cndba.cn_1 OPatch]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/19.3.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.3.0/grid/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/opatch2020-03-12_20-29-23PM_1.log
Lsinventory Output file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2020-03-12_20-29-23PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Grid Infrastructure 19c 19.0.0.0.0
There are 1 products installed in this Oracle Home.
Installed Products (102):
……
OPatch succeeded.
[grid@www.cndba.cn_1 OPatch]$
Oracle 用户：
[oracle@www.cndba.cn_1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.3.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.3.0/db_1/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatch/opatch2020-03-12_20-31-13PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2020-03-12_20-31-13PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Database 19c 19.0.0.0.0
There are 1 products installed in this Oracle Home.
Installed Products (128):
……
Patch Location in Inventory:
/u01/app/oracle/product/19.3.0/db_1/inventory/oneoffs/29517242
Patch Location in Storage area:
/u01/app/oracle/product/19.3.0/db_1/.patch_storage/29517242_Apr_17_2019_23_27_10
--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@www.cndba.cn_1 ~]$
2.4 下载19.6 的RU
直接从MOS下载。
Quick Reference to Patch Numbers for Database/GI PSU, SPU(CPU), Bundle Patches and Patchsets (文档 ID 1454618.1)
[root@www.cndba.cn_1 30501910]# ll
total 132
drwxr-x---. 5 oracle oinstall 62 Jan 7 13:25 30489227
drwxr-x---. 5 oracle oinstall 62 Jan 7 13:26 30489632
drwxr-x---. 5 oracle oinstall 81 Jan 7 13:22 30557433
drwxr-x---. 4 oracle oinstall 48 Jan 7 13:26 30655595
drwxr-x---. 2 oracle oinstall 4096 Jan 7 13:25 automation
-rw-rw-r--. 1 oracle oinstall 5054 Jan 7 21:43 bundle.xml
-rw-r--r--. 1 oracle oinstall 122266 Jan 7 21:32 README.html
-rw-r--r--. 1 oracle oinstall 0 Jan 7 13:25 README.txt
[root@www.cndba.cn_1 30501910]#
2.5 检查Patch 是否冲突
19.6 的GI RU里包含GI和DB的补丁，需要分别进行检测，在检测之前需要先修改权限，否则会报错。
GRID 用户执行：
chown grid:oinstall /u01/software/30501910 -R
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30557433
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30489227
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30489632
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30655595
[grid@www.cndba.cn_1 ~]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30655595
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.3.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.3.0/grid/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/opatch2020-03-12_20-43-04PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
Oracle 用户执行：
chown oracle:oinstall /u01/software/30501910 -R
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30557433
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30489227
[oracle@www.cndba.cn_1 ~]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/software/30501910/30489227
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.3.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.3.0/db_1/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatch/opatch2020-03-12_20-43-37PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
2.6 检查系统空间
在打Patch之前需要先确保ORACLE_HOME 所在的文件系统有足够的空闲空间，可以执行如下命令来检查。
GRID用户操作：
创建/tmp/patch_list_gihome.txt 文件，并添加如下内容：
[grid@www.cndba.cn_1 ~]$ cat /tmp/patch_list_gihome.txt
/u01/software/30501910/30557433
/u01/software/30501910/30489227
/u01/software/30501910/30489632
/u01/software/30501910/30655595
[grid@www.cndba.cn_1 ~]$
注意这里内容与官方有所不同，按readme文档会报如下错误：
Oracle OPatch 执行 返回 This command doesn’t support System Patch. 解决方法
https://www.cndba.cn/dave/article/4074
运行opatch命令检查GI HOME下是否有足够的空间：
[root@www.cndba.cn_1 software]# chown grid:oinstall /u01/software/30501910 -R
[grid@www.cndba.cn_1 ~]$ $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.3.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.3.0/grid/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/opatch2020-03-12_21-54-25PM_1.log
Invoking prereq "checksystemspace"
Prereq "checkSystemSpace" passed.
OPatch succeeded.
[grid@www.cndba.cn_1 ~]$
ORACLE 用户操作：
创建文件/tmp/patch_list_dbhome.txt并添加如下内容：
[oracle@www.cndba.cn_1 ~]$ cat /tmp/patch_list_dbhome.txt
/u01/software/30501910/30557433
/u01/software/30501910/30489227
运行opatch命令检查DB HOME下是否有足够的空间：
[root@www.cndba.cn_1 software]# chown oracle:oinstall /u01/software/30501910 -R
[oracle@www.cndba.cn_1 ~]$ $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.3.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.3.0/db_1/oraInst.loc
OPatch version : 12.2.0.1.17
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatch/opatch2020-03-12_20-58-02PM_1.log
Invoking prereq "checksystemspace"
Prereq "checkSystemSpace" passed.
OPatch succeeded.
[oracle@www.cndba.cn_1 ~]$
Oracle 用户执行正常。
2.7 安装前分析
在每个节点执行analyze：
/u01/app/19.0.0/grid/OPatch/opatchauto apply /u01/app/grid/soft/30501910 -analyze
“-analyze”选项可以模拟OPatchauto apply，提前检查所有检查项目，但是运行”-analyze”选项不会真正改变系统。
[root@www.cndba.cn_2 software]# /u01/app/19.3.0/grid/OPatch/opatchauto apply /tmp/30501910 -analyze
OPatchauto session is initiated at Fri Mar 13 14:20:49 2020
System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-03-13_02-21-06PM.log.
Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-03-13_02-23-32PM.log
The id for this session is AT3D
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/19.3.0/grid
Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Analysis for applying patches has completed successfully:
Host:rac2
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
==Following patches were SKIPPED:
Patch: /tmp/30501910/30489632
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/30501910/30655595
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /tmp/30501910/30489227
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_14-24-10PM_1.log
Patch: /tmp/30501910/30557433
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_14-24-10PM_1.log
Host:rac2
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /tmp/30501910/30489227
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_14-24-10PM_1.log
Patch: /tmp/30501910/30489632
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_14-24-10PM_1.log
Patch: /tmp/30501910/30655595
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_14-24-10PM_1.log
Patch: /tmp/30501910/30557433
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_14-24-10PM_1.log
OPatchauto session completed at Fri Mar 13 14:30:24 2020
Time taken to complete the session 9 minutes, 36 seconds
[root@www.cndba.cn_2 software]#
2.8 执行opatchauto 工具应用RU
在DB HOME的版本和GI HOME的版本一致的情况下，可以使用root用户执行opatchauto命令一次对GI 和 DB HOME进行Patch操作。如果GI 和 DB Home 版本不同，可以使用opatchauto 分别进行patch动作。
注意一点： 这里的RU补丁，一定要用GRID用户来解压缩，否则执行opatchauto的时候回报OPatch Prerequisite check “CheckApplicable” failed的错误。
其实是根据readme文档的说明，可以一次性打GI 和DB 的RU，但按文档操作，出现了很多问题。 一直没彻底解决的就是：
[Mar 13, 2020 3:54:58 PM] [SEVERE] OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/tmp/30501910/30489227/files/bin/crsd.bin' to '/u01/app/19.3.0/grid/bin/crsd.bin'...
Copy failed from '/tmp/30501910/30489227/files/bin/cssdagent' to '/u01/app/19.3.0/grid/bin/cssdagent'...
直接一次性打GI 和 DB HOME 中遇到的问题，参考我的博客：
Oracle 19c RAC 环境升级 19.6 RU OPatch Prerequisite check “CheckApplicable” failed 解决方法
https://www.cndba.cn/dave/article/4081
所以，对于RU的升级，建议按照nonrolling的方式进行，虽然要停机，但可以保证升级的成功。
对GI HOME 进行patch：
opatchauto apply /28660077 -oh -nonrolling
对一个或者多个DB HOME 进行patch：
opatchauto apply /28660077 -oh , -nonrolling
2.8.1打GI 的RU
先在节点1执行：
[root@www.cndba.cn_1 software]# export PATH=$PATH:/u01/app/19.3.0/grid/OPatch
[root@www.cndba.cn_1 software]# chown grid:oinstall 30501910 -R
[root@www.cndba.cn_1 software]# chmod 777 30501910 -R
[root@www.cndba.cn_1 software]# opatchauto apply /tmp/30501910 -oh /u01/app/19.3.0/grid -nonrolling
[root@www.cndba.cn_1 lib]# opatchauto apply /tmp/30501910 -oh /u01/app/19.3.0/grid -nonrolling
OPatchauto session is initiated at Fri Mar 13 20:37:31 2020
System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-03-13_08-37-39PM.log.
Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-03-13_08-37-48PM.log
The id for this session is AFS1
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid
Patch applicability verified successfully on home /u01/app/19.3.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:rac1
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /tmp/30501910/30489227
Reason: This patch is already been applied, so not going to apply again.
Patch: /tmp/30501910/30489632
Reason: This patch is already been applied, so not going to apply again.
Patch: /tmp/30501910/30655595
Reason: This patch is already been applied, so not going to apply again.
Patch: /tmp/30501910/30557433
Reason: This patch is already been applied, so not going to apply again.
OPatchauto session completed at Fri Mar 13 20:38:10 2020
Time taken to complete the session 0 minute, 40 seconds
[root@www.cndba.cn_1 lib]#
节点2执行：
[root@www.cndba.cn_2 cfgtoollogs]# crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac2 STABLE
ora.MGMT.GHCHKPT.advm
OFFLINE OFFLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac2 STABLE
ora.helper
OFFLINE OFFLINE rac2 IDLE,STABLE
ora.mgmt.ghchkpt.acfs
OFFLINE OFFLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.MGMT.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac2 169.254.21.159 192.1
68.222.181,STABLE
ora.OCR.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cndba.db
1 ONLINE OFFLINE STABLE
2 ONLINE OFFLINE rac2 Instance Shutdown,ST
ARTING
ora.cvu
1 ONLINE ONLINE rac2 STABLE
ora.mgmtdb
1 ONLINE OFFLINE rac2 Instance Shutdown,ST
ARTING
ora.qosmserver
1 ONLINE ONLINE rac2 STABLE
ora.rac1.vip
1 ONLINE INTERMEDIATE rac2 FAILED OVER,STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
[root@www.cndba.cn_2 cfgtoollogs]# opatchauto apply /tmp/30501910 -oh /u01/app/19.3.0/grid -nonrolling
OPatchauto session is initiated at Fri Mar 13 21:32:49 2020
System initialization log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-03-13_09-32-56PM.log.
Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2020-03-13_09-33-42PM.log
The id for this session is I2XJ
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.3.0/grid
Patch applicability verified successfully on home /u01/app/19.3.0/grid
Bringing down CRS service on home /u01/app/19.3.0/grid
CRS service brought down successfully on home /u01/app/19.3.0/grid
Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid
Starting CRS service on home /u01/app/19.3.0/grid
CRS service started successfully on home /u01/app/19.3.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:rac2
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/30501910/30489227
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_21-37-51PM_1.log
Patch: /tmp/30501910/30489632
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_21-37-51PM_1.log
Patch: /tmp/30501910/30557433
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_21-37-51PM_1.log
Patch: /tmp/30501910/30655595
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_21-37-51PM_1.log
OPatchauto session completed at Fri Mar 13 22:13:32 2020
Time taken to complete the session 40 minutes, 44 seconds
[root@www.cndba.cn_2 cfgtoollogs]#
注意：这里可能会遇到如下错误：
Oracle 19c opatchauto Failed to serialize remote node list 错误解决方法
https://www.cndba.cn/dave/article/4075
2.8.2打DB 的RU
2.8.2.1先打节点1：（测试RU 目录权限还是grid）
[root@www.cndba.cn_1 ~]# export PATH=$PATH:/u01/app/oracle/product/19.3.0/db_1
[root@www.cndba.cn_1 ~]# /u01/app/oracle/product/19.3.0/db_1/OPatch/opatchauto apply /tmp/30501910 -oh /u01/app/oracle/product/19.3.0/db_1 -nonrolling
OPatchauto session is initiated at Fri Mar 13 22:23:36 2020
System initialization log file is /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-03-13_10-23-51PM.log.
Session log file is /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-03-13_10-24-02PM.log
The id for this session is NXT9
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
No step execution required.........
Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:rac1
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /tmp/30501910/30489632
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/30501910/30655595
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/30501910/30489227
Reason: This patch is already been applied, so not going to apply again.
Patch: /tmp/30501910/30557433
Reason: This patch is already been applied, so not going to apply again.
OPatchauto session completed at Fri Mar 13 22:24:33 2020
Time taken to complete the session 0 minute, 58 seconds
[root@www.cndba.cn_1 ~]#
因此之前测试的时候，节点1已经成功打上了，所以这里跳过了。
2.8.2.2在打节点2：
[root@www.cndba.cn_2 ~]# export PATH=$PATH:/u01/app/oracle/product/19.3.0/db_1
[root@www.cndba.cn_2 ~]# /u01/app/oracle/product/19.3.0/db_1/OPatch/opatchauto apply /tmp/30501910 -oh /u01/app/oracle/product/19.3.0/db_1 -nonrolling
OPatchauto session is initiated at Fri Mar 13 22:49:14 2020
System initialization log file is /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-03-13_10-49-21PM.log.
Session log file is /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-03-13_10-50-20PM.log
The id for this session is GXFF
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.3.0/db_1
Preparing to bring down database service on home /u01/app/oracle/product/19.3.0/db_1
Successfully prepared home /u01/app/oracle/product/19.3.0/db_1 to bring down database service
Bringing down database service on home /u01/app/oracle/product/19.3.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.3.0/db_1
Performing prepatch operation on home /u01/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1
Start applying binary patch on home /u01/app/oracle/product/19.3.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1
Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db_1
Starting database service on home /u01/app/oracle/product/19.3.0/db_1
Database service successfully started on home /u01/app/oracle/product/19.3.0/db_1
Preparing home /u01/app/oracle/product/19.3.0/db_1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db_1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:rac2
RAC Home:/u01/app/oracle/product/19.3.0/db_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /tmp/30501910/30489632
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/30501910/30655595
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/30501910/30489227
Reason: This patch is already been applied, so not going to apply again.
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/30501910/30557433
Log: /u01/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-03-13_22-51-09PM_1.log
Patching session reported following warning(s):
_________________________________________________
[WARNING] The database instance 'cndba2' from '/u01/app/oracle/product/19.3.0/db_1', in host'rac2' is not running. SQL changes, if any, will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.
OPatchauto session completed at Fri Mar 13 23:00:17 2020
Time taken to complete the session 11 minutes, 4 seconds
[root@www.cndba.cn_2 ~]#
这里可能会遇到如下错误：
Oracle 19c RU opatchauto oui-patch.xml (Permission denied) 解决方法
https://www.cndba.cn/dave/article/4076
Oracle 19c 打19.6 的RU opatchauto 报错OPatchException: Unable to create patchObject 解决方法
https://www.cndba.cn/dave/article/4077
2.9 加载修改后的SQL到数据库
安装补丁之后，还需要将有变化的SQL加载到数据库中，这里可以直接运行Datapatch工具将这些修改的SQL重新加载到数据库中，RAC环境，只需要在一个节点执行就可以了。
根据readme的说明，操作步骤如下：
sqlplus /nolog
SQL> Connect / as sysdba
SQL> startup
SQL> alter pluggable database all open;
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose
datapatch命令只对打开的数据库生效，所有Oracle建议在执行该命令之前将CDB和所有的PDB都打开，一次更新掉。 但如果有部分PDB没有打开，也可以在打开之后，重新运行datapatch命令并制定pdb实例名，比如： ./datapatch -pdbs dave
这种情况可以参考之前的博客：
Oracle 18c 单实例 RUR 从18.3.0 升级到18.3.1 操作手册
https://www.cndba.cn/dave/article/3138
我们这里将所有的PDB 都打开后在执行datapatch工具。
启动数据库：
[oracle@www.cndba.cn_2 oneoffs]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Mar 13 23:16:06 2020
Version 19.6.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2466250400 bytes
Fixed Size 9137824 bytes
Variable Size 738197504 bytes
Database Buffers 1711276032 bytes
Redo Buffers 7639040 bytes
Database mounted.
Database opened.
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 DAVE READ WRITE NO
SQL>
运行datapatch：这里我们在节点2上执行：
[oracle@www.cndba.cn_2 oneoffs]$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 19.6.0.0.0 Production on Fri Mar 13 23:19:09 2020
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6158_2020_03_13_23_19_09/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
No interim patches found
Current state of release update SQL patches:
Binary registry:
19.6.0.0.0 Release_Update 191217155004: Installed
PDB CDB$ROOT:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 12-MAR-20 04.12.02.535145 PM
PDB DAVE:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 12-MAR-20 04.31.06.276026 PM
PDB PDB$SEED:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 12-MAR-20 04.31.06.276026 PM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED DAVE
No interim patches need to be rolled back
Patch 30557433 (Database Release Update : 19.6.0.0.200114 (30557433)):
Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.6.0.0.0 Release_Update 191217155004
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 3
Validating logfiles...done
Patch 30557433 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_CNDBA_CDBROOT_2020Mar13_23_21_42.log (no errors)
Patch 30557433 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_CNDBA_PDBSEED_2020Mar13_23_32_19.log (no errors)
Patch 30557433 apply (pdb DAVE): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply_CNDBA_DAVE_2020Mar13_23_32_21.log (no errors)
SQL Patching tool complete on Fri Mar 13 23:42:45 2020
2.10 处理无效对象
最后一步就是处理无效对象，因为之前datapatch命令会加载SQL，这个过程可能会产生无效对象。 可以执行@utlrp.sql脚本处理这些无效对象。
[oracle@www.cndba.cn_2 oneoffs]$ cd $ORACLE_HOME/rdbms/admin
[oracle@www.cndba.cn_2 admin]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Mar 13 23:44:38 2020
Version 19.6.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.6.0.0.0
SQL> @utlrp.sql
Session altered.
2.11 验证版本
节点2：
SQL> col action for a15
SQL> col status for a15
SQL> select PATCH_ID,PATCH_TYPE,ACTION,STATUS,TARGET_VERSION from dba_registry_sqlpatch;
PATCH_ID PATCH_TYPE ACTION STATUS TARGET_VERSION
---------- -------------------- --------------- --------------- ------------------------------
29517242 RU APPLY SUCCESS 19.3.0.0.0
30557433 RU APPLY SUCCESS 19.6.0.0.0
SQL>
[grid@www.cndba.cn_2 ~]$ /u01/app/19.3.0/grid/bin/kfod op=patches
---------------
List of Patches
===============
30489227
30489632
30557433
30655595
[grid@www.cndba.cn_2 ~]$ /u01/app/19.3.0/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2701864972
[grid@www.cndba.cn_2 ~]$
[grid@www.cndba.cn_2 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/19.3.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.3.0/grid/oraInst.loc
OPatch version : 12.2.0.1.19
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/opatch2020-03-13_23-49-10PM_1.log
Lsinventory Output file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2020-03-13_23-49-10PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac2
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Grid Infrastructure 19c 19.0.0.0.0
There are 1 products installed in this Oracle Home.
Interim patches (4) :
Patch 30655595 : applied on Fri Mar 13 21:48:16 CST 2020
Unique Patch ID: 23265660
Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (30655595)"
Created on 12 Dec 2019, 04:55:54 hrs PST8PDT
Bugs fixed:
29286300
Patch 30557433 : applied on Fri Mar 13 21:43:14 CST 2020
Unique Patch ID: 23305305
Patch description: "Database Release Update : 19.6.0.0.200114 (30557433)"
Created on 6 Jan 2020, 19:07:34 hrs PST8PDT
Bugs fixed:
30545281, 8476681, 14735102, 17428816, 19080742, 19697993, 20313356
21374587, 21965541, 23296836, 23606241, 24687075, 25756945, 25806201
25883179, 25986062, 25997810, 26476244, 26611353, 26739322, 26777814
26872233, 27036163, 27044169, 27101798, 27126938, 27195935, 27244999
27254335, 27359766, 27369515, 27406105, 27411022, 27423500, 27439716
……
节点1：
[grid@www.cndba.cn_1 ~]$ /u01/app/19.3.0/grid/bin/kfod op=patches
---------------
List of Patches
===============
30489227
30489632
30557433
30655595
[grid@www.cndba.cn_1 ~]$ /u01/app/19.3.0/grid/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2701864972
[grid@www.cndba.cn_1 ~]$
2.12 小结说明
之前计划使用滚动升级的方式进行，但测试一直过不去，所以在节点上按 nonrolling的方式进行了。 但是节点因为权限的问题导致CRS 无法启动。 在解决节点1的权限问题这块花了不少时间，最终解决方法是将节点2的/u01 目录打包到了节点2，然后重新执行了root.sh 脚本，详细过程参考我的博客：
Oracle 19c 升级19.6 RU 导致权限异常 gipcInternalConnectSync: failed sync request 解决方法
https://www.cndba.cn/dave/article/4080
[root@www.cndba.cn_1 crs]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.MGMT.GHCHKPT.advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.helper
OFFLINE OFFLINE rac1 IDLE,STABLE
OFFLINE OFFLINE rac2 IDLE,STABLE
ora.mgmt.ghchkpt.acfs
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.MGMT.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac2 169.254.21.159 192.1
68.222.181,STABLE
ora.OCR.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cndba.db
1 ONLINE ONLINE rac1 Open,HOME=/u01/app/o
racle/product/19.3.0
/db_1,STABLE
2 ONLINE ONLINE rac2 Open,HOME=/u01/app/o
racle/product/19.3.0
/db_1,STABLE
ora.cvu
1 ONLINE ONLINE rac2 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac2 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE rac2 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
[root@www.cndba.cn_1 crs]#