（一）环境介绍
esxi6.0 ，VMware vSphere Client6.0，linux 版本Centos7.6（最小化安装）Oracle 版本 oracle 11g 11.2.0.4，下载链接在博客尾页。
（二）虚拟化环境搭建
资源下载：
链接：https://pan.baidu.com/s/1AjXAPi2OBCm3pA88EtDi0Q
提取码：1111
个别机型去要第三方驱动，而在官方又找不到时候，可以选择ESXi-Customizer加载第三方驱动
工具下载链接如下：
链接：https://pan.baidu.com/s/1Ta5ZZK6KtSe05Q1LyPtU6A
提取码：1111
（三） 配置共享存储
esxi配置ssh远程连接,我这里配置的与主机一起启动，如果ssh没有启动，点击启动即可。
打开ssh连接工具，这里我使用的xshell，截图如下，创建共享磁盘时候忘记截图了，就截图做完毕之后的如下
创建共享磁盘命令如下，（一共是4块，）
vmkfstools -c 10240m -a lsilogic -d eagerzeroedthick mysharedisk_01.vmdk vmkfstools -c 10240m -a lsilogic -d eagerzeroedthick mysharedisk_02.vmdk vmkfstools -c 10240m -a lsilogic -d eagerzeroedthick mysharedisk_03.vmdk vmkfstools -c 10240m -a lsilogic -d eagerzeroedthick mysharedisk_04.vmdk
创建成功后，打开VMware vSphere Client，进入数据存储浏览器打开如下图：
两个linux节点，虚拟机中配置如下，这里列出rac1的配置：（两台Linux配置相同）
两台虚拟机scsi控制器1设置为虚拟属性。
将同步客户机时间与主机时间打勾，重启两台虚拟机。
（四）配置yum源
安装linux ,选择最小化安装，虚拟机配置双网卡，挂载安装镜像，配置yum源为本地源，挂载操作系统光盘文件，命令如下参考:
mkdir /cdrom mount /dev/cdrom /cdrom cd /etc/yum.repos.d mkdir bak mv *.* bak
编辑yum 配置文件CentOS-Base.repo配置如下：
vi CentOS-Base.repo添加如下
[base] name=CentOS baseurl=file:///cdrom gpgcheck=0 enabled=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
联网环境也可使用oracle官方源，使用命令下载:
wget http://public-yum.oracle.com/public-yum-ol7.repo
安装net-tools，安装net-tools，只是为了使用ifconfig 命令，命令如下：
yum inatall net-tools* -y
安装unzip ,解压oracle安装包使用unzip命令：
yum install unzip -y
（五）安装前环境配置
1.安装前下载
下载oracle 安装包，grid 安装包，准备红帽操作系统或者oracle linux 操作系统，（oracle 和grid 需要的包在centos7会缺少compat-libstdc++ ，红帽镜像中有）
本文中使用的oracle 11.2.0.4 安装包及补丁包如下：
百度云链接：
链接：https://pan.baidu.com/s/1WvdpiTs9m3es5vyTBzHb4w
提取码：1111
操作系统下载链接：
centos
http://mirrors.sohu.com/centos/7.6.1810/isos/x86_64/
redhat
链接：https://pan.baidu.com/s/16-TgKVk_nAxLeakRsOPKkg
提取码：1111
创建asm，前配置及规划，环境配置如下：
oracle linux
下载链接
https://yum.oracle.com/oracle-linux-isos.html
2.网络环境
rac1
网卡1：网卡名：ens192配置如下：
public ip :192.168.2.101
网卡1：网卡名：ens224配置如下：
private vip :10.10.10.11
网卡配置文件如下：
ens160
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens160 UUID=fe2552fd-fd89-4cd0-9b97-55d52c1287f0 DEVICE=ens160 ONBOOT=yes IPADDR=192.168.2.101 NETMASK=255.255.255.0
ens192
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens192 UUID=a2834e07-69f5-49cf-b480-c58cefe28e6e DEVICE=ens192 ONBOOT=yes IPADDR=10.10.10.11 NETMASK=255.255.255.0
rac2
网卡1：网卡名：ens192配置如下：
public ip :192.168.2.102
网卡1：网卡名：ens224配置如下
private vip :10.10.10.12
配置文件如下：
ens160
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens160 UUID=fe2552fd-fd89-4cd0-9b97-55d52c1287f0 DEVICE=ens160 ONBOOT=yes IPADDR=192.168.2.102 NETMASK=255.255.255.0
ens192
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens192 UUID=a2834e07-69f5-49cf-b480-c58cefe28e6e DEVICE=ens192 ONBOOT=yes IPADDR=10.10.10.12 NETMASK=255.255.255.0
两个节点都要配置
修改hosts 文件添加ip,及安装Oracle后生成的虚拟ip
192.168.2.103-104，192.168.2.105 为安装后Oracle配置的ip:
192.168.2.101 rac1 192.168.2.102 rac2 192.168.2.103 rac1-vip 192.168.2.104 rac2-vip 10.10.10.11 rac1-priv 10.10.10.12 rac2-priv 192.168.2.105 rac-san
3 .关闭防火墙关闭selinux
两节点执行如下命令， 命令如下：
systemctl stop firewalld
systemctl disable firewalld
vi /etc/selinux/config
编辑
SELINUX=disabled
如下图操作：
3.安装grid必须的包
grid必须的包官方参考官方链接：
https://docs.oracle.com/cd/E11882_01/install.112/e41961/prelinux.htm#CWLIN225
Table 2-9 Linux x86-64 Oracle Grid Infrastructure and Oracle RAC Package Requirements Oracle Linux 7 and Red Hat Enterprise Linux 7 The following packages (or later versions) must be installed: binutils-2.23.52.0.1-12.el7.x86_64 compat-libcap1-1.10-3.el7.x86_64 gcc-4.8.2-3.el7.x86_64 gcc-c++-4.8.2-3.el7.x86_64 glibc-2.17-36.el7.i686 glibc-2.17-36.el7.x86_64 glibc-devel-2.17-36.el7.i686 glibc-devel-2.17-36.el7.x86_64 ksh libaio-0.3.109-9.el7.i686 libaio-0.3.109-9.el7.x86_64 libaio-devel-0.3.109-9.el7.i686 libaio-devel-0.3.109-9.el7.x86_64 libgcc-4.8.2-3.el7.i686 libgcc-4.8.2-3.el7.x86_64 libstdc++-4.8.2-3.el7.i686 libstdc++-4.8.2-3.el7.x86_64 libstdc++-devel-4.8.2-3.el7.i686 libstdc++-devel-4.8.2-3.el7.x86_64 libXi-1.7.2-1.el7.i686 libXi-1.7.2-1.el7.x86_64 libXtst-1.2.2-1.el7.i686 libXtst-1.2.2-1.el7.x86_64 make-3.82-19.el7.x86_64 sysstat-10.1.5-1.el7.x86_64
修改命令如下：
yum install -y binutils* yum install -y compat-libcap1* yum install -y gcc* yum install -y gcc-c++* yum install -y glibc* yum install -y glibc-devel* yum install -y ksh* yum install -y libaio* yum install -y libgcc* yum install -y libstdc* yum install -y libstdc++-devel* yum install -y libXi* yum install -y libXtst* yum install -y make* yum install -y sysstat* yum install -y elfutils-libelf-devel*
centos7中缺少的包compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm，需要在红帽镜像中找到。使用rpm命令安装即可。
compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm下载链接（从红帽镜像里面拷贝出来的）
https://files.cnblogs.com/files/wenxiao1-2-3-4/compat-libstdc.rpm.zip
[root@rac2 ~]# rpm -ivh compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm warning: compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY Preparing... ################################# [100%] Updating / installing... 1:compat-libstdc++-33-3.2.3-69.el6 ################################# [100%]
（环境检查时候会缺少pdksh包，忽略即可）。
4.用户和组
4.1创建安装目录，创建用户
/usr/sbin/groupadd -g 1000 oinstall /usr/sbin/groupadd -g 1020 asmadmin /usr/sbin/groupadd -g 1021 asmdba /usr/sbin/groupadd -g 1022 asmoper /usr/sbin/groupadd -g 1031 dba /usr/sbin/groupadd -g 1032 oper useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle mkdir -p /u01/app/11.2.0/grid mkdir -p /u01/app/grid mkdir /u01/app/oracle chown -R grid:oinstall /u01 chown oracle:oinstall /u01/app/oracle chmod -R 775 /u01/
参考官方创建组和用户，链接如下：
https://docs.oracle.com/cd/E11882_01/install.112/e41961/prelinux.htm#CWLIN178
最后使用root用户更改grid,oracle 用户的密码
passwd oracle passwd grid
4.2配置环境变量
4.2.1 grid环境变量
节点1上配置，编辑 .bash_profile
export TMP=/tmp export TMPDIR=$TMP export ORACLE_SID=+ASM1 export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/app/11.2.0/grid export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib umask 022
节点2上配置，编辑 .bash_profile
export TMP=/tmp export TMPDIR=$TMP export ORACLE_SID=+ASM2 export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/app/11.2.0/grid export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib umask 022
4.2.2oracle 环境变量
节点1.bash_profile
export TMP=/tmp export TMPDIR=$TMP export ORACLE_SID=rac1 export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1 export TNS_ADMIN=$ORACLE_HOME/network/admin export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib umask 022
节点2.bash_profile
export TMP=/tmp export TMPDIR=$TMP export ORACLE_SID=rac2 export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1 export TNS_ADMIN=$ORACLE_HOME/network/admin export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib umask 022
保存完毕，执行source命令，生效环境变量。
5.linux 配置裸设备
一个节点执行分区即可，执行fdisk 创建分区,编辑配置文件/usr/lib/udev/rules.d/60-raw.rules（两节点都需要修改）
[root@rac1 ~]# ls /usr/lib/udev/rules.d/60-raw.rules /usr/lib/udev/rules.d/60-raw.rules [root@rac1 ~]# cat /usr/lib/udev/rules.d/60-raw.rules # # Enter raw device bindings here. # # An example would be: # ACTION=="add", KERNEL=="sda", RUN+="/usr/bin/raw /dev/raw/raw1 %N" # to bind /dev/raw/raw1 to /dev/sda, or # ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="1", RUN+="/usr/bin/raw /dev/raw/raw2 %M %m" # to bind /dev/raw/raw2 to the device with major 8, minor 1. ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N" ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N" ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N" ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N" KERNEL=="raw[1]", MODE="0660", OWNER="grid", GROUP="asmadmin" KERNEL=="raw[2]", MODE="0660", OWNER="grid", GROUP="asmadmin" KERNEL=="raw[3]", MODE="0660", OWNER="grid", GROUP="asmadmin" KERNEL=="raw[4]", MODE="0660", OWNER="grid", GROUP="asmadmin"
重启系统后看到/dev/raw/下出现新配的文件
[root@rac1 ~]# ls -l /dev/raw/ total 0 crw-rw---- 1 grid asmadmin 162, 1 Jul 15 21:23 raw1 crw-rw---- 1 grid asmadmin 162, 2 Jul 14 20:45 raw2 crw-rw---- 1 grid asmadmin 162, 3 Jul 14 20:45 raw3 crw-rw---- 1 grid asmadmin 162, 4 Jul 14 20:45 raw4 crw-rw---- 1 root disk 162, 0 Jul 14 20:45 rawctl
6.系统参数
修改，两个文件/etc/sysctl.conf，和/etc/security/limits.conf（两节点都需要修改）
/etc/sysctl.conf添加如下
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
net.ipv4.tcp_wmem = 262144 262144 262144
net.ipv4.tcp_rmem = 4194304 4194304 4194304
修改完毕执行sysctl -p。
/etc/security/limits.conf 添加如下
grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536
7.配置互信
两种方法配置互信，一种手动执行命令保证两节点key文件一致，另一种方法使用oracle 安装包自带的互信脚本执行。
方法一
两台linux rac1,rac2互信，包括Oracle，grid 用户
oracle 用户和grid 用户大致相同
步骤如下
rac1下操作
ssh-keygen -t rsa
回车默认
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
rac2下操作
ssh-keygen -t rsa
回车默认
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
scp .ssh/authorized_keys rac2:
在rac2上操作
cat authorized_keys >> .ssh/authorized_keys
执行
scp .ssh/authorized_keys rac1:
rac1上操作
cat authorized_keys >.ssh/authorized_keys
rac1互信配置完毕配置文件，权限如下
[
grid@rac1 ~]$ ll .ssh/ -d drwx------ 2 grid oinstall 80 Jul 13 05:02 .ssh/ [grid@rac1 ~]$ ll .ssh/ total 16 -rw-r--r-- 1 grid oinstall 782 Jul 13 05:04 authorized_keys -rw------- 1 grid oinstall 1675 Jul 13 05:01 id_rsa -rw-r--r-- 1 grid oinstall 391 Jul 13 05:01 id_rsa.pub -rw-r--r-- 1 grid oinstall 360 Jul 13 05:02 known_hosts [grid@rac1 ~]$ cat .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVuiEr9Dug/tLJ+rZbdUr0ZDIlviaRPAoKcPoPf2EBCCRuYa3O6sqcOqKlwMNbitRl6eTaGxktVFOru7r2AG+DlajNaZo1B5jizx+atKCZzxoils8XGZNEPbEbNZN/NwgYEq2DYv1RkjcRXMvvPhEKskkBV3GT6BOPow0YTsRwajBmaLSamhg7fHnBTkzV6cjhDlLnLCyPevvlFMcKm1Y338ApSQMNBaPO9DhprYhUaEbUK8SpqZVOOWKHpAsFqc2iAPdnXJX36W/pc+dZf/FzuaJU1bPz6nzizEBB1zsnY3MWopKrac2j6TzjHpO4HLWCLDdJLYIMsFDCL0kJEIgl/ grid@rac1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOTMVA9hhXpMZQX9YK282J0QWxCawAwZCP/AH3zIuXYXA/q5GHYhnFoG2xAM7wt9Vf2+LXYKV0QrVBBpZAlS9MzCna2upqqfB32QtBBMeAIxyBE++9ufPUz+QgvfkpET5YA9vxgm1plX0jB5OftO0YXXQzE5m0+2TI+WRsDCSpqZJTZzfZr0ncLfT6olguTwok70ls4dv+VLDW5RyG4yef8T5+PEdQa9igPD8iqsrAcunmhzsf+Hroz7Mkkl7IiFrFZc3OOPEkOa1GK3doOZeuu75fEHDd89IEMfDPJdW5Hfv+xOTRBd3mOv0jn/oiL+9wxP9TqfamJ9XRZpUpDFeR grid@rac2 [grid@rac1 ~]$
rac2上rac1互信配置完毕配置文件，权限如下
[grid@rac2 ~]$ ll .ssh/ -d drwx------. 2 grid oinstall 80 Jul 12 23:23 .ssh/ [grid@rac2 ~]$ ll .ssh/ total 16 -rw-r--r--. 1 grid oinstall 782 Jul 13 05:04 authorized_keys -rw-------. 1 grid oinstall 1679 Jul 12 23:22 id_rsa -rw-r--r--. 1 grid oinstall 391 Jul 12 23:22 id_rsa.pub -rw-r--r--. 1 grid oinstall 360 Jul 13 05:02 known_hosts [grid@rac2 ~]$ cat .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVuiEr9Dug/tLJ+rZbdUr0ZDIlviaRPAoKcPoPf2EBCCRuYa3O6sqcOqKlwMNbitRl6eTaGxktVFOru7r2AG+DlajNaZo1B5jizx+atKCZzxoils8XGZNEPbEbNZN/NwgYEq2DYv1RkjcRXMvvPhEKskkBV3GT6BOPow0YTsRwajBmaLSamhg7fHnBTkzV6cjhDlLnLCyPevvlFMcKm1Y338ApSQMNBaPO9DhprYhUaEbUK8SpqZVOOWKHpAsFqc2iAPdnXJX36W/pc+dZf/FzuaJU1bPz6nzizEBB1zsnY3MWopKrac2j6TzjHpO4HLWCLDdJLYIMsFDCL0kJEIgl/ grid@rac1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOTMVA9hhXpMZQX9YK282J0QWxCawAwZCP/AH3zIuXYXA/q5GHYhnFoG2xAM7wt9Vf2+LXYKV0QrVBBpZAlS9MzCna2upqqfB32QtBBMeAIxyBE++9ufPUz+QgvfkpET5YA9vxgm1plX0jB5OftO0YXXQzE5m0+2TI+WRsDCSpqZJTZzfZr0ncLfT6olguTwok70ls4dv+VLDW5RyG4yef8T5+PEdQa9igPD8iqsrAcunmhzsf+Hroz7Mkkl7IiFrFZc3OOPEkOa1GK3doOZeuu75fEHDd89IEMfDPJdW5Hfv+xOTRBd3mOv0jn/oiL+9wxP9TqfamJ9XRZpUpDFeR grid@rac2 [grid@rac2 ~]$
方法供参考，目的是两台机器的密钥文件写入到authorized_keys，拷贝到两台机器上，并且一致
互信测试执行
rac1
[grid@rac1 ~]$ ssh rac1 Last login: Wed Jul 15 03:30:57 2020 [grid@rac1 ~]$ [grid@rac1 ~]$ ssh rac2 Last login: Wed Jul 15 03:04:23 2020 from rac1 [grid@rac2 ~]$
rac2
grid@rac2 ~]$ ssh rac2 Last login: Wed Jul 15 03:44:49 2020 from rac1 [grid@rac2 ~]$ ssh rac1 Last login: Wed Jul 15 03:44:31 2020 from rac1 [grid@rac1 ~]$
如上输出即可正常。以上完毕。
方法二
使用oral测自带的互信脚本，执行即可
[root@rac1 database]# ls install readme.html response rpm runInstaller sshsetup stage welcome.html [root@rac1 database]# ./sshsetup/sshUserSetup.sh help=y Please specify a valid and existing cluster configuration file. Either user name or host information is missing Usage ./sshsetup/sshUserSetup.sh -user <user name> [ -hosts "<space separated hostlist>" | -hostfile <absolute path of cluster configuration file> ] [ -advanced ] [ -verify] [ -exverify ] [ -logfile <desired absolute path of logfile> ] [-confirm] [-shared] [-help] [-usePassphrase] [-noPromptPassphrase] [root@rac1 database]#
为为两主机rac1 ,rac2添加互信，用户是oralce
./sshUserSetup.sh -hosts "rac1 rac2" -user oracle -advanced
脚本输出如下
[root@rac1 sshsetup]# ./sshUserSetup.sh -hosts "rac1 rac2" -user oracle -advanced
The output of this script is also logged into /tmp/sshUserSetup_2021-09-16-06-04-32.log
Hosts are rac1 rac2
user is oracle
Platform:- Linux
Checking if the remote hosts are reachable
PING rac1 (192.168.10.20) 56(84) bytes of data.
64 bytes from rac1 (192.168.10.20): icmp_seq=1 ttl=64 time=0.018 ms
64 bytes from rac1 (192.168.10.20): icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from rac1 (192.168.10.20): icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from rac1 (192.168.10.20): icmp_seq=4 ttl=64 time=0.044 ms
64 bytes from rac1 (192.168.10.20): icmp_seq=5 ttl=64 time=0.048 ms
--- rac1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.018/0.039/0.048/0.012 ms
PING rac2 (192.168.10.21) 56(84) bytes of data.
64 bytes from rac2 (192.168.10.21): icmp_seq=1 ttl=64 time=0.600 ms
64 bytes from rac2 (192.168.10.21): icmp_seq=2 ttl=64 time=0.714 ms
64 bytes from rac2 (192.168.10.21): icmp_seq=3 ttl=64 time=0.701 ms
64 bytes from rac2 (192.168.10.21): icmp_seq=4 ttl=64 time=0.737 ms
64 bytes from rac2 (192.168.10.21): icmp_seq=5 ttl=64 time=0.643 ms
--- rac2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 0.600/0.679/0.737/0.050 ms
Remote host reachability check succeeded.
The following hosts are reachable: rac1 rac2.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost rac1
numhosts 2
The script will setup SSH connectivity from the host rac1 to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host rac1
and the remote hosts without being prompted for passwords or confirmations.
NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.
NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
directories.
Do you want to continue and let the script make the above mentioned changes (yes/no)?
yes
The user chose yes
Please specify if you want to specify a passphrase for the private key this script will create for the local host. Passphrase is used to encrypt the private key and makes SSH much more secure. Type 'yes' or 'no' and then press enter. In case you press 'yes', you would need to enter the passphrase whenever the script executes ssh or scp.
The estimated number of times the user would be prompted for a passphrase is 4. In addition, if the private-public files are also newly created, the user would have to specify the passphrase on one additional occasion.
Enter 'yes' or 'no'.
yes
The user chose yes
Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /root/.ssh/config, it would be backed up to /root/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:gKwItLJdlR8rhK15zllEw2+Em1TyPYfylmHZDO4i88M root@rac1
The key's randomart image is:
+---[RSA 1024]----+
| . o.++o. . |
|. ...o+ *+.o * |
|o. ++.+ Bo O + |
|o+ oo o.* o= = |
|o o + +S.. = |
| + = o |
| E |
| . |
| |
+----[SHA256]-----+
Creating .ssh directory and setting permissions on remote host rac1
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host rac1. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host rac1.
Warning: Permanently added 'rac1,192.168.10.20' (ECDSA) to the list of known hosts.
oracle@rac1's password:
Done with creating .ssh directory and setting permissions on remote host rac1.
Creating .ssh directory and setting permissions on remote host rac2
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host rac2. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host rac2.
Warning: Permanently added 'rac2,192.168.10.21' (ECDSA) to the list of known hosts.
oracle@rac2's password:
Done with creating .ssh directory and setting permissions on remote host rac2.
Copying local host public key to the remote host rac1
The user may be prompted for a password or passphrase here since the script would be using SCP for host rac1.
oracle@rac1's password:
Done copying local host public key to the remote host rac1
Copying local host public key to the remote host rac2
The user may be prompted for a password or passphrase here since the script would be using SCP for host rac2.
oracle@rac2's password:
Done copying local host public key to the remote host rac2
Creating keys on remote host rac1 if they do not exist already. This is required to setup SSH on host rac1.
Generating public/private rsa key pair.
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
SHA256:v5ssNTbEi0QV7DrGvPvRRlHtCFgDx1jdVwQvAP6bWBQ oracle@rac1
The key's randomart image is:
+---[RSA 1024]----+
| o=XE.o=+|
| ..+.o=..+|
| . o. o..oo|
| . +o ....|
| +S+ .+ |
| B.*= o |
| . =+o= |
| o. = |
| .+*. |
+----[SHA256]-----+
Creating keys on remote host rac2 if they do not exist already. This is required to setup SSH on host rac2.
Generating public/private rsa key pair.
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
SHA256:DOM3p67iPw0SutOj8yXX6imidsQxrCEcPqJqS00nwj4 oracle@rac2
The key's randomart image is:
+---[RSA 1024]----+
| |
| . |
|o o o |
|+= +.. + |
|+o=+oo. S . |
|o.=o+ .o + |
|.E.+..ooo |
|oo*.=+.+. |
|+o+Bo**o. |
+----[SHA256]-----+
Updating authorized_keys file on remote host rac1
Updating known_hosts file on remote host rac1
The script will run SSH on the remote machine rac1. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.
Updating authorized_keys file on remote host rac2
Updating known_hosts file on remote host rac2
The script will run SSH on the remote machine rac2. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.
SSH setup is complete.
------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user oracle.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the <OMS HOME>/sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--rac1:--
Running /usr/bin/ssh -x -l oracle rac1 date to verify SSH connectivity has been setup from local host to rac1.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
The script will run SSH on the remote machine rac1. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.
Thu Sep 16 06:05:04 EDT 2021
------------------------------------------------------------------------
--rac2:--
Running /usr/bin/ssh -x -l oracle rac2 date to verify SSH connectivity has been setup from local host to rac2.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
The script will run SSH on the remote machine rac2. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.
Thu Sep 16 06:05:20 EDT 2021
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from rac1 to rac1
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
bash: -c: line 0: unexpected EOF while looking for matching `"'
bash: -c: line 1: syntax error: unexpected end of file
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from rac1 to rac2
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
bash: -c: line 0: unexpected EOF while looking for matching `"'
bash: -c: line 1: syntax error: unexpected end of file
------------------------------------------------------------------------
-Verification from complete-
SSH verification complete.
执行完毕，执行验证
[oracle@rac1 ~]$ date;ssh rac1 date;ssh rac1 date;ssh rac2 date;ssh rac2 date;ssh rac1 date
Thu Sep 16 06:17:57 EDT 2021
Thu Sep 16 06:17:58 EDT 2021
Thu Sep 16 06:17:58 EDT 2021
Thu Sep 16 06:18:13 EDT 2021
Thu Sep 16 06:18:13 EDT 2021
Thu Sep 16 06:17:59 EDT 2021
[oracle@rac1 ~]$
（六）grid安装前检查
配置完毕执行grid环境检查脚本
./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -fixup -verbose
检查完毕，缺少的包pdksh 没有安装，安装了ksh，可以忽略，oracle 官方有解释到，参考链接：
https://docs.oracle.com/cd/E11882_01/relnotes.112/e23558/toc.htm#CJAFEIBJ
解释如下：
7.2.6 Missing Package Error During Oracle Database Installation on Oracle Linux 7
During Oracle Database 11.2.0.4 installation on Oracle Linux 7, the Oracle Universal Installer reports a missing pdksh-5.2.14 package.
Workaround:
Ignore the missing pdksh-5.2.14 package error and proceed with the installation.
This issue is tracked with Oracle bug 19947777.
（七）安装 grid 软件，oracle 及创建数据库
安装grid前，安装grid中cvuqdisk包，操作如下：
[root@rac2 ~]# ls -lh /u01/grid/ total 56K drwxr-xr-x. 4 grid oinstall 4.0K Aug 26 2013 install -rw-r--r--. 1 grid oinstall 30K Aug 27 2013 readme.html drwxr-xr-x. 2 grid oinstall 58 Jul 23 02:48 response drwxr-xr-x. 2 grid oinstall 34 Aug 26 2013 rpm -rwxr-xr-x. 1 grid oinstall 4.8K Aug 26 2013 runcluvfy.sh -rwxr-xr-x. 1 grid oinstall 3.2K Aug 26 2013 runInstaller drwxr-xr-x. 2 grid oinstall 29 Aug 26 2013 sshsetup drwxr-xr-x. 14 grid oinstall 4.0K Aug 26 2013 stage -rw-r--r--. 1 grid oinstall 500 Aug 27 2013 welcome.html [root@rac2 ~]# ls -lh /u01/grid/rpm/ total 12K -rw-r--r--. 1 grid oinstall 8.1K Aug 26 2013 cvuqdisk-1.0.9-1.rpm [root@rac2 ~]# rpm -ivh /u01/grid/rpm/cvuqdisk-1.0.9-1.rpm Preparing... ################################# [100%] ls: cannot access /usr/sbin/smartctl: No such file or directory /usr/sbin/smartctl not found. error: %pre(cvuqdisk-1.0.9-1.x86_64) scriptlet failed, exit status 1 error: cvuqdisk-1.0.9-1.x86_64: install failed [root@rac2 ~]# yum install -y smartmontools Loaded plugins: ulninfo Resolving Dependencies --> Running transaction check ---> Package smartmontools.x86_64 1:6.5-1.el7 will be installed --> Processing Dependency: mailx for package: 1:smartmontools-6.5-1.el7.x86_64 --> Running transaction check ---> Package mailx.x86_64 0:12.5-19.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =============================================================================================================================================================================================== Package Arch Version Repository Size =============================================================================================================================================================================================== Installing: smartmontools x86_64 1:6.5-1.el7 base 460 k Installing for dependencies: mailx x86_64 12.5-19.el7 base 244 k Transaction Summary =============================================================================================================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 704 k Installed size: 2.2 M Downloading packages: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 3.6 MB/s | 704 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : mailx-12.5-19.el7.x86_64 1/2 Installing : 1:smartmontools-6.5-1.el7.x86_64 2/2 Verifying : mailx-12.5-19.el7.x86_64 1/2 Verifying : 1:smartmontools-6.5-1.el7.x86_64 2/2 Installed: smartmontools.x86_64 1:6.5-1.el7 Dependency Installed: mailx.x86_64 0:12.5-19.el7 Complete! [root@rac2 ~]# rpm -ivh /u01/grid/rpm/cvuqdisk-1.0.9-1.rpm Preparing... ################################# [100%] Using default group oinstall to install package Updating / installing... 1:cvuqdisk-1.0.9-1 ################################# [100%] [root@rac2 ~]#
解压grid安装包，静默安装grid配置文件：（这里配置安装grid软件配置ASM）编辑grid_install.rsp,如下仅供参考：
解释这里：oracle.install.crs.config.networkInterfaceList=ens192:192.168.2.0:1,ens224:10.10.10.0:2
填写公有网卡名称和似有网卡名和ip 段就可以了
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v11_2_0 ORACLE_HOSTNAME=rac1 INVENTORY_LOCATION=/u01/app/oraInventory SELECTED_LANGUAGES=en oracle.install.option=CRS_CONFIG ORACLE_BASE=/u01/app/grid ORACLE_HOME=/u01/app/11.2.0/grid oracle.install.asm.OSDBA=asmdba oracle.install.asm.OSOPER=asmoper oracle.install.asm.OSASM=asmadmin oracle.install.crs.config.gpnp.scanName=rac-san oracle.install.crs.config.gpnp.scanPort=1521 oracle.install.crs.config.clusterName=cluster-san oracle.install.crs.config.gpnp.configureGNS=false oracle.install.crs.config.gpnp.gnsSubDomain= oracle.install.crs.config.gpnp.gnsVIPAddress= oracle.install.crs.config.autoConfigureClusterNodeVIP=false oracle.install.crs.config.clusterNodes=rac1:rac1-vip,rac2:rac2-vip oracle.install.crs.config.networkInterfaceList=ens192:192.168.2.0:1,ens224:10.10.10.0:2 oracle.install.crs.config.storageOption=ASM_STORAGE oracle.install.crs.config.sharedFileSystemStorage.diskDriveMapping= oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations= oracle.install.crs.config.sharedFileSystemStorage.votingDiskRedundancy=NORMAL oracle.install.crs.config.sharedFileSystemStorage.ocrLocations= oracle.install.crs.config.sharedFileSystemStorage.ocrRedundancy=NORMAL oracle.install.crs.config.useIPMI=false oracle.install.crs.config.ipmi.bmcUsername= oracle.install.crs.config.ipmi.bmcPassword= oracle.install.asm.SYSASMPassword=oracle oracle.install.asm.diskGroup.name=DATA oracle.install.asm.diskGroup.redundancy=NORMAL oracle.install.asm.diskGroup.AUSize=1 oracle.install.asm.diskGroup.disks=/dev/raw/raw1,/dev/raw/raw2,/dev/raw/raw3,/dev/raw/raw4 oracle.install.asm.diskGroup.diskDiscoveryString=/dev/raw/* oracle.install.asm.monitorPassword=oracle oracle.install.crs.upgrade.clusterNodes= oracle.install.asm.upgradeASM=false oracle.installer.autoupdates.option=SKIP_UPDATES oracle.installer.autoupdates.downloadUpdatesLoc= AUTOUPDATES_MYORACLESUPPORT_USERNAME= AUTOUPDATES_MYORACLESUPPORT_PASSWORD= PROXY_HOST= PROXY_PORT= PROXY_USER= PROXY_PWD= PROXY_REALM=
安装命令：
./runInstaller -silent -ignorePrereq -showProgress -responseFile /u01/ora_app/grid/response/grid_install01.rsp
-ignorePrereq表示忽略警告。
操作步骤如下：
[grid@rac1 grid]$ ./runInstaller -silent -ignorePrereq -showProgress -responseFile /u01/ora_app/grid/response/grid_install01.rsp Starting Oracle Universal Installer... Checking Temp space: must be greater than 120 MB. Actual 9901 MB Passed Checking swap space: must be greater than 150 MB. Actual 4091 MB Passed Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-07-24_06-21-45PM. Please wait ... [grid@rac1 grid]$ [WARNING] [INS-30011] The SYS password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. [WARNING] [INS-30011] The ASMSNMP password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. You can find the log of this install session at: /u01/app/oraInventory/logs/installActions2020-07-24_06-21-45PM.log Prepare in progress. .................................................. 9% Done. Prepare successful. Copy files in progress. .................................................. 15% Done. .................................................. 20% Done. .................................................. 25% Done. .................................................. 30% Done. .................................................. 35% Done. .................................................. 40% Done. .................................................. 45% Done. ........................................ Copy files successful. Link binaries in progress. Link binaries successful. .................................................. 62% Done. Setup files in progress. Setup files successful. .................................................. 76% Done. Perform remote operations in progress. .................................................. 89% Done. Perform remote operations successful. The installation of Oracle Grid Infrastructure 11g was successful. Please check '/u01/app/oraInventory/logs/silentInstall2020-07-24_06-21-45PM.log' for more details. .................................................. 94% Done. Execute Root Scripts in progress. As a root user, execute the following script(s): 1. /u01/app/oraInventory/orainstRoot.sh 2. /u01/app/11.2.0/grid/root.sh Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes: [rac1, rac2] Execute /u01/app/11.2.0/grid/root.sh on the following nodes: [rac1, rac2] .................................................. 100% Done. Execute Root Scripts successful. As install user, execute the following script to complete the configuration. 1. /u01/app/11.2.0/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=<response_file> Note: 1. This script must be run on the same host from where installer was run. 2. This script needs a small password properties file for configuration assistants that require passwords (refer to install guide documentation). Successfully Setup Software.
两个节点上执行脚本，执行脚本前更新oracle补丁，参见文章末尾步骤，安装补丁18370031。
补丁更新完毕，接着执行root脚本步骤如下：
[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@rac1 ~]# /u01/app/11.2.0/grid/root.sh Check /u01/app/11.2.0/grid/install/root_rac1_2020-07-24_18-40-41.log for the output of root script
新开另外一个终端：
[root@rac1 ~]# tail -f /u01/app/11.2.0/grid/install/root_rac1_2020-08-24_06-11-09.log | nl 1 Copying oraenv to /usr/local/bin ... 2 Copying coraenv to /usr/local/bin ... 3 Entries will be added to the /etc/oratab file as needed by 4 Database Configuration Assistant when a database is created 5 Finished running generic part of root script. 6 Now product-specific root actions will be performed. 7 Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params 8 User ignored Prerequisites during installation 9 Installing Trace File Analyzer 10 OLR initialization - successful 11 root wallet 12 root wallet cert 13 root cert export 14 peer wallet 15 profile reader wallet 16 pa wallet 17 peer wallet keys 18 pa wallet keys 19 peer cert request 20 pa cert request 21 peer cert 22 pa cert 23 peer root cert TP 24 profile reader root cert TP 25 pa root cert TP 26 peer pa cert TP 27 pa peer cert TP 28 profile reader pa cert TP 29 profile reader peer cert TP 30 peer user cert 31 pa user cert 32 Adding Clusterware entries to oracle-ohasd.service 33 CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1' 34 CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded 35 CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1' 36 CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded 37 CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1' 38 CRS-2672: Attempting to start 'ora.gipcd' on 'rac1' 39 CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded 40 CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded 41 CRS-2672: Attempting to start 'ora.cssd' on 'rac1' 42 CRS-2672: Attempting to start 'ora.diskmon' on 'rac1' 43 CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded 44 CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded 45 ASM created and started successfully. 46 Disk Group DATA created successfully. 47 clscfg: -install mode specified 48 Successfully accumulated necessary OCR keys. 49 Creating OCR keys for user 'root', privgrp 'root'.. 50 Operation successful. 51 CRS-4256: Updating the profile 52 Successful addition of voting disk 5daea1cfaac24f1ebf870b7f9bb964c3. 53 Successful addition of voting disk 28267865584c4fc3bf672e650d0e28ba. 54 Successful addition of voting disk 1e8c192b947d4f43bf96dba5e650f11f. 55 Successfully replaced voting disk group with +DATA. 56 CRS-4256: Updating the profile 57 CRS-4266: Voting file(s) successfully replaced 58 ## STATE File Universal Id File Name Disk group 59 -- ----- ----------------- --------- --------- 60 1. ONLINE 5daea1cfaac24f1ebf870b7f9bb964c3 (/dev/raw/raw1) [DATA] 61 2. ONLINE 28267865584c4fc3bf672e650d0e28ba (/dev/raw/raw2) [DATA] 62 3. ONLINE 1e8c192b947d4f43bf96dba5e650f11f (/dev/raw/raw3) [DATA] 63 Located 3 voting disk(s). 64 CRS-2672: Attempting to start 'ora.asm' on 'rac1' 65 CRS-2676: Start of 'ora.asm' on 'rac1' succeeded 66 CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1' 67 CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded 68 Configure Oracle Grid Infrastructure for a Cluster ... succeeded
节点二
[root@rac2 ~]# tail -f /u01/app/11.2.0/grid/install/root_rac2_2020-07-24_18-48-51.log | nl 1 Creating /etc/oratab file... 2 Entries will be added to the /etc/oratab file as needed by 3 Database Configuration Assistant when a database is created 4 Finished running generic part of root script. 5 Now product-specific root actions will be performed. 6 Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params 7 Creating trace directory 8 User ignored Prerequisites during installation 9 Installing Trace File Analyzer 10 OLR initialization - successful 11 Adding Clusterware entries to oracle-ohasd.service 12 CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating 13 An active cluster was found during exclusive startup, restarting to join the cluster 14 Configure Oracle Grid Infrastructure for a Cluster ... succeeded
执行完毕成功。
7.1.安装oracle 软件
oracle 必须的软件包参考
https://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCGJCEA
db_install.rsp文件配置如下：
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v11_2_0 oracle.install.option=INSTALL_DB_AND_CONFIG ORACLE_HOSTNAME=rac1 UNIX_GROUP_NAME=oinstall INVENTORY_LOCATION=/u01/app/oraInventory SELECTED_LANGUAGES=en ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 ORACLE_BASE=/u01/app/oracle oracle.install.db.InstallEdition=EE oracle.install.db.EEOptionsSelection=false oracle.install.db.optionalComponents=oracle.rdbms.partitioning:11.2.0.4.0,oracle.oraolap:11.2.0.4.0,oracle.rdbms.dm:11.2.0.4.0,oracle.rdbms.dv:11.2.0.4.0,oracle.rdbms.lbac:11.2.0.4.0,oracle.rdbms.rat:11.2.0.4.0 oracle.install.db.DBA_GROUP=dba oracle.install.db.OPER_GROUP=oper oracle.install.db.CLUSTER_NODES=rac1,rac2 oracle.install.db.isRACOneInstall= oracle.install.db.racOneServiceName= oracle.install.db.config.starterdb.type=GENERAL_PURPOSE oracle.install.db.config.starterdb.globalDBName=rac oracle.install.db.config.starterdb.SID=rac oracle.install.db.config.starterdb.characterSet=ZHS16GBK oracle.install.db.config.starterdb.memoryOption=true oracle.install.db.config.starterdb.memoryLimit=512 oracle.install.db.config.starterdb.installExampleSchemas=false oracle.install.db.config.starterdb.enableSecuritySettings=true oracle.install.db.config.starterdb.password.ALL=oracle oracle.install.db.config.starterdb.password.SYS= oracle.install.db.config.starterdb.password.SYSTEM= oracle.install.db.config.starterdb.password.SYSMAN= oracle.install.db.config.starterdb.password.DBSNMP= oracle.install.db.config.starterdb.control=DB_CONTROL oracle.install.db.config.starterdb.gridcontrol.gridControlServiceURL= oracle.install.db.config.starterdb.automatedBackup.enable=false oracle.install.db.config.starterdb.automatedBackup.osuid= oracle.install.db.config.starterdb.automatedBackup.ospwd= oracle.install.db.config.starterdb.storageType=ASM_STORAGE oracle.install.db.config.starterdb.fileSystemStorage.dataLocation= oracle.install.db.config.starterdb.fileSystemStorage.recoveryLocation= oracle.install.db.config.asm.diskGroup=DATA oracle.install.db.config.asm.ASMSNMPPassword=oracle MYORACLESUPPORT_USERNAME= MYORACLESUPPORT_PASSWORD= SECURITY_UPDATES_VIA_MYORACLESUPPORT=false DECLINE_SECURITY_UPDATES=true PROXY_HOST= PROXY_PORT= PROXY_USER= PROXY_PWD= PROXY_REALM= COLLECTOR_SUPPORTHUB_URL= oracle.installer.autoupdates.option=SKIP_UPDATES oracle.installer.autoupdates.downloadUpdatesLoc= AUTOUPDATES_MYORACLESUPPORT_USERNAME= AUTOUPDATES_MYORACLESUPPORT_PASSWORD=
执行安装
./runInstaller -ignoreSysPrereqs -silent -showProgress -responseFile /u01/ora_app/database/response/db_install.rsp
这里会出现一个错误：
[oracle@rac1 database]$ ./runInstaller -ignoreSysPrereqs -silent -showProgress -responseFile /u01/ora_app/database/response/db_install.rsp Starting Oracle Universal Installer... Checking Temp space: must be greater than 120 MB. Actual 8932 MB Passed Checking swap space: must be greater than 150 MB. Actual 4049 MB Passed Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-07-24_07-00-02PM. Please wait ...[oracle@rac1 database]$ [FATAL]
[INS-35354] The system on which you are attempting to install Oracle RAC
is not part of a valid cluster. CAUSE: Before you can install Oracle RAC, you must install Oracle Grid Infrastructure on all servers (Oracle Clusterware and Oracle ASM)
to create a cluster. ACTION: Oracle Grid Infrastructure is not installed. Install it either from the separate installation media included in your media pack,
or install
it by downloading it from Electronic Product Delivery (EPD)
or the Oracle Technology Network (OTN).
Oracle Grid Infrastructure normally is installed by a different operating system user than the one used for Oracle Database.
It may need to be installed by your system administrator.
See the installation guide for more details.
修正错误见微章末尾：修改grid配置文件inventory.xml。
修正后重新执行安装如下：
[oracle@rac1 database]$./runInstaller -ignoreSysPrereqs -silent -showProgress -responseFile /u01/ora_app/database/response/db_install.rsp
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB.
Actual 13597 MB Passed Checking swap space:
must be greater than 150 MB. Actual 4095 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-08-24_09-54-03PM. Please wait ...oracle@rac1 database]$
[FATAL] [INS-30502] No ASM disk group found.
CAUSE: There were no disk groups managed by the ASM instance +ASM1.
ACTION: Use Automatic Storage Management Configuration Assistant to add disk groups.
这里提示找不到磁盘组，我这里磁盘组是正常的，
[grid@rac1 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DATA.dg ora....up.type ONLINE ONLINE rac1 ora....N1.lsnr ora....er.type ONLINE ONLINE rac1 ora.asm ora.asm.type ONLINE ONLINE rac1 ora.cvu ora.cvu.type ONLINE ONLINE rac1 ora.gsd ora.gsd.type OFFLINE OFFLINE ora....network ora....rk.type ONLINE ONLINE rac1 ora.oc4j ora.oc4j.type ONLINE ONLINE rac2 ora.ons ora.ons.type ONLINE ONLINE rac1 ora....SM1.asm application ONLINE ONLINE rac1 ora.rac1.gsd application OFFLINE OFFLINE ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip ora....t1.type ONLINE ONLINE rac1 ora....SM2.asm application ONLINE ONLINE rac2 ora.rac2.gsd application OFFLINE OFFLINE ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip ora....t1.type ONLINE ONLINE rac2 ora.scan1.vip ora....ip.type ONLINE ONLINE rac1
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 24 22:25:56 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> column path format 20
SP2-0246: Illegal FORMAT string "20"
SQL> column path format a20
SQL> set lines 200
SQL> select group_number ,disk_number,mode_status ,name ,path from v$asm_disk;
GROUP_NUMBER DISK_NUMBER MODE_ST NAME PATH
------------ ----------- ------- ------------------------------ --------------------
1 3 ONLINE DATA_0003 /dev/raw/raw4
1 2 ONLINE DATA_0002 /dev/raw/raw3
1 1 ONLINE DATA_0001 /dev/raw/raw2
1 0 ONLINE DATA_0000 /dev/raw/raw1
SQL>
修正步骤，
[root@rac1 ~]# ls /u01/app/11.2.0/grid/bin/oracle -l -rwxr-x--x 1 grid oinstall 209836184 Aug 24 06:01 /u01/app/11.2.0/grid/bin/oracle [root@rac1 ~]# chmod +s /u01/app/11.2.0/grid/bin/oracle [root@rac1 ~]# ls /u01/app/11.2.0/grid/bin/oracle -l -rwsr-s--x 1 grid oinstall 209836184 Aug 24 06:01 /u01/app/11.2.0/grid/bin/oracle [root@rac1 ~]#
更改后重新运行安装，
[oracle@rac1 database]$ ./runInstaller -ignoreSysPrereqs -silent -showProgress -responseFile /u01/ora_app/database/response/db_install.rsp
Starting Oracle Universal Installer... Checking Temp space: must be greater than 120 MB. Actual 13597 MB Passed Checking swap space: must be greater than 150 MB. Actual 4084 MB Passed Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-08-24_11-03-43PM. Please wait ...[oracle@rac1 database]$ [WARNING] [INS-30011] The ADMIN password entered does not conform to the Oracle recommended standards. CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9]. ACTION: Provide a password that conforms to the Oracle recommended standards. [WARNING] [INS-13014] Target environment do not meet some optional requirements. CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/installActions2020-08-24_11-03-43PM.log ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/installActions2020-08-24_11-03-43PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually. You can find the log of this install session at: /u01/app/oraInventory/logs/installActions2020-08-24_11-03-43PM.log Prepare in progress. .................................................. 8% Done. Prepare successful. Copy files in progress. .................................................. 13% Done. .................................................. 18% Done. .................................................. 23% Done. .................................................. 28% Done. .................................................. 33% Done. .................................................. 38% Done. .................................................. 43% Done. .................... Copy files successful. Link binaries in progress. .......... Link binaries successful. .................................................. 53% Done. Setup files in progress. Setup files successful. .................................................. 65% Done. Perform remote operations in progress. .................................................. 76% Done. Perform remote operations successful. SEVERE:Remote 'AttachHome' failed on nodes: 'rac2'. Refer to '/u01/app/oraInventory/logs/installActions2020-08-24_11-03-43PM.log' for details. It is recommended that the following command needs to be manually run on the failed nodes: /u01/app/oracle/product/11.2.0/db_1/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 ORACLE_HOME_NAME=OraDb11g_home1 CLUSTER_NODES=rt1,rt2 "INVENTORY_LOCATION=/u01/app/oraInventory" LOCAL_NODE=<node on which command is to be run>. Please refer 'AttachHome' logs under central inventory of remote nodes where failure occurred for more details. The installation of Oracle Database 11g was successful on the local node but failed on remote nodes. Please check '/u01/app/oraInventory/logs/silentInstall2020-08-24_11-03-43PM.log' for more details. Oracle Database Configuration Assistant in progress. .................................................. 95% Done. Oracle Database Configuration Assistant failed. [WARNING] [INS-32091] Software installation was successful. But some configuration assistants failed, were cancelled or skipped. ACTION: Refer to the logs or contact Oracle Support Services.
这里静默安装oracle软件安装成功了，监听和创建数据库需要手动执行下。
查看日志如下：
[oracle@rac1 ~]$ cat /u01/app/oraInventory/logs/silentInstall2020-08-24_11-03-43PM.log silentInstall2020-08-24_11-03-43PM.log sNativeVolName:/u01/app/oracle/product/11.2.0/db_1/ m_asNodeArray:rac1,rac2 m_sLocalNode:rac1 sNativeVolName:/tmp/ m_asNodeArray:rac1,rac2 m_sLocalNode:rac1 sNativeVolName:/u01/app/oraInventory/ m_asNodeArray:rac1,rac2 m_sLocalNode:rac1 Error in invoking target 'agent nmhs' of makefile '/u01/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk'. See '/u01/app/oraInventory/logs/installActions2020-08-24_11-03-43PM.log' for details. sNativeVolName:/u01/app/oracle/ m_asNodeArray:rac1,rac2 m_sLocalNode:rac1 SEVERE:Remote 'AttachHome' failed on nodes: 'rt2'. Refer to '/u01/app/oraInventory/logs/installActions2020-08-24_11-03-43PM.log' for details. It is recommended that the following command needs to be manually run on the failed nodes: /u01/app/oracle/product/11.2.0/db_1/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 ORACLE_HOME_NAME=OraDb11g_home1 CLUSTER_NODES=rt1,rt2 "INVENTORY_LOCATION=/u01/app/oraInventory" LOCAL_NODE=<node on which command is to be run>. Please refer 'AttachHome' logs under central inventory of remote nodes where failure occurred for more details. The installation of Oracle Database 11g was successful on the local node but failed on remote nodes. [oracle@rac1 ~]$
打补丁19692824，执行如下：
[oracle@rac1 ~]$ cd /u01/oracle_app/19692824/ [oracle@rac1 19692824]$ find /u01/app/oracle/ -iname "opatch" /u01/app/oracle/product/11.2.0/db_1/inventory/Templates/OPatch /u01/app/oracle/product/11.2.0/db_1/OPatch /u01/app/oracle/product/11.2.0/db_1/OPatch/opatchprereqs/opatch /u01/app/oracle/product/11.2.0/db_1/OPatch/opatch /u01/app/oracle/product/11.2.0/db_1/oc4j/cfgtoollogs/opatch /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch [oracle@rac1 19692824]$ /u01/app/oracle/product/11.2.0/db_1/OPatch/opatch apply
7.2创建数据库
首先配置监听程序
监听配置文件
[GENERAL] RESPONSEFILE_VERSION="11.2" CREATE_TYPE="CUSTOM" [oracle.net.ca] INSTALLED_COMPONENTS={"server","net8","javavm"} INSTALL_TYPE=""typical"" LISTENER_NUMBER=1 LISTENER_NAMES={"LISTENER"} LISTENER_PROTOCOLS={"TCP;1521"} LISTENER_START=""LISTENER"" NAMING_METHODS={"TNSNAMES","ONAMES","HOSTNAME"} NSN_NUMBER=1 NSN_NAMES={"EXTPROC_CONNECTION_DATA"} NSN_SERVICE={"PLSExtProc"} NSN_PROTOCOLS={"TCP;HOSTNAME;1521"}
创建监听需要使用grid用户执行，执行如下
[grid@rac1 ~]$ netca -silent -responsefile /u01/netca.rsp
Parsing command line arguments:
Parameter "silent" = true
Parameter "responsefile" = /u01/netca.rsp
Done parsing command line arguments.
Oracle Net Services Configuration:
Profile configuration complete.
Profile configuration complete.
rac1...
rac2...
Oracle Net Listener Startup:
Listener started successfully.
Listener configuration complete.
Oracle Net Services configuration successful. The exit code is 0
[grid@rac1 ~]$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 25-AUG-2020 02:13:49
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 25-AUG-2020 01:21:42
Uptime 0 days 0 hr. 52 min. 6 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File /u01/app/grid/diag/tnslsnr/rac1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.101)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.103)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "rac" has 1 instance(s).
Instance "rac1", status READY, has 1 handler(s) for this service...
Service "racXDB" has 1 instance(s).
Instance "rac1", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@rac1 ~]$
dbca.rsp配置文件如下
[GENERAL] RESPONSEFILE_VERSION = "11.2.0" OPERATION_TYPE = "createDatabase" [CREATEDATABASE] GDBNAME = "rac" SID = "rac" NODELIST=rac1,rac2 TEMPLATENAME = "General_Purpose.dbc" SYSPASSWORD = "oracle" SYSTEMPASSWORD = "oracle" SYSMANPASSWORD = "oracle" DBSNMPPASSWORD = "oracle" STORAGETYPE=ASM DISKGROUPNAME=DATA ASMSNMP_PASSWORD="oracle" #RECOVERYGROUPNAME=ARCH CHARACTERSET = "ZHS16GBK" NATIONALCHARACTERSET= "UTF8"
参数配置：TEMPLATENAME = "General_Purpose.dbc" General_Purpose.dbc文件可在$ORACLE_HOME//assistants/dbca/templates/中找到：
[oracle@rac2 response]$ ls -lh /u01/app/oracle/product/11.2.0/db_1/assistants/dbca/templates/ total 295M -rw-r--r-- 1 oracle oinstall 5.0K Aug 24 2013 Data_Warehouse.dbc -rwxr-xr-x 1 oracle oinstall 21M Aug 27 2013 example01.dfb -rwxr-xr-x 1 oracle oinstall 1.5M Aug 27 2013 example.dmp -rw-r--r-- 1 oracle oinstall 4.9K Aug 24 2013 General_Purpose.dbc -rw-r--r-- 1 oracle oinstall 12K May 1 2013 New_Database.dbt -rwxr-xr-x 1 oracle oinstall 9.3M Aug 27 2013 Seed_Database.ctl -rwxr-xr-x 1 oracle oinstall 263M Aug 27 2013 Seed_Database.dfb [oracle@rac2 response]$ [oracle@rac1 ~]$ cat /u01/dbca.rsp [GENERAL] RESPONSEFILE_VERSION = "11.2.0" OPERATION_TYPE = "createDatabase" [CREATEDATABASE] GDBNAME = "rac" SID = "rac" NODELIST=rac1,rac2 TEMPLATENAME = "General_Purpose.dbc" SYSPASSWORD = "oracle" SYSTEMPASSWORD = "oracle" SYSMANPASSWORD = "oracle" DBSNMPPASSWORD = "oracle" STORAGETYPE=ASM [oracle@rac1 ~]$ dbca -silent -responsefile /u01/dbca.rsp Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/rac1.log" for further details. [oracle@rac1 ~]$ cat /u01/app/oracle/cfgtoollogs/dbca/rac1.log Could not connect to ASM due to following error: ORA-12547: TNS:lost contact [oracle@rac1 ~]$ ls -l /u01/app/11.2.0/grid/bin/oracle -rwxr-x--x 1 grid oinstall 209836184 Aug 24 06:01 /u01/app/11.2.0/grid/bin/oracle [oracle@rac1 ~]$ chmod +s /u01/app/11.2.0/grid/bin/oracle chmod: changing permissions of ‘/u01/app/11.2.0/grid/bin/oracle’: Operation not permitted [oracle@rac1 ~]$ su - grid Password: Last login: Tue Aug 25 01:21:15 EDT 2020 on pts/1 [grid@rac1 ~]$ chmod +s /u01/app/11.2.0/grid/bin/oracle [grid@rac1 ~]$ ls -l /u01/app/11.2.0/grid/bin/oracle -rwsr-s--x 1 grid oinstall 209836184 Aug 24 06:01 /u01/app/11.2.0/grid/bin/oracle [grid@rac1 ~]$ [oracle@rac1 ~]$ dbca -silent -responsefile /u01/dbca.rsp Copying database files 1% complete 3% complete 9% complete 15% complete 21% complete 27% complete 30% complete Creating and starting Oracle instance 32% complete 36% complete 40% complete 44% complete 45% complete 48% complete 50% complete Creating cluster database views 52% complete 70% complete Completing Database Creation 73% complete 76% complete 85% complete 94% complete 100% complete Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/rac/rac.log" for further details.
出现 Could not connect to ASM due to following error: ORA-12547: TNS:lost contact
检查两节点 文件 /u01/app/11.2.0/grid/bin/oracle权限是否有所有者和所属组是否有s权限
[oracle@rac2 oracle]$ ls -l /u01/app/11.2.0/grid/bin/oracle && ssh rac1 ls -l /u01/app/11.2.0/grid/bin/oracle -rwsr-s--x 1 grid oinstall 209836184 Aug 24 06:02 /u01/app/11.2.0/grid/bin/oracle -rwsr-s--x 1 grid oinstall 209836184 Aug 24 06:01 /u01/app/11.2.0/grid/bin/oracle [oracle@rac2 oracle]$
安装后测试登录检查实例状态
[oracle@rac1 ~]$ sqlplus system/oracle@rac-san:1521/rac
SQL*Plus: Release 11.2.0.4.0 Production on Tue Aug 25 02:19:20 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> SQL> desc gv$instance Name Null? Type ----------------------------------------- -------- ---------------------------- INST_ID NUMBER INSTANCE_NUMBER NUMBER INSTANCE_NAME VARCHAR2(16) HOST_NAME VARCHAR2(64) VERSION VARCHAR2(17) STARTUP_TIME DATE STATUS VARCHAR2(12) PARALLEL VARCHAR2(3) THREAD# NUMBER ARCHIVER VARCHAR2(7) LOG_SWITCH_WAIT VARCHAR2(15) LOGINS VARCHAR2(10) SHUTDOWN_PENDING VARCHAR2(3) DATABASE_STATUS VARCHAR2(17) INSTANCE_ROLE VARCHAR2(18) ACTIVE_STATE VARCHAR2(9) BLOCKED VARCHAR2(3) SQL> select host_name,status,instance_name from gv$instance 2 / HOST_NAME STATUS ---------------------------------------------------------------- ------------ INSTANCE_NAME ---------------- rac1 OPEN rac1 rac2 OPEN rac2 SQL> column host_name format a30 SQL> / HOST_NAME STATUS INSTANCE_NAME ------------------------------ ------------ ---------------- rac1 OPEN rac1 rac2 OPEN rac2 SQL>
（八）安装后测试
srvctl 命令示例：
[grid@rac2 ~]$ srvctl status instance -d rac -i rac1,rac2
Instance rac1 is running on node rac1
Instance rac2 is running on node rac2
[grid@rac2 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rac2,rac1
[grid@rac2 ~]$ srvctl status asm -a
ASM is running on rac2,rac1
ASM is enabled.
[grid@rac2 ~]$ srvctl status database -d rac
Instance rac1 is running on node rac1
Instance rac2 is running on node rac2
[grid@rac2 ~]$ srvctl status diskgroup -g DATA
Disk Group DATA is running on rac2,rac1
[grid@rac2 ~]$ srvctl status scan_listener -i 1 -v
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac2
[grid@rac2 ~]$
测试oracle 操作系统验证执行如下
[oracle@rac1 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 15 22:37:05 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> 查看节点信息 SQL> desc gv$instance Name Null? Type ----------------------------------------- -------- ---------------------------- INST_ID NUMBER INSTANCE_NUMBER NUMBER INSTANCE_NAME VARCHAR2(16) HOST_NAME VARCHAR2(64) VERSION VARCHAR2(17) STARTUP_TIME DATE STATUS VARCHAR2(12) PARALLEL VARCHAR2(3) THREAD# NUMBER ARCHIVER VARCHAR2(7) LOG_SWITCH_WAIT VARCHAR2(15) LOGINS VARCHAR2(10) SHUTDOWN_PENDING VARCHAR2(3) DATABASE_STATUS VARCHAR2(17) INSTANCE_ROLE VARCHAR2(18) ACTIVE_STATE VARCHAR2(9) BLOCKED VARCHAR2(3) SQL> column host_name format a10 SQL> select host_name ,instance_name,status,database_status from gv$instance 2 / HOST_NAME INSTANCE_NAME STATUS DATABASE_STATUS ---------- ---------------- ------------ ----------------- rac1 rac1 OPEN ACTIVE rac2 rac2 OPEN ACTIVE SQL> 使用ip端口服务名连接 [oracle@rac1 ~]$ sqlplus system/oracle@192.168.2.105:1521/rac SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 15 22:38:34 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL>
查看监听运行在节点rac2上，
[grid@rac2 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac2
测试把rac2上的两个网卡全部停用,然后再测oracle连接
执行如下命令 ifdown ens192, ifdown ens224
rac2的远程已经失去连接
登录rac1测试登录Oracle
[oracle@rac1 ~]$ sqlplus system/oracle@192.168.2.105:1521/rac
SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 15 23:21:56 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> desc gv$instance
Name Null? Type
----------------------------------------- -------- ----------------------------
INST_ID NUMBER
INSTANCE_NUMBER NUMBER
INSTANCE_NAME VARCHAR2(16)
HOST_NAME VARCHAR2(64)
VERSION VARCHAR2(17)
STARTUP_TIME DATE
STATUS VARCHAR2(12)
PARALLEL VARCHAR2(3)
THREAD# NUMBER
ARCHIVER VARCHAR2(7)
LOG_SWITCH_WAIT VARCHAR2(15)
LOGINS VARCHAR2(10)
SHUTDOWN_PENDING VARCHAR2(3)
DATABASE_STATUS VARCHAR2(17)
INSTANCE_ROLE VARCHAR2(18)
ACTIVE_STATE VARCHAR2(9)
BLOCKED VARCHAR2(3)
SQL> column host_name format a10
SQL> select host_name,instance_name.status.database_status from gv$instance
2 /
select host_name,instance_name.status.database_status from gv$instance
*
ERROR at line 1:
ORA-00904: "INSTANCE_NAME"."STATUS"."DATABASE_STATUS": invalid identifier
SQL> c/./,
1* select host_name,instance_name,status.database_status from gv$instance
SQL> c/./,
1* select host_name,instance_name,status,database_status from gv$instance
SQL> /
HOST_NAME INSTANCE_NAME STATUS DATABASE_STATUS
---------- ---------------- ------------ -----------------
rac1 rac1 OPEN ACTIVE
SQL>
查看rac1上的ip地址，查看192.168.2.105 rac-san已经被rac1节点接管，
测试完毕。
（九）安装问题解决及总结
9.1：执行root.sh脚本时候报错
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
解决方案：安装oracle补丁p18370031_112040_Linux-x86-64.zip
oracle 官方对于11.2.0.4grid 解释：
https://docs.oracle.com/cd/E11882_01/relnotes.112/e23558/toc.htm#CJAJEBGG
7.2.3 Oracle Grid Infrastructure Installation Issue
During the Oracle Grid Infrastructure installation, you must apply patch 18370031 before configuring the software that is installed. The timing of applying the patch is important and is described in detail in the Note 1951613.1 on My Oracle Support. This patch ensures that the clusterware stack is configured to use systemd for clusterware processes, as Oracle Linux 7 uses systemd for all services.
This issue is tracked with Oracle bug 18370031, which was logged for release 12.1.0.2, but the patch is for release 11.2.0.4.
错误解决，安装补丁18370031，操作步骤如下
- 1root下执行yum install perl*
- 2.执行脚本 回退root.sh操作/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
- 3.解压压缩包，切换到18370031目录执行 /u01/app/11.2.0/grid/OPatch/opatch apply 完成更新，输出没有报错信息，重新运行root脚本。
执行步骤如下：下载更新包，解压后切换到更新包路径，执行如下：
[grid@rac1 ~]$ cd 18370031/
[grid@rac1 18370031]$
[grid@rac1 18370031]$ ls
custom etc files
[grid@rac1 18370031]$ /u01/app/11.2.0/grid/OPatch/opatch apply
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.0/grid/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/18370031_Jul_24_2020_18_36_32/apply2020-07-24_18-36-32PM_1.log
Applying interim patch '18370031' to OH '/u01/app/11.2.0/grid'
Verifying environment and performing prerequisite checks...
All checks passed.
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'rac2'
Local node: 'rac1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/11.2.0/grid')
Is the local system ready for patching? [y|n]
Could not recognize input. Please re-enter.
y
User Responded with: Y
Backing up files...
Patching component oracle.crs, 11.2.0.4.0...
Verifying the update...
The local system has been patched. You can restart Oracle instances on it.
Patching in rolling mode.
The node 'rac2' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac2'.
(Oracle Home = '/u01/app/11.2.0/grid')
Is the node ready for patching? [y|n]
Y
User Responded with: Y
Updating nodes 'rac2'
Apply-related files are:
FP = "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/copy_files.txt"
DP = "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/copy_dirs.txt"
MP = "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/make_cmds.txt"
RC = "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/remote_cmds.txt"
Instantiating the file "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/11.2.0/grid/.patch_storage/18370031_Aug_15_2014_16_14_40/rac/make_cmds.txt" with actual path.
Running command on remote node 'rac2':
cd /u01/app/11.2.0/grid/srvm/lib; /usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=/u01/app/11.2.0/grid || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'rac2':
cd /u01/app/11.2.0/grid/racg/lib; /usr/bin/make -f ins_has.mk install ORACLE_HOME=/u01/app/11.2.0/grid || echo REMOTE_MAKE_FAILED::>&2
The node 'rac2' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'rac2' .
The following make commands were invoked on remote nodes:
'cd /u01/app/11.2.0/grid/srvm/lib; /usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=/u01/app/11.2.0/grid
cd /u01/app/11.2.0/grid/racg/lib; /usr/bin/make -f ins_has.mk install ORACLE_HOME=/u01/app/11.2.0/grid
'
Patch 18370031 successfully applied
Log file location: /u01/app/11.2.0/grid/cfgtoollogs/opatch/18370031_Jul_24_2020_18_36_32/apply2020-07-24_18-36-32PM_1.log
OPatch succeeded.
[grid@rac1 18370031]$
更新完毕。
9.2：安装oracle软件时候报错：INS-35354
错误解释：安装oracle提示信息如下：
The system on which you are attempting to install Oracle RAC is not part of a valid cluster.
解决方案：编辑/u01/app/oraInventory/ContentsXML/inventory.xml，找到属性 <HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1">添加CRS="true"属性，两个节点都要修改，修改完毕重新执行安装
如下
[oracle@rac2 ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
<SAVED_WITH>11.2.0.4.0</SAVED_WITH>
<MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
<NODE_LIST>
<NODE NAME="rac1"/>
<NODE NAME="rac2"/>
</NODE_LIST>
</HOME>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/db_1" TYPE="O" IDX="2">
<NODE_LIST>
<NODE NAME="rac1"/>
<NODE NAME="rac2"/>
</NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@rac2 ~]$
9.3 附加链接静默安装Oracle
参考个人新浪博客链接如下：
静默安装Oracle（，由于新浪博客无法修改博文，其中个别参数写的有错误，其中错误无法修复），仅供参考
http://blog.sina.com.cn/s/blog_1722831d10102xm4x.html
附加参考：
oracle 11.2.0.4说明：
只安装oracle仅仅需要下载p13390677_112040_Linux-x86-64_1of7.zip和p13390677_112040_Linux-x86-64_2of7.zip，安装grid需要下载p13390677_112040_Linux-x86-64_3of7.zip， 可运行grid安装包中的readme.html查看。
（十） asm增加磁盘操作
ssh登录esxi主机命令如下
- 虚拟机中配置
两节点都要配置
配置完毕。
- rac节点linux操作
登录linux (两节点都要配置），操作如下：
执行fdisk -l 查看如下：发现新增的两块磁盘
节点一
节点二
对磁盘进行创建分区
对磁盘 /dev/sdi /dev/sdj 进行创建分区，此操作只在一个节点上执行即可。
编辑文件 /usr/lib/udev/rules.d/60-raw.rules
编辑新增的两块磁盘
vi /usr/lib/udev/rules.d/60-raw.rules
修改完毕使用scp命令拷贝到rac2节点上
scp /usr/lib/udev/rules.d/60-raw.rules rac2:/usr/lib/udev/rules.d/60-raw.rules
修改完毕，
先重启rac1，节点一，节点一重启完毕，再重启二节点，重启期间oracle服务不会中断。
重启后执行ls -l /dev/raw/ 查看新增设备。
- grid用户sql命令示例
操作步骤
操作系统切换grid 用户，sqlplus / as sysasm 登录，操作如下
贴入文本操作如下：
[root@rac1 ~]# su - grid
Last login: Fri Jul 17 01:03:00 EDT 2020 on pts/1
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 17 01:40:36 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL>
SQL>
SQL>
SQL> set lines 200
SQL> column path format a50
SQL> select group_number ,disk_number,mode_status ,name ,path from v$asm_disk
2 /
GROUP_NUMBER DISK_NUMBER MODE_ST NAME PATH
------------ ----------- ------- ------------------------------ --------------------------------------------------
0 1 ONLINE /dev/raw/raw9
0 2 ONLINE /dev/raw/raw8
1 0 ONLINE DATA_0000 /dev/raw/raw1
1 3 ONLINE DATA_0004 /dev/raw/raw4
1 6 ONLINE DATA_0007 /dev/raw/raw7
1 5 ONLINE DATA_0006 /dev/raw/raw6
1 4 ONLINE DATA_0005 /dev/raw/raw5
1 2 ONLINE DATA_0003 /dev/raw/raw3
1 1 ONLINE DATA_0001 /dev/raw/raw2
9 rows selected.
SQL> alter diskgroup DATA add disk '/dev/raw/raw8' name DATA_0008
2 /
Diskgroup altered.
SQL> c/8/9
1* alter diskgroup DATA add disk '/dev/raw/raw9' name DATA_0008
SQL> c/8/9
1* alter diskgroup DATA add disk '/dev/raw/raw9' name DATA_0009
SQL> /
Diskgroup altered.
SQL> select group_number ,disk_number,mode_status ,name ,path from v$asm_disk
2 /
GROUP_NUMBER DISK_NUMBER MODE_ST NAME PATH
------------ ----------- ------- ------------------------------ --------------------------------------------------
1 0 ONLINE DATA_0000 /dev/raw/raw1
1 8 ONLINE DATA_0009 /dev/raw/raw9
1 7 ONLINE DATA_0008 /dev/raw/raw8
1 3 ONLINE DATA_0004 /dev/raw/raw4
1 6 ONLINE DATA_0007 /dev/raw/raw7
1 5 ONLINE DATA_0006 /dev/raw/raw6
1 4 ONLINE DATA_0005 /dev/raw/raw5
1 2 ONLINE DATA_0003 /dev/raw/raw3
1 1 ONLINE DATA_0001 /dev/raw/raw2
9 rows selected.
SQL> c/name/name ,total_mb
1* select group_number ,disk_number,mode_status ,name ,total_mb ,path from v$asm_disk
SQL> /
GROUP_NUMBER DISK_NUMBER MODE_ST NAME TOTAL_MB PATH
------------ ----------- ------- ------------------------------ ---------- --------------------------------------------------
1 0 ONLINE DATA_0000 10239 /dev/raw/raw1
1 8 ONLINE DATA_0009 20479 /dev/raw/raw9
1 7 ONLINE DATA_0008 20479 /dev/raw/raw8
1 3 ONLINE DATA_0004 10239 /dev/raw/raw4
1 6 ONLINE DATA_0007 1023 /dev/raw/raw7
1 5 ONLINE DATA_0006 1023 /dev/raw/raw6
1 4 ONLINE DATA_0005 1023 /dev/raw/raw5
1 2 ONLINE DATA_0003 10239 /dev/raw/raw3
1 1 ONLINE DATA_0001 10239 /dev/raw/raw2
9 rows selected.
SQL>
ASM Creating Disk Groups 操作参考官方链接如下：
https://docs.oracle.com/cd/E11882_01/server.112/e18951/asmdiskgrps.htm#OSTMG10040
完毕
（十一）参考链接
- 官方链接：
安装参考
Grid Infrastructure Installation Guide for linux
https://docs.oracle.com/cd/E11882_01/install.112/e41961/toc.htm
Database 2 Day + Real Application Clusters Guide
https://docs.oracle.com/cd/E11882_01/rac.112/e17264/install_rac.htm#TDPRC169
命令参考
sqlplus 命令
https://docs.oracle.com/cd/E11882_01/server.112/e16604/toc.htm
crsctl命令参考
https://docs.oracle.com/cd/E11882_01/rac.112/e41959/crsref.htm#CWADD91145
srvctl 命令参考
https://docs.oracle.com/cd/E11882_01/rac.112/e41960/srvctladmin.htm#RACAD005
grid必须的包官方参考：
官方链接
https://docs.oracle.com/cd/E11882_01/install.112/e41961/prelinux.htm#CWLIN225
参考官方创建组和用户，链接如下：
https://docs.oracle.com/cd/E11882_01/install.112/e41961/prelinux.htm#CWLIN178
本文中使用的oracle 11.2.0.4 安装包及补丁包如下
百度云链接：
链接：https://pan.baidu.com/s/1WvdpiTs9m3es5vyTBzHb4w
提取码：1111
本实验环境，安装基本操作脚本示例：
https://files.cnblogs.com/files/wenxiao1-2-3-4/OracleRAC%E8%84%9A%E6%9C%AC.zip
操作系统下载链接：
centos镜像下载
http://mirrors.sohu.com/centos/7.6.1810/isos/x86_64/
redhat
链接：https://pan.baidu.com/s/16-TgKVk_nAxLeakRsOPKkg
提取码：1111
技术交流：
邮箱链接：wenxiaomst@outlook.com
wechat ：w8686512
