Preface
Oracle claimed that 11g RAC is supported on Redhat Linux 7 and above version,but there're still some incompatible issues in inplementing 11g GI on it.There're some main steps showed below.
Precedure
###Step 1.PREPARE PHASE###
1. Modify /etc/hosts(all nodes).
1 vim /etc/hosts 2 cat /etc/hosts 3 #public ip 4 192.168.56.11 rac1 rac1-pub 5 192.168.56.12 rac2 rac2-pub 6 #private ip 7 10.10.10.11 rac1-priv 8 10.10.10.12 rac2-priv 9 #virtual ip 10 192.168.56.13 rac1-vip 11 192.168.56.14 rac2-vip 12 #scan ip 13 192.168.56.15 rac-scan
2. Disable firewall & selinux(all nodes).
1 systemctl stop firewalld 2 systemctl status firewalld 3 systemctl status firewalld 4 vim /etc/selinux/config 5 cat /etc/selinux/config 6 SELINUX=disabled 7 sestatus -v 8 SELinux status: disabled
3. Install packages(all nodes).
1 yum -y install binutils compat-db compat-libcap1 compat-libstdc++-33 control-center elfutils-libelf-devel gcc gcc-c++ glibc glibc-common gnome-libs libaio-devel libstdc++ libstdc++-devel make pdksh sysstat xscreensaver libaio openmotif21 smartmontools
4. Configure ssh connectivities(all nodes,both grid & oracle).
1 rac1: 2 ssh-keygen -t rsa 3 rac2: 4 ssh-keygen -t rsa 5 rac1: 6 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 7 ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 8 rac2: 9 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 10 ssh rac1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 11 12 //Test execute command on remote host without password. 13 ssh rac1 date 14 ssh rac2 date
5. Add user & usergroups(all nodes).
1 groupadd oinstall 2 groupadd dba 3 groupadd oper 4 groupadd asmdba 5 groupadd asmoper 6 groupadd asmadmin 7 useradd -g oinstall -G dba,oper,asmdba oracle 8 useradd -g oinstall -G asmdba,dba,asmadmin,asmoper grid
6. Create directory & grant privileges(all nodes).
1 mkdir /u01 2 mkdir /u01/gridbase 3 mkdir /u01/gridhome 4 mkdir /u01/oracle 5 chown -R grid:oinstall /u01 6 chown -R oracle:oinstall /u01/oracle 7 chmod -R g+w /u01
7. Modify environment variables((all nodes,both grid & oracle).
1 grid(rac1): 2 vim /home/grid/.bash_profile 3 cat /home/grid/.bash_profile 4 ORACLE_BASE = /u01/gridbase 5 ORACLE_HOME = /u01/gridhome 6 ORACLE_SID = +ASM1 7 PATH = $ORACLE_HOME/bin:$PATH 8 LD_LIBRARY_PATH = $ORACLE_HOME/lib:$LD_LIBRARY_PATH 9 export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH LD_LIBRARY_PATH 10 11 grid(rac2): 12 vim /home/grid/.bash_profile 13 cat /home/grid/.bash_profile 14 ORACLE_BASE = /u01/gridbase 15 ORACLE_HOME = /u01/gridhome 16 ORACLE_SID = +ASM2 17 PATH=$ORACLE_HOME/bin:$PATH 18 LD_LIBRARY_PATH = $ORACLE_HOME/lib:$LD_LIBRARY_PATH 19 export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH LD_LIBRARY_PATH 20 21 oracle(rac1): 22 vim /home/oracle/.bash_profile 23 cat /home/oracle/.bash_profile 24 ORACLE_BASE = /u01/oracle 25 ORACLE_HOME = /u01/oracle/db 26 ORACLE_SID = ora11g1 27 PATH = $ORACLE_HOME/bin:$PATH 28 LD_LIBRARY_PATH = $ORACLE_HOME/lib:$LD_LIBRARY_PATH 29 export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH LD_LIBRARY_PATH 30 31 oracle(rac2): 32 vim /home/oracle/.bash_profile 33 cat /home/oracle/.bash_profile 34 ORACLE_BASE = /u01/oracle 35 ORACLE_HOME = /u01/oracle/db 36 ORACLE_SID = ora11g2 37 PATH = $ORACLE_HOME/bin:$PATH 38 LD_LIBRARY_PATH = $ORACLE_HOME/lib:$LD_LIBRARY_PATH 39 export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH LD_LIBRARY_PATH 40 41 //Make it take effect(rac1 & rac2). 42 source /home/oracle/.bash_profile 43 source /home/grid/.bash_profile
8. Modify kernel parameters(all nodes).
1 vim /etc/sysctl.conf 2 cat vim /etc/sysctl.conf 3 kernel.msgmnb = 65536 4 kernel.msgmax = 65536 5 kernel.shmmax = 1073741824 //1G memory for Oracle. 6 kernel.shmall = 2097152 7 kernel.shmmni = 4096 8 fs.aio-max-nr = 1048576 9 fs.file-max = 6815744 10 kernel.sem = 250 32000 100 128 11 net.ipv4.ip_local_port_range = 9000 65500 12 net.core.rmem_default = 262144 13 net.core.rmem_max = 4194304 14 net.core.wmem_default = 262144 15 net.core.wmem_max = 1048576 16 17 //Make the kernel parameters take effect. 18 sysctl -p
9. Modify limites(all nodes).
1 vim /etc/security/limits.conf 2 cat /etc/security/limits.conf 3 #oracle 4 oracle soft nproc 2047 5 oracle hard nproc 16384 6 oracle soft nofile 1024 7 oracle hard nofile 65536 8 oracle soft stack 10240 9 #grid 10 grid soft nproc 2047 11 grid hard nproc 16384 12 grid soft nofile 1024 13 grid hard nofile 65536 14 grid soft stack 10240
10. Modify pam limits(all nodes).
1 vim /etc/pam.d/login 2 cat /etc/pam.d/login 3 session required /lib64/security/pam_limits.so 4 session required pam_limits.so
11. Modify profile(all nodes).
1 vim /etc/profile 2 cat /etc/profile 3 if [ $USER = "oracle" ]||[$USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else 4 43/ 136 5 ulimit -u 16384 -n 65536 6 fi 7 fi
###Step 2.CREATE SHARED DISKS WITH ASMLIB###
1. Create three new disks and make partitions on them with "fdisk"(rac1 only).
1 fdisk -l 2 fdisk /dev/sdc 3 n 4 enter 5 enter 6 enter 7 w 8 9 fdisk /dev/sdd 10 n 11 enter 12 enter 13 enter 14 w 15 16 fdisk /dev/sde 17 n 18 enter 19 enter 20 enter 21 w 22 23 ll /dev/sd* 24 ... 25 /dev/sdc1 26 /dev/sdd1 27 /dev/sde1
2. Install asmlib packages(both rac1 & rac2).
1 yum -y localinstall kmod-oracleasm oracleasm-support oracleasmlib
3. Bind the shared disks with "oracleasm"(root execute).
1 rac1: 2 oracleasm configure -i 3 grid 4 asmadmin 5 y 6 y 7 8 oracleasm init 9 10 oracleasm createdisk ASMDISK1 sdc1 11 oracleasm createdisk ASMDISK2 sdd1 12 oracleasm createdisk ASMDISK3 sde1 13 14 oracleasm listdisks 15 ASMDISK1 16 ASMDISK2 17 ASMDISK3 18 19 oracleasm scandisks 20 21 rac2: 22 oracleasm configure -i 23 grid 24 asmadmin 25 y 26 y 27 28 oracleasm init 29 30 //Reboot rac2 and check the shared disks. 31 oracleasm listdisks 32 ASMDISK1 33 ASMDISK2 34 ASMDISK3
###Step 3.SOFTWARE INSTALLATION(rac1 only)###
1. Unzip the grid software package.
1 cd /u01 2 unzip p13390677_112040_Linux-x86-64_3of7.zip 3 chown -R grid:oinstall /u01/grid
2. Check the prerequisite of installation.
1 cd /u01/grid 2 ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -fixup -verbose
3. Install X windows System of linux.
1 yum -y groupinstall "X Windows System"
4. Set xhost and DISPLAY variable.
1 xhost + 2 export DISPLAY=192.168.56.99:0.0
5. Install grid software with Oracle GUI.
1 ./runInstaller -ignorePrereq 2 3 //I've tried many times not add "-ignorePrereq" option,but the OUI will stuck at 61% when checking the semaphore.I'm afraid it's the incompatible poit installing 11g rac on CentOS 7.2.The picture of stuck point shows below.
6. Execute below scripts in RAC1,RAC2 in order(This step really took me quit a long time to solve it,So I'd like to put the details below).
1 /u01/oraInventory/orainstRoot.sh 2 /u01/gridhome/root.sh 3 4 I got an error bellow when execute "/u01/gridhome/root.sh" script: 5 6 Adding Clusterware entries to inittab 7 ohasd failed to start 8 Failed to start the Clusterware. Last 20 lines of the alert log follow: 9 2018-08-27 03:07:20.078: 10 [client(2762)]CRS-2101:The OLR was formatted using version 3. 11 2018-08-27 03:13:24.742: 12 [client(3948)]CRS-2101:The OLR was formatted using version 3. 13 2018-08-27 03:17:25.615: 14 [client(5037)]CRS-2101:The OLR was formatted using version 3. 15 16 //It's an oracle bug 18370031,a patch should be applied before execute the "root.sh". 17 18 su - root 19 chown -R grid:oinstall /u01/ 20 su - grid 21 export PATH=$PATH:$ORACLE_HOME/OPatch 22 mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch112034 23 cd /u01 24 unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME 25 $ORACLE_HOME/OPatch/opatch version 26 OPatch Version: 11.2.0.3.19 27 28 OPatch succeeded. 29 30 cd ~ 31 $ORACLE_HOME/OPatch/ocm/bin/emocmrsp 32 OCM Installation Response Generator 10.3.7.0.0 - Production 33 Copyright (c) 2005, 2012, Oracle and/or its affiliates. All rights reserved. 34 35 Provide your email address to be informed of security issues, install and 36 initiate Oracle Configuration Manager. Easier for you if you use your My 37 Oracle Support Email address/User Name. 38 Visit http://www.oracle.com/support/policies.html for details. 39 Email address/User Name: 40 41 You have not provided an email address for notification of security issues. 42 Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y 43 The OCM configuration response file (ocm.rsp) was successfully created. 44 45 unzip p18370031_112040_Linux-x86-64.zip 46 opatch apply /u01/18370031/ -oh $ORACLE_HOME -ocmrf /home/grid/ocm.rsp 47 Oracle Interim Patch Installer version 11.2.0.3.19 48 Copyright (c) 2018, Oracle Corporation. All rights reserved. 49 50 51 Oracle Home : /u01/gridhome 52 Central Inventory : /u01/oraInventory 53 from : /u01/gridhome/oraInst.loc 54 OPatch version : 11.2.0.3.19 55 OUI version : 11.2.0.4.0 56 Log file location : /u01/gridhome/cfgtoollogs/opatch/opatch2018-08-27_06-28-31AM_1.log 57 58 Verifying environment and performing prerequisite checks... 59 OPatch continues with these patches: 18370031 60 61 Do you want to proceed? [y|n] 62 y 63 User Responded with: Y 64 All checks passed. 65 66 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. 67 (Oracle Home = '/u01/gridhome') 68 69 70 Is the local system ready for patching? [y|n] 71 y 72 User Responded with: Y 73 Backing up files... 74 Applying interim patch '18370031' to OH '/u01/gridhome' 75 76 Patching component oracle.crs, 11.2.0.4.0... 77 78 Copy failed from '/u01/18370031/files/bin/ohasd.bin' to '/u01/gridhome/bin/ohasd.bin'... 79 Please verify all applications associated with the Oracle Home '/u01/gridhome' are shut down. If this is AIX, please perform solution documented in Note 739963.1 on https://myoraclesupport.oracle.com. 80 81 Do you want to retry copying the file? [y|n] 82 y 83 User Responded with: Y 84 85 Retry copying the file to '/u01/gridhome/bin/ohasd.bin'. 86 87 The following actions have failed: 88 Copy failed from '/u01/18370031/files/bin/ohasd.bin' to '/u01/gridhome/bin/ohasd.bin'... 89 90 91 Do you want to proceed? [y|n] 92 Y 93 User Responded with: Y 94 Patch 18370031 successfully applied. 95 OPatch Session completed with warnings. 96 Log file location: /u01/gridhome/cfgtoollogs/opatch/opatch2018-08-27_06-28-31AM_1.log 97 98 OPatch completed with warnings. 99 [grid@rac2 u01]$ ps -ef|grep ohasd 100 root 20730 1 0 06:08 ? 00:00:00 /u01/gridhome/bin/ohasd.bin reboot 101 grid 21998 21139 0 06:32 pts/0 00:00:00 grep --color=auto ohasd 102 103 //If you want to copy it right now,then kill the ohasd process first(I didn't do that). 104 105 kill -9 20730 106 cp /u01/18370031/ohasd.bin /u01/gridhome/bin/ohasd.bin 107 108 //If you are using OEL 7.2,there's another patch "19404309" need to apply to guarantee the GRID Softerware can be normally implement. 109 110 [root@rac1 ~]# /u01/gridhome/root.sh 111 Performing root user operation for Oracle 11g 112 113 The following environment variables are set as: 114 ORACLE_OWNER= grid 115 ORACLE_HOME= /u01/gridhome 116 117 Enter the full pathname of the local bin directory: [/usr/local/bin]: 118 The contents of "dbhome" have not changed. No need to overwrite. 119 The contents of "oraenv" have not changed. No need to overwrite. 120 The contents of "coraenv" have not changed. No need to overwrite. 121 122 Entries will be added to the /etc/oratab file as needed by 123 Database Configuration Assistant when a database is created 124 Finished running generic part of root script. 125 Now product-specific root actions will be performed. 126 Using configuration parameter file: /u01/gridhome/crs/install/crsconfig_params 127 User ignored Prerequisites during installation 128 Installing Trace File Analyzer 129 Adding Clusterware entries to oracle-ohasd.service 130 CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1' 131 CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded 132 CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1' 133 CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded 134 CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1' 135 CRS-2672: Attempting to start 'ora.gipcd' on 'rac1' 136 CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded 137 CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded 138 CRS-2672: Attempting to start 'ora.cssd' on 'rac1' 139 CRS-2672: Attempting to start 'ora.diskmon' on 'rac1' 140 CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded 141 CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded 142 143 ASM created and started successfully. 144 145 Disk Group OCRVOTE created successfully. 146 147 clscfg: -install mode specified 148 Successfully accumulated necessary OCR keys. 149 Creating OCR keys for user 'root', privgrp 'root'.. 150 Operation successful. 151 CRS-4256: Updating the profile 152 Successful addition of voting disk a11c41b9357d4f48bf1b7c5e03c1510a. 153 Successfully replaced voting disk group with +OCRVOTE. 154 CRS-4256: Updating the profile 155 CRS-4266: Voting file(s) successfully replaced 156 ## STATE File Universal Id File Name Disk group 157 -- ----- ----------------- --------- --------- 158 1. ONLINE a11c41b9357d4f48bf1b7c5e03c1510a (ORCL:ASMDISK1) [OCRVOTE] 159 Located 1 voting disk(s). 160 sh: /bin/netstat: No such file or directory 161 CRS-2672: Attempting to start 'ora.asm' on 'rac1' 162 CRS-2676: Start of 'ora.asm' on 'rac1' succeeded 163 CRS-2672: Attempting to start 'ora.OCRVOTE.dg' on 'rac1' 164 CRS-2676: Start of 'ora.OCRVOTE.dg' on 'rac1' succeeded 165 Preparing packages... 166 ls: cannot access /usr/sbin/smartctl: No such file or directory 167 /usr/sbin/smartctl not found. 168 error: %pre(cvuqdisk-1.0.9-1.x86_64) scriptlet failed, exit status 1 169 error: cvuqdisk-1.0.9-1.x86_64: install failed 170 Configure Oracle Grid Infrastructure for a Cluster ... succeeded 171 [root@rac1 ~]# crs_stat -t 172 Name Type Target State Host 173 ------------------------------------------------------------ 174 ora....N1.lsnr ora....er.type ONLINE ONLINE rac1 175 ora.OCRVOTE.dg ora....up.type ONLINE ONLINE rac1 176 ora.asm ora.asm.type ONLINE ONLINE rac1 177 ora.cvu ora.cvu.type ONLINE ONLINE rac1 178 ora.gsd ora.gsd.type OFFLINE OFFLINE 179 ora....network ora....rk.type ONLINE ONLINE rac1 180 ora.oc4j ora.oc4j.type ONLINE ONLINE rac1 181 ora.ons ora.ons.type ONLINE ONLINE rac1 182 ora....SM1.asm application ONLINE ONLINE rac1 183 ora.rac1.gsd application OFFLINE OFFLINE 184 ora.rac1.ons application ONLINE ONLINE rac1 185 ora.rac1.vip ora....t1.type ONLINE ONLINE rac1 186 ora.scan1.vip ora....ip.type ONLINE ONLINE rac1 187 188 //Omitted the procedure of apply patch "18370031" on RAC2. 189 190 [root@rac2 ~]# /u01/gridhome/root.sh 191 Performing root user operation for Oracle 11g 192 193 The following environment variables are set as: 194 ORACLE_OWNER= grid 195 ORACLE_HOME= /u01/gridhome 196 197 Enter the full pathname of the local bin directory: [/usr/local/bin]: 198 The contents of "dbhome" have not changed. No need to overwrite. 199 The contents of "oraenv" have not changed. No need to overwrite. 200 The contents of "coraenv" have not changed. No need to overwrite. 201 202 Entries will be added to the /etc/oratab file as needed by 203 Database Configuration Assistant when a database is created 204 Finished running generic part of root script. 205 Now product-specific root actions will be performed. 206 Using configuration parameter file: /u01/gridhome/crs/install/crsconfig_params 207 User ignored Prerequisites during installation 208 Installing Trace File Analyzer 209 CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2' 210 CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded 211 CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2' 212 CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded 213 CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2' 214 CRS-2672: Attempting to start 'ora.gipcd' on 'rac2' 215 CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded 216 CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded 217 CRS-2672: Attempting to start 'ora.cssd' on 'rac2' 218 CRS-2672: Attempting to start 'ora.diskmon' on 'rac2' 219 CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded 220 CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded 221 sh: /bin/netstat: No such file or directory 222 Preparing packages... 223 ls: cannot access /usr/sbin/smartctl: No such file or directory 224 /usr/sbin/smartctl not found. 225 error: %pre(cvuqdisk-1.0.9-1.x86_64) scriptlet failed, exit status 1 226 error: cvuqdisk-1.0.9-1.x86_64: install failed 227 Configure Oracle Grid Infrastructure for a Cluster ... succeeded 228 [root@rac2 ~]# crs_stat -t 229 Name Type Target State Host 230 ------------------------------------------------------------ 231 ora....N1.lsnr ora....er.type ONLINE ONLINE rac1 232 ora.OCRVOTE.dg ora....up.type ONLINE ONLINE rac1 233 ora.asm ora.asm.type ONLINE ONLINE rac1 234 ora.cvu ora.cvu.type ONLINE ONLINE rac1 235 ora.gsd ora.gsd.type OFFLINE OFFLINE 236 ora....network ora....rk.type ONLINE ONLINE rac1 237 ora.oc4j ora.oc4j.type ONLINE ONLINE rac1 238 ora.ons ora.ons.type ONLINE ONLINE rac1 239 ora....SM1.asm application ONLINE ONLINE rac1 240 ora.rac1.gsd application OFFLINE OFFLINE 241 ora.rac1.ons application ONLINE ONLINE rac1 242 ora.rac1.vip ora....t1.type ONLINE ONLINE rac1 243 ora....SM2.asm application ONLINE ONLINE rac2 244 ora.rac2.gsd application OFFLINE OFFLINE 245 ora.rac2.ons application ONLINE ONLINE rac2 246 ora.rac2.vip ora....t1.type ONLINE ONLINE rac2 247 ora.scan1.vip ora....ip.type ONLINE ONLINE rac1
7. Skip the subsequent failure of checking NETCA & CVU and finish the installation of GI and check crs status.
1 [root@rac1 u01]# crsctl check crs 2 CRS-4638: Oracle High Availability Services is online 3 CRS-4537: Cluster Ready Services is online 4 CRS-4529: Cluster Synchronization Services is online 5 CRS-4533: Event Manager is online 6 [root@rac2 u01]# crsctl check crs 7 CRS-4638: Oracle High Availability Services is online 8 CRS-4537: Cluster Ready Services is online 9 CRS-4529: Cluster Synchronization Services is online 10 CRS-4533: Event Manager is online