|
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 564580.1] |
||
|
|
||
|
Modified 24-JAN-2010 Type HOWTO Status PUBLISHED |
||
In this Document
Goal
Solution
Deprecation of Support for Raw Devices
A Bit About Udev and Device Name Persistency
Multipath, Raw and Udev
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
Assumptions
1. Configure SCSI_ID to Return Unique Device Identifiers
1a. Whitelist SCSI devices
1b. List all SCSI (Clusterware) devices
1c. Obtain Clusterware device unique SCSI identifiers
2. Configure Multipath for Persistent Naming of Clusterware Devices
2a. Configure Multipathing
2b. Verify Multipath Devices
3. Create Raw Devices
4. Test Raw Device Accessibility
5. Script Creation of Raw Bindings and Permissions
6. Test the Raw Device Script
7. Install Oracle 10gR2 Clusterware
References
Applies to:
Linux OS - Version: 5.0 to 5.0
Linux x86
Linux x86-64
Linux Itanium
Linux Kernel - Version: 5.0 to 5.0
Goal
This article is intended for Oracle on Linux Database and System Administrators, particularly those intending to install (or migrate to) Oracle Real Application Clusters 10g Release 2 (10.2.0) on Red Hat/Oracle Enterprise Linux 5 (EL5). The article is intended to focus on the configuration of raw devices against multipathed devices on EL5 in preparation for RAC Clusterware usage, rather than on multipathing or installation of the Clusterware.
Examples were taken from a working system of the following configuration:
- Enterprise Linux 5 (GA) - 2.6.18-8.el5
- Oracle Clusterware 10g Release 2 (10.2.0)
- Shared storage for Clusterware files served via iSCSI
Note: this document differs to Note.465001.1 that describes configuration of raw devices against single pathed devices. This Note describes configuration of raw devices against multipathed devices.
Solution
Deprecation of Support for Raw Devices
In versions prior to EL5, applications such as Oracle, could access unstructured data on block devices by binding to them via character raw devices, such as/dev/raw/raw1 using the raw(8) command. Persistent device assignments could be configured using the /etc/sysconfig/rawdevices file in conjunction with the rawdevices service.
Support for raw devices was initially deprecated in the Linux 2.6 kernel (EL5 < U4) in favour of directio (O_DIRECT) access, however was later undeprecated from EL5 U4 (initscripts-8.45.30-2).
For details of the deprecation and undeprecation of support for rawio, refer to Linux kernel/version documentation including:
- /usr/share/doc/kernel-doc-2.6.18/Documentation/feature-removal-schedule.txt
- Red Hat Enterprise Linux 4/5 Release notes
Both the /etc/sysconfig/rawdevices file (EL4) and /etc/udev/rules.d/60-raw.rules file (EL5) similarly discuss deprecation of raw.
OCFS2, Oracle's Cluster Filesystem version 2 (http://oss.oracle.com/projects/ocfs2), is an extent based, POSIX-compliant file system that provides for shared, O_DIRECT file access. For certified ports and distributions, Oracle extends free support of OCFS2 users with an Oracle database license for use in storing Oracle datafiles, redologs, archivelogs, control files, voting disk (CRS), cluster registry (OCR), etc. along with shared Oracle home.
A Bit About Udev and Device Name Persistency
Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. This can, however, give rise to the possibility that device file names may change - a device that may once have been named /dev/sdd say, may be named /dev/sdf, say, after reboot. Without specific configuration, if udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to become corrupt.
Multipath, Raw and Udev
The necessity for high availability access to storage is well understood. For singlepath environments, raw devices can easily be configured via udev rules as described in Note.465001.1. For multipath environments, however, configuration of raw devices against multipathed devices via udev is more complex. In fact, significant modification of default udev rules can introduce issues with supportability. Therefore, other means are recommended to achieve configuration of raw devices against multipathed devices with multipath device naming persistency.
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
The following procedure outlines the steps necessary to configure persistent multipath device naming and creation of raw devices (including permissions) in preparation for Oracle 10gR2 (10.2.0) Clusterware devices. From Oracle11g Release 1 (11.1.0), Clusterware files may be placed on either block or raw devices located on shared disk partitions, therefore the following procedure only strictly applies when using Oracle 10gR2 (10.2.0) and multipathing.
Therefore, take this opportunity to consider whether you wish to proceed using 10gR2 or 11gR1 Clusterware to manage your 10gR2 databases - multipath device configuration for Oracle 11g Clusterware is described in Note 605828.1. The following procedure may also be used as a basis for configuring raw devices on EL4 (Update 2 or higher). Unless otherwise stated, all steps should be performed on each cluster node and as a privileged user.
Assumptions
The following procedure assumes the following to have occured:
- Clusterware devices have been created on supported shared storage
- Clusterware devices have been appropriately sized according to Oracle10g Release 2 (10.2.0) RAC documentation
- Clusterware devices have been partitioned
- All cluster nodes have multipath access to shared devices
- Cluster nodes are configured to satisfy Oracle Universal Installer (OUI) requirements
1. Configure SCSI_ID to Return Unique Device Identifiers
1a. Whitelist SCSI devices
Before being able to configure udev to explicitly name devices, scsi_id(8) should first be configured to return SCSI device identifiers. Modify the/etc/scsi_id.config file - add or replace the option=-b parameter/value pair (if exists) with option=-g, for example:
# grep -v ^# /etc/scsi_id.configvendor="ATA",options=-p 0x80options=-g
1b. List all SCSI (Clusterware) devices
Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe <device>' or '/sbin/sfdisk -r <device>', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.
Run the fdisk(8) and/or 'cat /proc/partitions' commands to ensure Clusterware devices are visible, for example:
# cat /proc/partitionsmajor minor #blocks name 8 0 6291456 sda 8 1 5735173 sda1 8 2 554242 sda2 8 16 987966 sdb 8 17 987681 sdb1 8 32 987966 sdc 8 33 987681 sdc1 8 48 987966 sdd 8 49 987681 sdd1 8 64 987966 sde 8 65 987681 sde1 8 80 987966 sdf 8 81 987681 sdf1 8 96 987966 sdg 8 97 987681 sdg1 8 112 1004031 sdh 8 113 1003873 sdh1 8 128 1004031 sdi 8 129 1003873 sdi1 8 144 1004031 sdj 8 145 1003873 sdj1 8 160 1004031 sdk 8 161 1003873 sdk1 8 176 1004031 sdl 8 177 1003873 sdl1 8 192 1004031 sdm 8 193 1003873 sdm1
Above, though perhaps not entirely evident, the kernel has assigned two device files per multipathed device i.e. devices/dev/sdb and /dev/sdc both refer to the same device/LUN on shared storage, as do /dev/sdd and /dev/sde and so on.
Note, at this point, each cluster node may refer to the would-be Clusterware devices by different device file names - this is expected.
1c. Obtain Clusterware device unique SCSI identifiers
Run the scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. When running the scsi_id(8)command with the -s argument, the device path and name passed should be that relative to sysfs directory /sys/ i.e. /block/<device> when referring to/sys/block/<device>. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2a.), for example:
# for i in `cat /proc/partitions | awk {'print $4'} |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done...### sdb: 1494554000000000000000000010000005c3900000d000000### sdb1:### sdc: 1494554000000000000000000010000005c3900000d000000### sdc1:### sdd: 149455400000000000000000001000000843900000d000000### sdd1:### sde: 149455400000000000000000001000000843900000d000000### sde1:### sdf: 149455400000000000000000001000000ae3900000d000000### sdf1:### sdg: 149455400000000000000000001000000ae3900000d000000### sdg1:### sdh: 149455400000000000000000001000000d03900000d000000### sdh1:### sdi: 149455400000000000000000001000000d03900000d000000### sdi1:### sdj: 149455400000000000000000001000000e63900000d000000### sdj1:### sdk: 149455400000000000000000001000000e63900000d000000### sdk1:### sdl: 149455400000000000000000001000000083a00000d000000### sdl1:### sdm: 149455400000000000000000001000000083a00000d000000### sdm1:
From the output above, note that multiple devices share common SCSI identifiers. It's should now be evident that devices such as /dev/sdb and /dev/sdc refer to the same shared storage device (LUN).
Note: Irrespective of which cluster node the scsi_id(8) command is run from, the value returned for a given device (LUN) should always be the same.
2. Configure Multipath for Persistent Naming of Clusterware Devices
The purpose of this step is to provide persistent and meaningful, user-defined Clusterware multipath device names. This step is provided to ensure correct use of the intended Clusterware multipath devices that could otherwise be confused if solely relying on default multipath-assigned names (mpathn/mpathnpn), especially when many devices are involved.
2a. Configure Multipathing
Configure multipathing by modifying multipath configuration file /etc/multipath.conf. Comment and uncomment various stanzas accordingly to include (whitelist) or exclude (blacklist) specific devices/types as candidates for multipathing. Specific devices, such as our intended Clusterware devices, should be explicitly whitelisted as multipathing candidates. This can be accomplished by creating dedicated multipath stanzas for each device. Ideally, at a minimum, each device stanza should include the device wwid and an alias, for example:
# cat /etc/multipath.conf... multipath { wwid 1494554000000000000000000010000005c3900000d000000 alias voting1 }...
Following is a sample multipath.conf file. Modify your configuration according to your own environment and preferences, but ensuring to include Clusterware device-specific multipath stanzas - substitite wwid values for your own i.e. those returned from running Step 1c. above.
# grep -v ^# /etc/multipath.confdefaults { user_friendly_names yes}defaults { udev_dir /dev polling_interval 10 selector "round-robin 0" path_grouping_policy failover getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout /bin/true path_checker readsector0 rr_min_io 100 rr_weight priorities failback immediate #no_path_retry fail user_friendly_name yes}devnode_blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" devnode "^sda" devnode "^cciss!c[0-9]d[0-9]*"}multipaths { multipath { wwid 1494554000000000000000000010000005c3900000d000000 alias voting1 } multipath { wwid 149455400000000000000000001000000843900000d000000 alias voting2 } multipath { wwid 149455400000000000000000001000000ae3900000d000000 alias voting3 } multipath { wwid 149455400000000000000000001000000d03900000d000000 alias ocr1 } multipath { wwid 149455400000000000000000001000000e63900000d000000 alias ocr2 } multipath { wwid 149455400000000000000000001000000083a00000d000000 alias ocr3 }}
In the example above, devices with a specific wwid (per scsi_id(8)) are assigned persistent, user-defined names (aliases) i.e. voting1,voting2, voting3, ocr1, ocr2 and ocr3.
2b. Verify Multipath Devices
Once multipathing has been configured and multipathd service started, you should now have multipathed Clusterware devices referable by user-defined names, for example:
# multipath -ll
ocr3 (149455400000000000000000001000000083a00000d000000) dm-9 IET,VIRTUAL-DISK[size=980M][features=0][hwhandler=0]/_ round-robin 0 [prio=0][active] /_ 2:0:0:10 sdl 8:176 [active][ready]/_ round-robin 0 [prio=0][enabled] /_ 2:0:0:11 sdm 8:192 [active][ready]
ocr2 (149455400000000000000000001000000e63900000d000000) dm-3 IET,VIRTUAL-DISK[size=980M][features=0][hwhandler=0]/_ round-robin 0 [prio=0][active] /_ 2:0:0:8 sdj 8:144 [active][ready]/_ round-robin 0 [prio=0][enabled] /_ 2:0:0:9 sdk 8:160 [active][ready]
ocr1 (149455400000000000000000001000000d03900000d000000) dm-6 IET,VIRTUAL-DISK[size=980M][features=0][hwhandler=0]/_ round-robin 0 [prio=0][active] /_ 2:0:0:6 sdh 8:112 [active][ready]/_ round-robin 0 [prio=0][enabled] /_ 2:0:0:7 sdi 8:128 [active][ready]
voting3 (149455400000000000000000001000000ae3900000d000000) dm-2 IET,VIRTUAL-DISK[size=965M][features=0][hwhandler=0]/_ round-robin 0 [prio=0][active] /_ 2:0:0:4 sdf 8:80 [active][ready]/_ round-robin 0 [prio=0][enabled] /_ 2:0:0:5 sdg 8:96 [active][ready]
voting2 (149455400000000000000000001000000843900000d000000) dm-1 IET,VIRTUAL-DISK[size=965M][features=0][hwhandler=0]/_ round-robin 0 [prio=0][active] /_ 2:0:0:2 sdd 8:48 [active][ready]/_ round-robin 0 [prio=0][enabled] /_ 2:0:0:3 sde 8:64 [active][ready]
voting1 (1494554000000000000000000010000005c3900000d000000) dm-0 IET,VIRTUAL-DISK[size=965M][features=0][hwhandler=0]/_ round-robin 0 [prio=0][active] /_ 2:0:0:0 sdb 8:16 [active][ready]/_ round-robin 0 [prio=0][enabled] /_ 2:0:0:1 sdc 8:32 [active][ready]
In fact, various device names are created and used to refer to multipathed devices i.e.:
# dmsetup ls | sortocr1 (253, 6)ocr1p1 (253, 11)ocr2 (253, 3)ocr2p1 (253, 8)ocr3 (253, 9)ocr3p1 (253, 10)voting1 (253, 0)voting1p1 (253, 5)voting2 (253, 1)voting2p1 (253, 4)voting3 (253, 2)voting3p1 (253, 7)# ll /dev/disk/by-id/lrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000083a00000d000000 -> http://www.cnblogs.com/sdmlrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000083a00000d000000-part1 -> http://www.cnblogs.com/sdm1lrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-1494554000000000000000000010000005c3900000d000000 -> http://www.cnblogs.com/sdclrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-1494554000000000000000000010000005c3900000d000000-part1 -> http://www.cnblogs.com/sdc1lrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000843900000d000000 -> http://www.cnblogs.com/sdelrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000843900000d000000-part1 -> http://www.cnblogs.com/sde1lrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000ae3900000d000000 -> http://www.cnblogs.com/sdglrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000ae3900000d000000-part1 -> http://www.cnblogs.com/sdg1lrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000d03900000d000000 -> http://www.cnblogs.com/sdilrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000d03900000d000000-part1 -> http://www.cnblogs.com/sdi1lrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000e63900000d000000 -> http://www.cnblogs.com/sdklrwxrwxrwx 1 root root 9 Apr 23 11:05 scsi-149455400000000000000000001000000e63900000d000000-part1 -> http://www.cnblogs.com/sdk1# ls -l /dev/dm-*brw-rw---- 1 root root 253, 0 Apr 23 11:15 /dev/dm-0brw-rw---- 1 root root 253, 1 Apr 23 11:15 /dev/dm-1brw-rw---- 1 root root 253, 10 Apr 23 11:15 /dev/dm-10brw-rw---- 1 root root 253, 11 Apr 23 11:15 /dev/dm-11brw-rw---- 1 root root 253, 2 Apr 23 11:15 /dev/dm-2brw-rw---- 1 root root 253, 3 Apr 23 11:15 /dev/dm-3brw-rw---- 1 root root 253, 4 Apr 23 11:15 /dev/dm-4brw-rw---- 1 root root 253, 5 Apr 23 11:15 /dev/dm-5brw-rw---- 1 root root 253, 6 Apr 23 11:15 /dev/dm-6brw-rw---- 1 root root 253, 7 Apr 23 11:15 /dev/dm-7brw-rw---- 1 root root 253, 8 Apr 23 11:15 /dev/dm-8brw-rw---- 1 root root 253, 9 Apr 23 11:15 /dev/dm-9# ll /dev/mpath/lrwxrwxrwx 1 root root 7 Apr 23 11:15 ocr1 -> ../dm-6lrwxrwxrwx 1 root root 7 Apr 23 11:15 ocr1p1 -> ../dm-11lrwxrwxrwx 1 root root 7 Apr 23 11:15 ocr2 -> ../dm-3lrwxrwxrwx 1 root root 7 Apr 23 11:15 ocr2p1 -> ../dm-8lrwxrwxrwx 1 root root 7 Apr 23 11:15 ocr3 -> ../dm-9lrwxrwxrwx 1 root root 7 Apr 23 11:15 ocr3p1 -> ../dm-10lrwxrwxrwx 1 root root 7 Apr 23 11:15 voting1 -> ../dm-0lrwxrwxrwx 1 root root 7 Apr 23 11:15 voting1p1 -> ../dm-5lrwxrwxrwx 1 root root 7 Apr 23 11:15 voting2 -> ../dm-1lrwxrwxrwx 1 root root 7 Apr 23 11:15 voting2p1 -> ../dm-4lrwxrwxrwx 1 root root 7 Apr 23 11:15 voting3 -> ../dm-2lrwxrwxrwx 1 root root 7 Apr 23 11:15 voting3p1 -> ../dm-7# ll /dev/mapper/brw-rw---- 1 root disk 253, 6 Apr 23 11:15 ocr1brw-rw---- 1 root disk 253, 11 Apr 23 11:15 ocr1p1brw-rw---- 1 root disk 253, 3 Apr 23 11:15 ocr2brw-rw---- 1 root disk 253, 8 Apr 23 11:15 ocr2p1brw-rw---- 1 root disk 253, 9 Apr 23 11:15 ocr3brw-rw---- 1 root disk 253, 10 Apr 23 11:15 ocr3p1brw-rw---- 1 root disk 253, 0 Apr 23 11:15 voting1brw-rw---- 1 root disk 253, 5 Apr 23 11:15 voting1p1brw-rw---- 1 root disk 253, 1 Apr 23 11:15 voting2brw-rw---- 1 root disk 253, 4 Apr 23 11:15 voting2p1brw-rw---- 1 root disk 253, 2 Apr 23 11:15 voting3brw-rw---- 1 root disk 253, 7 Apr 23 11:15 voting3p1
The /dev/dm-N devices are used internally by device-mapper-multipath and are non-persistent across reboot, so should not be used. The /dev/mpath/devices are created for multipath devices to be visible together, however, may not be available during early stages of boot, so, again, should not be used. However, /dev/mapper/ devices are persistent and created sufficiently early during boot - use only these devices to access and interact with multipathed devices.
3. Create Raw Devices
During the installation of Oracle Clusterware 10g Release 2 (10.2.0), the Universal Installer (OUI) is unable to verify the sharedness of block devices, therefore requires the use of raw devices (whether to singlepath or multipath devices) to be specified for OCR and voting disks. As mentioned earlier, this is no longer the case from Oracle11g R1 (11.1.0) that can use multipathed block devices directly.
Manually create raw devices to bind against multipathed device partitions (/dev/mapper/*pN). Disregard device permissions for now - this will be addressed later. For example:
# raw /dev/raw/raw1 /dev/mapper/ocr1p1 /dev/raw/raw1: bound to major 253, minor 11 # raw /dev/raw/raw2 /dev/mapper/ocr2p1 /dev/raw/raw2: bound to major 253, minor 8 # raw /dev/raw/raw3 /dev/mapper/ocr3p1 /dev/raw/raw3: bound to major 253, minor 10 # raw /dev/raw/raw4 /dev/mapper/voting1p1 /dev/raw/raw4: bound to major 253, minor 5 # raw /dev/raw/raw5 /dev/mapper/voting2p1 /dev/raw/raw5: bound to major 253, minor 4 # raw /dev/raw/raw6 /dev/mapper/voting3p1 /dev/raw/raw6: bound to major 253, minor 7 # raw -qa /dev/raw/raw1: bound to major 253, minor 11 /dev/raw/raw2: bound to major 253, minor 8 /dev/raw/raw3: bound to major 253, minor 10 /dev/raw/raw4: bound to major 253, minor 5 /dev/raw/raw5: bound to major 253, minor 4 /dev/raw/raw6: bound to major 253, minor 7 # ls -l /dev/raw/ crw------- 1 root root 162, 1 Apr 23 11:52 raw1 crw------- 1 root root 162, 2 Apr 23 11:52 raw2 crw------- 1 root root 162, 3 Apr 23 11:52 raw3 crw------- 1 root root 162, 4 Apr 23 11:52 raw4 crw------- 1 root root 162, 5 Apr 23 11:52 raw5 crw------- 1 root root 162, 6 Apr 23 11:52 raw6
At this point, you should have raw devices bound to multipathed device partitions using user-defined names.
4. Test Raw Device Accessibility
Test read/write accessibility to and from raw devices from and between cluster nodes, for example:
# dd if=/dev/zero of=/dev/raw/raw1 bs=1024 count=100 100+0 records in 100+0 records out 102400 bytes (102 kB) copied, 0.762352 seconds, 134 kB/s # su - oracle $ dd if=/dev/zero of=/dev/raw/raw1 bs=1024 count=100 dd: opening `/dev/raw/raw1': Permission denied # dd if=/dev/zero of=/dev/mapper/ocr1p1 bs=1024 count=100 100+0 records in 100+0 records out 102400 bytes (102 kB) copied, 0.0468961 seconds, 2.2 MB/s ... Once testing is complete, unbind all raw devices, for example: # raw /dev/raw/raw1 0 0 /dev/raw/raw1: bound to major 0, minor 0 # raw /dev/raw/raw2 0 0 /dev/raw/raw2: bound to major 0, minor 0 # raw /dev/raw/raw3 0 0 /dev/raw/raw3: bound to major 0, minor 0 # raw /dev/raw/raw4 0 0 /dev/raw/raw4: bound to major 0, minor 0 # raw /dev/raw/raw5 0 0 /dev/raw/raw5: bound to major 0, minor 0 # raw /dev/raw/raw6 0 0 /dev/raw/raw6: bound to major 0, minor 0
5. Script Creation of Raw Bindings and Permissions
Once raw devices have been created and their accessibility and usability established, configure raw device bindings and permissions. Factoring the undeprecation of raw devices from EL5 Update 4 (initscripts-8.45.30-2), depending on your versioning, configure raw devices accordingly.
For >= EL5U4, configure raw devices via /etc/sysconfig/rawdevices in conjunction with the rawdevices service.
For < EL5U4, use a custom or existing script such as /etc/rc.local to configure raw devices, for example:
# cat /etc/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local ##### # Oracle Cluster Registry (OCR) devices ##### chown root:oinstall /dev/mapper/ocr** chmod 660 /dev/mapper/ocr* raw /dev/raw/raw1 /dev/mapper/ocr1p1 raw /dev/raw/raw2 /dev/mapper/ocr2p2 raw /dev/raw/raw3 /dev/mapper/ocr3p3 sleep 2 chown root:oinstall /dev/raw/raw1 chown root:oinstall /dev/raw/raw2 chown root:oinstall /dev/raw/raw3 chmod 660 /dev/raw/raw1 chmod 660 /dev/raw/raw2 chmod 660 /dev/raw/raw3 ##### # Oracle Cluster Voting disks ##### chown oracle:oinstall /dev/mapper/voting* chmod 660 /dev/mapper/voting* raw /dev/raw/raw4 /dev/mapper/voting1p1 raw /dev/raw/raw5 /dev/mapper/voting2p1 raw /dev/raw/raw6 /dev/mapper/voting3p1 sleep 2 chown oracle:oinstall /dev/raw/raw4 chown oracle:oinstall /dev/raw/raw5 chown oracle:oinstall /dev/raw/raw6 chmod 660 /dev/raw/raw4 chmod 660 /dev/raw/raw5 chmod 660 /dev/raw/raw6
Note: depending on the type and speed of the underlying storage, a sleep(1) of one or two seconds may be necessary between raw device creation and ownership/permission setting.
6. Test the Raw Device Script
Restart the rawdevices service and/or execute the /etc/rc.local script to test the proper creation and permission setting of both raw and multipath devices. Additionally, reboot the server(s) to further verify proper boot-time creation of devices, for example:
# /etc/rc.local /dev/raw/raw1: bound to major 253, minor 11 /dev/raw/raw2: bound to major 253, minor 8 /dev/raw/raw3: bound to major 253, minor 10 /dev/raw/raw4: bound to major 253, minor 5 /dev/raw/raw5: bound to major 253, minor 4 /dev/raw/raw6: bound to major 253, minor 7 # ll /dev/mapper/ brw-rw---- 1 root disk 253, 6 Apr 23 11:15 ocr1 brw-rw---- 1 root disk 253, 11 Apr 23 11:15 ocr1p1 brw-rw---- 1 root disk 253, 3 Apr 23 11:15 ocr2 brw-rw---- 1 root disk 253, 8 Apr 23 11:15 ocr2p1 brw-rw---- 1 root disk 253, 9 Apr 23 11:15 ocr3 brw-rw---- 1 root disk 253, 10 Apr 23 11:15 ocr3p1 brw-rw---- 1 root disk 253, 0 Apr 23 11:15 voting1 brw-rw---- 1 root disk 253, 5 Apr 23 11:15 voting1p1 brw-rw---- 1 root disk 253, 1 Apr 23 11:15 voting2 brw-rw---- 1 root disk 253, 4 Apr 23 11:15 voting2p1 brw-rw---- 1 root disk 253, 2 Apr 23 11:15 voting3 brw-rw---- 1 root disk 253, 7 Apr 23 11:15 voting3p1 # ls -l /dev/raw/ crw-rw---- 1 root oinstall 162, 1 Apr 23 11:57 raw1 crw-rw---- 1 root oinstall 162, 2 Apr 23 11:57 raw2 crw-rw---- 1 root oinstall 162, 3 Apr 23 11:57 raw3 crw-rw---- 1 oracle oinstall 162, 4 Apr 23 11:57 raw4 crw-rw---- 1 oracle oinstall 162, 5 Apr 23 11:57 raw5 crw-rw---- 1 oracle oinstall 162, 6 Apr 23 11:57 raw6
7. Install Oracle 10gR2 Clusterware
Proceed to install Oracle Clusterware 10g Release 2 (10.2.0), ensuring to specify the appropriate raw devices (/dev/raw/rawN) for OCR and voting disks. OCR devices are initialised (formatted) as part of running the root.sh script. Before running root.sh, be aware that several known issues exist that will cause the Clusterware installation to fail, namely:
- Bug.4679769 FAILED TO FORMAT OCR DISK USING CLSFMT
- Note.414163.1 10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures)
Due to Bug.4679769, initialisation of multipathed OCR devices will fail. Therefore, before running root.sh, download and apply patch for Bug.4679769. If root.sh was already run without first having applied patch for Bug.4679769, remove (null) the failed, partially initialised OCR structures from all OCR devices, for example:
# dd if=/dev/zero of=/dev/raw/raw1 bs=1M count=2525+0 records in25+0 records out# dd if=/dev/zero of=/dev/raw/raw2 bs=1M count=2525+0 records in25+0 records out
Before re-running root.sh, review Note.414163.1 to proactively address several known (vipca) issues that would otherwise need to be separately resolved later. With the above complete, the running (or re-running) of root.sh should result in proper initialisation of multipathed OCR/voting devices and successful completion of Oracle Clusterware i.e.:
[oracle@oel5a crs]$ sudo ./root.sh WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root WARNING: directory '/u01/app' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root WARNING: directory '/u01/app' is not owned by root assigning default hostname oel5a for node 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: oel5a oel5a-int oel5a Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Now formatting voting device: /dev/raw/raw4 Now formatting voting device: /dev/raw/raw5 Now formatting voting device: /dev/raw/raw6 Format of 3 voting devices complete. Startup will be queued to init within 90 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. oel5a CSS is active on all nodes. Waiting for the Oracle CRSD and EVMD to start Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps ... Upon completion of Oracle 10gR2 Clusterware installation, the Clusterware should be up and running, making use of raw devices bound to multipathed devices i.e.: [oracle@oel5a crs]$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262144 Used space (kbytes) : 1164 Available space (kbytes) : 260980 ID : 1749049955 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded Cluster registry integrity check succeeded [root@orl5a /]# crsctl query css votedisk 0. 0 /dev/raw/raw4 1. 0 /dev/raw/raw5 2. 0 /dev/raw/raw6 located 3 votedisk(s). [root@oel5a /]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy
Refer to Note.394848.1 for any issues, such as -16 EBUSY [Device or resource busy], arising from the continued use of raw devices being bound to multipathed devices.
The requirement to use raw devices for OCR and voting devices solely applies to the initial installation of Oracle 10gR2 Clusterware. Once the installation is complete, OCR and voting devices can be switched to use multipath devices directly - refer to Note.401132.1 for further details.
------------------------------------------------------------------------------
Blog： http://blog.csdn.net/tianlesoftware
网上资源： http://tianlesoftware.download.csdn.net
相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx
DBA1 群：62697716(满); DBA2 群：62697977(满)
DBA3 群：62697850 DBA 超级群：63306533;
聊天 群：40132017
--加群需要在备注说明Oracle表空间和数据文件的关系，否则拒绝申请