The trouble with Linux? No… the trouble with computers in general - is that they keep changing! Solaris 10 comes out, Oracle 11g, Red Hat 5… and everything works different!! It’s a full-time job just trying to keep up with everything.

Almost exactly one year ago I wrote about using udev on 2.6 kernels to set the proper permissions for Oracle RAC. Two weeks after that post (March 14) Red Hat Enterprise Linux 5 was released and changed everything.

In my original post, I demonstrated how to create a PERMISSIONS file that udev would use when creating the device nodes. This worked on RHEL4 and SLES9. However this week I’ve been helping a client deploy 11g RAC on a RHEL5-based cluster - and I remembered that the PERMISSIONS facility was removed from udev in RH5. Seems like I remember reading something about having a single source of configuration for udev, which makes sense… so maybe they picked the RULES. (You’ll remember from my previous post that RULES are processes right before PERMISSIONS.) This is just as well since RULES are actually quite a bit more powerful than PERMISSIONS.

So on RHEL5 and OEL5 - in order to conform to Linux Best Practices - we now have to set correct RAC file permissions using udev RULES. To get started, we need to review how RULES work. The udev manual page gives a good overview of rules processing. But of course there are plenty of great tutorials that go deeper if you’re looking for more.

Block Devices and Raw Devices

Now of course I’m not the first person to notice that there’s no permissions.d directory on RHEL5 and OEL5. Last September, Grégory Guillou from Pythian blogged about installing on Red Hat 5 and referenced a post on his French blog that shows how to setup a RULES file for SCSI block devices. I don’t speak French but I was able to copy the text of his RULES files:

# Oracle Configuration Registry
KERNEL=="sdb[8-9]", OWNER="root", GROUP="oinstall", MODE="640"
# Voting Disks
KERNEL=="sdb1[0-2]", OWNER="oracle", GROUP="oinstall", MODE="640"
# ASM Devices
KERNEL=="sdb[5-7]", OWNER="oracle", GROUP="dba", MODE="660"

Many people are still configuring raw devices for their voting disks and cluster registries even though this is not necessary. This Oracle Forums thread gives an sample of using a RULES file to set permissions for raw devices. It says to create a file called /etc/udev/rules.d/65-raw-permissions.rules with these contents:

# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"

In fact there’s a great Oracle Whitepaper on the udev and multipathing that was published all the way back in June of 2007. It gives another sample configuration for raw devices on Red Hat 5.

Linux Multipath (Device Mapper) Devices

That’s all great. However… I’m doing an implementation on RHEL5 right now and we’re using the device mapper to multipath connections to the SAN. And the Oracle white paper - which goes into marvelous depth about multipath, udev and RH5 - never tells us anything about configuring multipath, udev and RH5 all together! So I have to figure this one out on my own. No harm done; it provided an interesting challenge for the day. :)

Multipath devices are a bit tricky. Usually if you’re using multipath then you don’t want to assume that devices will always be discovered in the same order. (This is the purpose of assigning friendly names by WWN in the multipath.conf file with the alias directive.) In fact, on our cluster the devices were assigned in different orders on different nodes - the ocr was dm-9 on one node and dm-10 on another. How do you write a RULES entry if you don’t know what the name of the device is?

I finally did get the multipath rules file to gave the proper permissions to my OCR and Voting Disks based on alias. However I was only able to change the /dev/dm-* files and not the /dev/mapper/* nodes. (Those nodes are not created by udev.) Therefor I had to use the aliases in /dev/mpath - not the aliases in /dev/mapper - when running the Oracle installer. How does it work? The symlinks in /dev/mpath are created by udev RULES - so all I had to do was piggyback on the udev config that created them and modify the permissions of the underlying devices.

This client used the aliases vote1, vote2, vote3, ocr1 and ocr2. Here’s the config file /etc/udev/rules.d/40-multipath.rules with my changes in bold:

# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", ACTION=="add", PROGRAM=="/bin/bash -c '/sbin/lsmod | /bin/grep \
    ^dm_multipath'", RUN+="/sbin/multipath -v0 %M:%m"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
ACTION=="add", RUN+="/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p p' \
    -j %M -m %m"
PROGRAM=="/sbin/dmsetup ls --target multipath --exec /bin/basename -j %M -m %m", \
    RESULT=="?*", NAME="%k", SYMLINK="mpath/%c", GOTO="check_cluster_devs"
PROGRAM!="/bin/bash -c '/sbin/dmsetup info -c --noheadings -j %M -m %m | /bin/grep \
    -q .*:.*:.*:.*:.*:.*:.*:part[0-9]*-mpath-'", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup ls --target linear --exec /bin/basename -j %M -m %m", NAME="%k", \
    RESULT=="?*", SYMLINK="mpath/%c", OPTIONS="last_rule"
GOTO="end_mpath"
LABEL="check_cluster_devs"
RESULT=="ocr*", GROUP="dba", MODE="640"
RESULT=="vote*", OWNER="oracle", GROUP="dba", MODE="640"
OPTIONS="last_rule"
LABEL="end_mpath"

Comments