Comments on: Oracle Clusterware on RHEL5/OEL5 with udev and multipath

By: Robert

Robert — Tue, 19 Jan 2010 16:39:03 +0000

With the installation of the following package updates, I was able to get UDEV to work with device-mapper-multipath without modifying any standard UDEV rules. This is for Red Hat 5.3 x86-64 baseline. device-mapper-multipath.x86_64 0.4.7-30.el5_4.4 kpartx.x86_64 0.4.7-30.el5_4.4 The key for me was to create a rule file that would run before the 40-multipath.rules as the OPTIONS="last_rule" in the 40-multipath.rules file would ignore all later rules. My rule file is /etc/udev/rules.d/39-crs.rules and contains the following. It creates a symbolic link to the dm-X device based upon the UUID. It then modifies the ownership and permission on the dm-X device.


KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m", \
RESULT=="part8-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="crs_vote_1", \
OWNER="oracle", GROUP="oinstall", MODE="0660"

KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m", \
RESULT=="part9-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="crs_vote_2", \
OWNER="oracle", GROUP="oinstall", MODE="0660"

KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m", \
RESULT=="part10-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="crs_vote_3", \
OWNER="oracle", GROUP="oinstall", MODE="0660"

KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m", \
RESULT=="part11-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="ocr_file_1", \
OWNER="root", GROUP="oinstall", MODE="0640"

KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o UUID -j %M -m %m", \
RESULT=="part12-mpath-360060e8005aff9000000aff9000012d9", NAME="%k", SYMLINK="ocr_file_2", \
OWNER="root", GROUP="oinstall", MODE="0640"

By: Scott5

Scott5 — Mon, 30 Nov 2009 16:19:06 +0000

@Dave –

Thanks for sharing. You bring up a good point RE: PowerPath. No intention of using it ATM, but will consider the additional R&D we’ll need to go through if it becomes part of the stack.

Could udev rules supplant printing as the most painful SA responsibility? I was certain it would be PAM configuration. :P

By: Dave

Dave — Mon, 30 Nov 2009 15:13:32 +0000

You *can* use one or the other, but since you mentioned EMC arrays, look at it this way: If you are using pow”r”path instead of mpath, you almost have to use udev rules of some kind. If you are using mpath the *implied* recommended way to ensure device naming consistency accross reboots is with multipath.conf tying the wwid to some sort of alias, and that subsequent alias which is created under /dev/mapper must be used by the volume manager whatever it is. It’s all there in the man pages and in the RHN articles relating to multipathing. It’s not really that confusing but I agree, there is little consistency out there for concrete configuration info in RAC environments.
If your storage vendor does not support native multipathing then you have to use what you got, and setting the bits can be done with the rules or by some other sequence of events at boot. With more vendors supporting native multipathing, seems like a no brainer to us and a pretty solid solution too. Just no big red easy button to push.

By: Scott5

Scott5 — Wed, 25 Nov 2009 21:16:05 +0000

Heh – a blog post from almost two years ago becomes the best online source for udev/multipath support.

Seriously, I’ve had a number of RAC clusters set up for 6 months or so now; 11gR1 on OEL 5.3, various types of SAN disk; Symmetrix, Clariion, EVA, and I have stayed away from udev modifications like they smell. (don’t they?) I don’t understand what or why to modify the udev rules. With aliases mapped to wwids in the multipath.conf file I *think* I’m covered, but no matter how much I research this udev stuff, I just do not understand it at all. I get the concept, but knowing whether I’m at risk for devices changing arbitrarily at boot is another story.

Like what John mentioned on the 20th, it *seems* like you can use one or the other. How to tell if devices are at risk?

By: John

John — Mon, 23 Nov 2009 17:55:23 +0000

OK, got everything running correctly now, and persistent across reboots. I was having my ASM disks get the correct permissions, but the OCR and Voting disks were not. I re-applied the patch from RH, apparently something didn’t go correctly the first time, and now it’s working. Thanks a ton for all the help here, great resource, and specifically to Dave.

As an aside, the srvctl problem ended up being a line added for a RHEL 3 bug to set LD_ASSUME_KERNEL=2.4.xxx, along with a trailing “/” in an ORACLE_HOME= line that made the path read as /u01/app/oracle/product/10//lib (extra / before lib). Once I found and removed those errors, srvctl found the correct libraries and ran great.

PEBKAC error once again. :-)

By: Dave

Dave — Fri, 20 Nov 2009 18:56:26 +0000

What I was told was that you don’t need udev “if you don’t need it” and if multipath.conf does all you need, it is way simpler to administer IMO. udev is pretty ugly to us and we used that in our RHEL4 10 g cluster and it sucks. This cluster is RHEL5.3 and 11g. I think if you are using yum and updated to the latest greatest, it should work for you. Remember to to do a multipath -f and then -r to apply your changes. Oh yeah, and the formatting of that file is touchy, and if the formatting is “off” you won’t get aliases and wierdness will surround you. Good luck.

By: John

John — Fri, 20 Nov 2009 18:48:45 +0000

Thanks a lot for getting back to me. I thought that was what you were saying, so I went ahead and applied the patch. So far, it hasn’t worked. I still have had to manually chown and chmod the /dev/mapper files in order for certain parts of the RAC install process to continue or pass.

If they’re defined in /etc/multipath.conf, do I still need to have udev rules? I haven’t set any up because I thought that it was one place or the other, not both.

I’ve also run into problems after the install with srvctl complaining that it can’t load libpthread.so.0. I’m hoping that’s fixed in the 10.2.0.4 patch (10.2.0.1 might have problems using the i686 version of glibc-2.5 rather than the i386 version it replaced?), found the error when I tried to shut down ASM using srvctl to apply the patch.

We’re still way ahead of our first RAC on Linux adventure, which threw IA64 into the mix just for fun, about 3.5 years ago. I wasn’t around for that one, but I hear it took nearly 3 months of calls between our DBAs, Oracle, Redhat, and HP before they could even get the installer to complete. Not certain who to blame for that one. :-)

Thanks again for the great article and comments!

By: Dave

Dave — Fri, 20 Nov 2009 18:35:55 +0000

Yeah, when we patched we now can dynamically bring in new devices, set the uid, mode, gid from within multipath.conf, and it works slick. The DBAs then scan for new disks in /dev/mapper and no problems at all (at least for adding them). Now, the only problem we have seen is removing LUNs on the fly from ASM is easy. But when trying to remove and cleanup the OS, it gets pi$$y and usually requires a reboot. But who shrinks storage? We do, and we discovered this the hard way a year or so back in our production RAC. Sometimes you over allocate and want to remove a couple LUNs and give to another RAC or something. Adding them is a piece of cake. Removing them is a piece of carp.

By: John

John — Thu, 19 Nov 2009 17:19:08 +0000

@Dave:
Are you saying you applied the patch in the linked RH Bugzilla bug, and that cleared up the issues?

We’re trying to get RAC set up today, and I’m hoping that’s the case.

By: Dave

Dave — Tue, 29 Sep 2009 13:30:03 +0000

Fred – the problem was not the multipath settings, the problem was the DBA not using pattern match or looking to the right directory for the devices. We applied the latest errata for multipath and it fixed the problems for the most part of not being able to apply mode, uid, gid to new devices. The point is not that it is “required” to set the right mode on /dev/mapper devices, the point is that others have tried and seen intermittent results. The documentation for multipath.conf explicitly says you should be able to set mode/gid/uid among other settings on the fly and using this configuration file. It works for us on RHEL5.3 doing “multipath -f” and “multipath -r” after modifying the file. You can easily set these persistenantly rather than using udev rules, especially if alot of the devices need a custom alias, etc.