Comments on: Oracle Clusterware on RHEL5/OEL5 with udev and multipath http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/ Jeremy Schneider Wed, 30 Nov 2011 21:26:13 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: John http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-2067 John Wed, 25 May 2011 22:37:57 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-2067 I never had to rebuild initrd with any systems after RHEL 5.3 without the patch referenced above. The patch is integrated into later versions of the OS (and more specifically, device-mapper-multipath-0.4.7-34.el5 and later) If you're running > RHEL 5.3 (or >= device-mapper-multipath-0.4.7-34.el5), I don't think you should need to rebuild initrd. I tested this again today with this system (which is due for an update too, but it illustrates the point): [root@serv ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.5 (Tikanga) [root@serv ~]# rpm -qa | grep multipath device-mapper-multipath-0.4.7-34.el5_5.6 [root@serv ~]# ls -al /dev/mapper total 0 drwxr-xr-x 2 root root 100 May 25 17:06 . drwxr-xr-x 16 root root 4540 May 25 17:21 .. crw------- 1 root root 10, 63 Mar 10 12:30 control brw-r----- 1 oracle dba 253, 0 May 25 17:06 dguard brw-r----- 1 oracle dba 253, 1 May 25 17:06 dguardp1 This is with the following change being made to /etc/multipath.conf: multipath { wwid 3603208b7200a3e264000a30001bc0000 alias dguard path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 18 rr_min_io 100 mode 640 uid 54321 gid 54322 } (oracle is UID 54321, dba is gid 54322 thanks to oracle-validated) HTH, and I think it's great that the discussions have continued. I still have this thread bookmarked and refer back to it every now and again. Thanks! John I never had to rebuild initrd with any systems after RHEL 5.3 without the patch referenced above.
The patch is integrated into later versions of the OS (and more specifically, device-mapper-multipath-0.4.7-34.el5 and later)

If you’re running > RHEL 5.3 (or >= device-mapper-multipath-0.4.7-34.el5), I don’t think you should need to rebuild initrd.

I tested this again today with this system (which is due for an update too, but it illustrates the point):
[root@serv ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
[root@serv ~]# rpm -qa | grep multipath
device-mapper-multipath-0.4.7-34.el5_5.6
[root@serv ~]# ls -al /dev/mapper
total 0
drwxr-xr-x 2 root root 100 May 25 17:06 .
drwxr-xr-x 16 root root 4540 May 25 17:21 ..
crw——- 1 root root 10, 63 Mar 10 12:30 control
brw-r—– 1 oracle dba 253, 0 May 25 17:06 dguard
brw-r—– 1 oracle dba 253, 1 May 25 17:06 dguardp1

This is with the following change being made to /etc/multipath.conf:
multipath {
wwid 3603208b7200a3e264000a30001bc0000
alias dguard
path_grouping_policy group_by_prio
path_selector “round-robin 0″
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
mode 640
uid 54321
gid 54322
}
(oracle is UID 54321, dba is gid 54322 thanks to oracle-validated)

HTH, and I think it’s great that the discussions have continued. I still have this thread bookmarked and refer back to it every now and again.

Thanks!
John

]]> By: Alex Hopcroft http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-2063 Alex Hopcroft Tue, 17 May 2011 16:32:49 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-2063 I know it's a long time after SST's post but I've used his post today and had the same issue as John. The solution if your going to use /etc/multpath.conf to set the permissions as per SST's post is to recreate your initrd after updating the mutlipath.conf. It worked fine for me and all Oracle devices now have the correct permissions every time. E.G [root@~]# vi /etc/multipath.conf [root@~]# cd /boot [root@boot]# uname -r 2.6.18-238.9.1.el5 [root@boot]# mkinitrd -f initrd-2.6.18-238.9.1.el5.img 2.6.18-238.9.1.el5 Modulefile is /etc/modprobe.conf [root@boot]# reboot I know it’s a long time after SST’s post but I’ve used his post today and had the same issue as John. The solution if your going to use /etc/multpath.conf to set the permissions as per SST’s post is to recreate your initrd after updating the mutlipath.conf. It worked fine for me and all Oracle devices now have the correct permissions every time.

E.G
[root@~]# vi /etc/multipath.conf
[root@~]# cd /boot
[root@boot]# uname -r
2.6.18-238.9.1.el5
[root@boot]# mkinitrd -f initrd-2.6.18-238.9.1.el5.img 2.6.18-238.9.1.el5
Modulefile is /etc/modprobe.conf
[root@boot]# reboot

]]> By: MP Derelle http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1790 MP Derelle Thu, 18 Nov 2010 17:04:06 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1790 With Asmlib, you can directly take advantage of device-mapper-multipath just configure /etc/sysconfig/oracleasm (RedHat 4/5) : " # ORACLEASM_SCANORDER: Matching patterns to order disk scanning ORACLEASM_SCANORDER="dm-" # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan ORACLEASM_SCANEXCLUDE="sd" " /etc/init.d/oracleasm will then scan /dev/dm- device instead of /dev/sd, forcing use of multipath. This will map persistently by ASM label the good major/minor device in /dev/oracleasm/disks You just need to configure wwid persistence for ocr/voting disk and configure rawdevices like this : (we have created small partitions on the same volume) "# This file and interface are deprecated. # Applications needing raw device access should open regular # block devices with O_DIRECT. # raw device bindings # format: # # example: /dev/raw/raw1 /dev/sdb1 # /dev/raw/raw2 8 5 /dev/raw/raw1 /dev/mapper/clup1 /dev/raw/raw2 /dev/mapper/clup2 /dev/raw/raw5 /dev/mapper/clup5 /dev/raw/raw6 /dev/mapper/clup6 /dev/raw/raw7 /dev/mapper/clup7 " With Asmlib, you can directly take advantage of device-mapper-multipath

just configure /etc/sysconfig/oracleasm (RedHat 4/5) :

”
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=”dm-”

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=”sd”
”

/etc/init.d/oracleasm will then scan /dev/dm- device instead of /dev/sd, forcing use of multipath. This will map persistently by ASM label the good major/minor device in /dev/oracleasm/disks

You just need to configure wwid persistence for ocr/voting disk and configure rawdevices like this :

(we have created small partitions on the same volume)

“# This file and interface are deprecated.
# Applications needing raw device access should open regular
# block devices with O_DIRECT.
# raw device bindings
# format:
#
# example: /dev/raw/raw1 /dev/sdb1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/mapper/clup1
/dev/raw/raw2 /dev/mapper/clup2
/dev/raw/raw5 /dev/mapper/clup5
/dev/raw/raw6 /dev/mapper/clup6
/dev/raw/raw7 /dev/mapper/clup7
“

]]> By: Jeremy Schneider http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1765 Jeremy Schneider Mon, 04 Oct 2010 16:59:04 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1765 Hello all - thanks for all the discussion. Sorry I haven't been so responsive on this particular post! There are lots of great comments here. First, @Ez's recent comment about ASMlib being the "right" way: this article is almost 3 years old, and it's just an update of an even older article. Before ASMlib was released with Oracle 10.1 this was how we used to do it. ASMlib is not the "right" way but just one choice. I've discussed this in another old post about <a href="http://www.ardentperf.com/2008/10/08/asmlib-performance-vs-udev/" rel="nofollow">ASMLIB Performance vs Udev</a>. I will say that today, I do think ASMlib is the right choice in most situations. @Dave and @Scott5[55] and @SST and @Robert - thanks for answering other people's questions in the thread while I wasn't around. :) Lots of discussion here about using multipath.conf - I was aware of that when I wrote the post several years ago, and I've used it before. It works fine too, though in this particular case we were using UDEV for a lot more than just multipath devices. They were just one type of device. It was awhile ago though... can't remember all the reasons I didn't use multipath.conf... it works too. As several people have mentioned in the comments, generally you shouldn't need to use both multipath.conf and UDEV - just one or the other should do the job. Regarding /etc/rc.local - the problem with this approach is that it only takes effect at boot-time. With larger production systems it's not uncommon at all to attach and detach storage at run-time without rebooting (if possible). UDEV handles this very nicely. (I think that multipath.conf does too.) And of course, ASMlib is probably my preferred solution if there aren't linux guys around who can handle multipath.conf or UDEV. @Emmanuel - Off the top of my head, I'm not sure why ohasd was throwing an error. I hope you've been able to fix it by now. Thanks again everybody. :) -Jeremy Hello all – thanks for all the discussion. Sorry I haven’t been so responsive on this particular post! There are lots of great comments here.

First, @Ez’s recent comment about ASMlib being the “right” way: this article is almost 3 years old, and it’s just an update of an even older article. Before ASMlib was released with Oracle 10.1 this was how we used to do it. ASMlib is not the “right” way but just one choice. I’ve discussed this in another old post about ASMLIB Performance vs Udev. I will say that today, I do think ASMlib is the right choice in most situations.

@Dave and @Scott5[55] and @SST and @Robert – thanks for answering other people’s questions in the thread while I wasn’t around. :)

Lots of discussion here about using multipath.conf – I was aware of that when I wrote the post several years ago, and I’ve used it before. It works fine too, though in this particular case we were using UDEV for a lot more than just multipath devices. They were just one type of device. It was awhile ago though… can’t remember all the reasons I didn’t use multipath.conf… it works too. As several people have mentioned in the comments, generally you shouldn’t need to use both multipath.conf and UDEV – just one or the other should do the job.

Regarding /etc/rc.local – the problem with this approach is that it only takes effect at boot-time. With larger production systems it’s not uncommon at all to attach and detach storage at run-time without rebooting (if possible). UDEV handles this very nicely. (I think that multipath.conf does too.) And of course, ASMlib is probably my preferred solution if there aren’t linux guys around who can handle multipath.conf or UDEV.

@Emmanuel – Off the top of my head, I’m not sure why ohasd was throwing an error. I hope you’ve been able to fix it by now.

Thanks again everybody. :)

-Jeremy

]]> By: Ez Aton http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1764 Ez Aton Mon, 04 Oct 2010 12:40:13 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1764 You could use ASMLib, as you are using ASM. This is the "right" way of doing it. This simplifies things, as ASM sets its own (required) permissions according to your predefinitions. Simple and elegant. You don't need to define RAW devices, and ASMLib makes use of labels, so life is good even if you have many devices, or if you experience LUN (sd) device renames. Also, God forbid, if both (some of) your servers see different LUN names. You needn't fight with the system to match the mpath names, permissions, binding, etc. Just label them with ASMLib, and you're happy. Ez You could use ASMLib, as you are using ASM. This is the “right” way of doing it. This simplifies things, as ASM sets its own (required) permissions according to your predefinitions. Simple and elegant.
You don’t need to define RAW devices, and ASMLib makes use of labels, so life is good even if you have many devices, or if you experience LUN (sd) device renames. Also, God forbid, if both (some of) your servers see different LUN names. You needn’t fight with the system to match the mpath names, permissions, binding, etc. Just label them with ASMLib, and you’re happy.

Ez

]]> By: Emmanuel http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1596 Emmanuel Tue, 01 Jun 2010 05:32:41 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1596 Hi Jeremy, I realize that it is what I did with multipath.conf : multipaths { multipath { wwid 3600a0b80004891dc00000ffd4bea8199 alias ocr_vote_01 } multipath { wwid 3600a0b8000423486000016404bed1ace alias dg_data_01 } It works because I get my aliases under /dev/mapper/ . I am also able to mount the partition and format them with mkfs.ext3. I can create the OCR disk with oracleasm create disk successfully. It crashes when ohasd tries to format the partition to create the OCR. I do not think that it would change anything with Udev. What do you think ? Regards, Emmanuel Hi Jeremy,

I realize that it is what I did with multipath.conf :

multipaths {
multipath {
wwid 3600a0b80004891dc00000ffd4bea8199
alias ocr_vote_01
}
multipath {
wwid 3600a0b8000423486000016404bed1ace
alias dg_data_01
}

It works because I get my aliases under /dev/mapper/ . I am also able to mount the partition and format them with mkfs.ext3. I can create the OCR disk with oracleasm create disk successfully. It crashes when ohasd tries to format the partition to create the OCR.

I do not think that it would change anything with Udev.

What do you think ?

Regards,

Emmanuel

]]> By: Emmanuel http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1595 Emmanuel Tue, 01 Jun 2010 05:10:10 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1595 hi Scott555, Thank for your answer. I will try and let you know. Regards, Emmanuel hi Scott555,

Thank for your answer. I will try and let you know.

Regards,

Emmanuel

]]> By: Scott555 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1594 Scott555 Fri, 28 May 2010 14:11:16 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1594 @Emmanuel - I recently went back and forth with one of our DBAs over this. (I'm an SA, and we handle the systems and storage, but the DBAs handle the ASM config and Oracle/RAC installation.) The understanding we came to was that the Oracle installation was essentially broken in that it wouldn't use the multipathed (mapper) devices during the installation, and a set of standard SCSI devices had to be presented and set up within UDEV for the installation to auto-configure as ASM storage. I don't know if that's a fully accurate interpretation of the new 11GR2 install requirements, but it worked for us. I like using the wwid of the LUNs when identifying storage devices, and that's how I do it in the multipath.conf. For straight-up SCSI devices, though; I created this udev rules file to provide a couple of disks to the Oracle install: Part of the criteria included isolating devices to individual HBAs, as I wouldn't want to present both LUNs down the same path # cat /etc/udev/rules.d/45-oracle-asmdevices.rules # asm1 device; only use the SCSI devices available to 0000:11:00.1 KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p" RESULT=="3600508b4000c39390001100000600000", BUS=="pci", ID=="0000:11:00.1", NAME:="asm1", OWNER:="oracle", GROUP:="oinstall", MODE:="0660" # asm2 device; only use the SCSI devices available to 0000:0d:00.0 KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p" RESULT=="3600508b4000c39390001100000630000", BUS=="pci", ID=="0000:0d:00.0", NAME:="asm2", OWNER:="oracle", GROUP:="oinstall", MODE:="0660" Keep in mind that we did *NOT* partition these devices or create file systems on them; they're essentially "raw" SCSI block devices. This gives me two SCSI devices; /dev/asm1, and /dev/asm2, and the installation seems to be happy with this. @Emmanuel –

I recently went back and forth with one of our DBAs over this. (I’m an SA, and we handle the systems and storage, but the DBAs handle the ASM config and Oracle/RAC installation.) The understanding we came to was that the Oracle installation was essentially broken in that it wouldn’t use the multipathed (mapper) devices during the installation, and a set of standard SCSI devices had to be presented and set up within UDEV for the installation to auto-configure as ASM storage. I don’t know if that’s a fully accurate interpretation of the new 11GR2 install requirements, but it worked for us.

I like using the wwid of the LUNs when identifying storage devices, and that’s how I do it in the multipath.conf. For straight-up SCSI devices, though; I created this udev rules file to provide a couple of disks to the Oracle install:

Part of the criteria included isolating devices to individual HBAs, as I wouldn’t want to present both LUNs down the same path

# cat /etc/udev/rules.d/45-oracle-asmdevices.rules
# asm1 device; only use the SCSI devices available to 0000:11:00.1
KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”
RESULT==”3600508b4000c39390001100000600000″, BUS==”pci”, ID==”0000:11:00.1″, NAME:=”asm1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”0660″
# asm2 device; only use the SCSI devices available to 0000:0d:00.0
KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”
RESULT==”3600508b4000c39390001100000630000″, BUS==”pci”, ID==”0000:0d:00.0″, NAME:=”asm2″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”0660″

Keep in mind that we did *NOT* partition these devices or create file systems on them; they’re essentially “raw” SCSI block devices.

This gives me two SCSI devices; /dev/asm1, and /dev/asm2, and the installation seems to be happy with this.

]]> By: Emmanuel http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1593 Emmanuel Fri, 28 May 2010 12:23:31 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1593 Hi Jeremy, here is a trace I found in the ohasd.log, may be It can help : 2010-05-28 10:49:24.482: [ default][3872035408] OHASD Daemon Starting. Command string :reboot 2010-05-28 10:49:24.489: [ default][3872035408] Initializing OLR 2010-05-28 10:49:24.494: [ OCROSD][3872035408]utgdv:2:ocr loc file /etc/oracle/olr.loc cannot be opened. errno 2 2010-05-28 10:49:24.494: [ OCRRAW][3872035408]proprioo: cannot get current configuration (33) 2010-05-28 10:49:24.495: [ OCRRAW][3872035408]proprinit: Could not open raw device 2010-05-28 10:49:24.495: [ OCRAPI][3872035408]a_init:16!: Backend init unsuccessful : [33] 2010-05-28 10:49:24.496: [ CRSOCR][3872035408] OCR context init failure. Error: PROCL-33: Oracle Local Registry is not configured Storage layer error [Error opening olr.loc file. No such file or directory] [2] 2010-05-28 10:49:24.496: [ default][3872035408] OLR initalization failured, rc=33 2010-05-28 10:49:24.499: [ default][3872035408]Created alert : (:OHAS00106:) : Failed to initialize Oracle Local Registry 2010-05-28 10:49:24.499: [ default][3872035408][PANIC] OHASD exiting; Could not init OLR 2010-05-28 10:49:24.499: [ default][3872035408] Done. It is really weird, because I am successful in create ext3 on my /dev/mapper/ocr_vote_01p1 partition, which is the one I use to create the ASM disk. By the way, I do not know why but the OUI does not see my ASM volume created at step 8, so I have to change the discovery path to /dev/mapper and select myself /dev/mapper/ocr_vote_01p1 . Regards, Emmanuel Hi Jeremy,

here is a trace I found in the ohasd.log, may be It can help :

2010-05-28 10:49:24.482: [ default][3872035408] OHASD Daemon Starting. Command string :reboot
2010-05-28 10:49:24.489: [ default][3872035408] Initializing OLR
2010-05-28 10:49:24.494: [ OCROSD][3872035408]utgdv:2:ocr loc file /etc/oracle/olr.loc cannot be opened. errno 2
2010-05-28 10:49:24.494: [ OCRRAW][3872035408]proprioo: cannot get current configuration (33)
2010-05-28 10:49:24.495: [ OCRRAW][3872035408]proprinit: Could not open raw device
2010-05-28 10:49:24.495: [ OCRAPI][3872035408]a_init:16!: Backend init unsuccessful : [33]
2010-05-28 10:49:24.496: [ CRSOCR][3872035408] OCR context init failure. Error: PROCL-33: Oracle Local Registry is not configured Storage layer error [Error opening olr.loc file. No such file or directory] [2]
2010-05-28 10:49:24.496: [ default][3872035408] OLR initalization failured, rc=33
2010-05-28 10:49:24.499: [ default][3872035408]Created alert : (:OHAS00106:) : Failed to initialize Oracle Local Registry
2010-05-28 10:49:24.499: [ default][3872035408][PANIC] OHASD exiting; Could not init OLR
2010-05-28 10:49:24.499: [ default][3872035408] Done.

It is really weird, because I am successful in create ext3 on my /dev/mapper/ocr_vote_01p1 partition, which is the one I use to create the ASM disk.

By the way, I do not know why but the OUI does not see my ASM volume created at step 8, so I have to change the discovery path to /dev/mapper and select myself /dev/mapper/ocr_vote_01p1 .

Regards,

Emmanuel

]]> By: Emmanuel http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1592 Emmanuel Fri, 28 May 2010 11:33:50 +0000 http://www.ardentperf.com/2008/02/13/oracle-clusterware-on-rhel5oel5-with-udev-and-multipath/#comment-1592 Hi, I am trying to install the 11GR2 Clusterware on RHEL5 servers using multipath. I am able to create the ASM disk with oracleasm without any trouble, based on /dev/mapper/ocr_vote_01 which is the alias I specified in my multipath.conf. I am also able to create an ext3 partition on that device, and mount it. The thing is that I am not successful at the end of the clusterware installation : CRS-4124: Oracle High Availability Services startup failed. CRS-4000: Command Start failed, or completed with errors. ohasd failed to start: Inappropriate ioctl for device ohasd failed to start at /u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443. I tried and checked a lot of things. Remember it is 11GR2, so /dev/mapper/ link has to be owned by grid:asmdba . I also tried to disable IPV6, with no result. Any idea ? Regards, Emmanuel Hi,

I am trying to install the 11GR2 Clusterware on RHEL5 servers using multipath.

I am able to create the ASM disk with oracleasm without any trouble, based on /dev/mapper/ocr_vote_01 which is the alias I specified in my multipath.conf.

I am also able to create an ext3 partition on that device, and mount it.

The thing is that I am not successful at the end of the clusterware installation :

CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

I tried and checked a lot of things. Remember it is 11GR2, so /dev/mapper/ link has to be owned by grid:asmdba . I also tried to disable IPV6, with no result.

Any idea ?

Regards,

Emmanuel

]]>