That 11i/RAC/ASM project turned out to be fairly interesting for quite a few reasons – another reason was that it gave me an opportunity to become quite a bit more familiar with RConfig.

RConfig an Oracle provided java-based utility to fully automate the process of converting a database from single-instance to RAC. It will also automatically move your database from a filesystem into ASM. (In case you’re wondering, it just uses RMAN in the background; really RMAN is still the only tool that can actually move a database into ASM.) RConfig is the backend used by Grid Control for its “Convert to Cluster Database” wizard.

After digging into RConfig a bit I have two main observations:
› Read more

I recently mentioned working with a client to convert their E-Business Suite environment to RAC and ASM. I enjoyed this project; there are a few good high-level guides to the process but there are still a lot of small hiccups and configuration decisions where I could put my RAC experience to work.

The main guide that we used for this conversion process was metalink note 362135.1. However there were a few divergences; the two main ones were that (1) the client had already upgraded their source database to 10g and (2) the metalink note was updated with a few new steps in the middle of our project but we chose to stick with the original document that we’d started with. The flowchart to the left gives a pretty good summary of our conversion process (click on it to see a larger image).

RConfig simplified the actual conversion to RAC and ASM but there were quite a few additional steps that we had to perform manually before and after RConfig. Here’s a breakdown of the database-related steps:
› Read more

Was just perusing Sergio Leunissen’s blog this morning and a couple of his recent posts caught my attention:

First off, as someone who frequently installs Oracle on Linux, his post last month about the recently released RPM oracle-validated was great – can’t believe I’d missed that. Basically it’s an RPM that makes sure you have all the required OS packages for the Oracle RDBMS and even sets kernel parameters and creates an oracle user and dba/oinstall groups. Sergio has a nice demo in his post; I’m going to have to start using this!

Secondly, Sergio linked to a great post from Wim back at the end of February about the differences between RedHat and OEL. The most important point: OEL is not a fork. In fact I didn’t realize this was available but there’s even a short PDF which lists every single package that’s different – and what the differences are. Have a look; it’s pretty much just logo and branding changes. In short, OEL is like Centos or White Box with real Oracle Corp support.

If you’re comparing OEL and RedHat there are really only two things to compare: (1) availability of OS for “proof of concept” or development systems – OEL lets you easily DL and run as many copies as you want for free like Centos and White Box while RedHat doesn’t and (2) support organizations – do you think that Red Hat or Oracle will do a better job of supporting your Linux operating system when you do decide to go into production and purchase support for it (and honestly that seems to be the bigger question in my mind).

Another thing to keep in mind is that Oracle will also support RedHat installations; you can even update your RedHat system to point to Oracle’s servers for new updates rather than RedHat’s servers. (Then for example you can automatically pick up packages like oracle-validated and ocfs.)

Just three quick updates…

First of all, yesterday we re-ran rconfig a few times and confirmed – for sure – that the PRKP-1001 and CRS-0215 errors tuesday were from the listener/ONS problem.

Secondly, just an observation – I noticed yesterday that somehow a few of the listeners are still registering with ONS even with the ons.config file pointing to a different port. If you look at the output I posted Thursday you can even see that the listener from our ASM home is connected to ONS – but the ASM ons.config file points to local port 6102. This begs the question, how do the listeners know what port ONS is running on? And why did the ASM listener connect but the other 10g listeners on the box didn’t? Furthermore, on Friday we configured a listener with no ons.config file at all (renamed it to fix the crazy connection problem) – and then that listener connected to ONS as normal without causing any problems! Yet the third listener on the box still doesn’t connect to ONS. If anyone has any ideas then please leave a comment…

Lastly I just thought I’d post the input and output of a successful rconfig run for reference. FYI, we were moving an Oracle Apps 11i (11.5.10.2) database from filesystem to ASM and from single instance to RAC. The whole run took about 20 minutes (it is still a relatively empty Apps database).
› Read more

And the saga continues… Tuesday I mentioned a PRKP-1001 from rconfig – however I think that the root cause might have been something quite different from my original suspicion. Yesterday we discovered a rather major issue on the server… when trying to configure EM dbcontrol I started receiving errors indicating that the ports weren’t free. This didn’t make sense because nothing else was using those ports.

A quick glance at netstat revealed that the ports had been used by outbound connections and were sitting in a TIME_WAIT state (indicating that the connection had closed already; the entry sits in the kernel table for a preconfigured time that defaults to about 2 minutes in linux but configurable through /proc/sys/net/ipv4/tcp_fin_timeout). Some process was opening about 60,000 connections in a matter of seconds — effectively blocking every available port and launching a DOS attack on itself. So it was time to do a little investigation. We ended up finally cracking the case this morning and I think that the troubleshooting process we went through might be informative since it could help troubleshoot other similar problems.

So where did we start? I’m not going to be terribly verbose but just show you the commands I used to track it down and show you the output. You can adapt these methods to your problem. :)
› Read more

This week I’m working with a client to help them get an Oracle Applications environment up and running on RAC. We started with metalink note 362135.1 as a general guide for the process. Although I am much more familiar with RMAN this note recommended using rconfig to convert the database to RAC+ASM. (rconfig is the tool used by OEM on the backend to do conversions and is documented in Appendix D of the Clusterware Install Manual.)

rconfig is a pretty cool utility. Although it does not allow you to control much of the conversion process it is very slick and easy to use. However we did run into one rather odd problem…
› Read more

How ironic that just this afternoon I read James Morle’s recent whitepaper about Connection Management in an Oracle RAC Configuration. One of the first things that James unearthed during his testing was a bug in how Oracle’s assistants configured RAC networking settings – specifically in how they don’t correctly set the LOCAL_LISTENER parameter. Of course you don’t need to use DBCA or NETCA to have this problem – you can set it incorrectly or forget to set it yourself just as easily when manually creating a database.

The reason it’s ironic that I read his paper today is because about two hours later I ran into a very similar problem myself – I was receiving ORA-12545 from the client every time I tried to connect despite the fact that my TNSNAMES file was exactly correct. Although my problem was slightly different from James’ they both had the same solution: set the LOCAL_LISTENER correctly.

Let’s recap exactly how I received the ORA-12545 error.
› Read more