Went straight from Mandie’s to Laura’s - and it was awesome! Unfortunately I was having so much fun that I stayed WAY later than I should have… and didn’t finish up the documents I was supposed to do this weekend for my last client in Raleigh. Plus, it’s late now and I’ve got an EARLY morning tomorrow to get out to the office. Oh man…

Hehe…

Looks like Solomon didn’t get enough sleep last night. No worries though, he’s catching up while we work on painting a few eggs. We had a little trouble with the eggs though. Note for next year: easy on the water when making the coloring stuff. Don’t dilute it so much.

Just three quick updates…

First of all, yesterday we re-ran rconfig a few times and confirmed - for sure - that the PRKP-1001 and CRS-0215 errors tuesday were from the listener/ONS problem.

Secondly, just an observation - I noticed yesterday that somehow a few of the listeners are still registering with ONS even with the ons.config file pointing to a different port. If you look at the output I posted Thursday you can even see that the listener from our ASM home is connected to ONS - but the ASM ons.config file points to local port 6102. This begs the question, how do the listeners know what port ONS is running on? And why did the ASM listener connect but the other 10g listeners on the box didn’t? Furthermore, on Friday we configured a listener with no ons.config file at all (renamed it to fix the crazy connection problem) - and then that listener connected to ONS as normal without causing any problems! Yet the third listener on the box still doesn’t connect to ONS. If anyone has any ideas then please leave a comment…

Lastly I just thought I’d post the input and output of a successful rconfig run for reference. FYI, we were moving an Oracle Apps 11i (11.5.10.2) database from filesystem to ASM and from single instance to RAC. The whole run took about 20 minutes (it is still a relatively empty Apps database).
Read more

And the saga continues… Tuesday I mentioned a PRKP-1001 from rconfig - however I think that the root cause might have been something quite different from my original suspicion. Yesterday we discovered a rather major issue on the server… when trying to configure EM dbcontrol I started receiving errors indicating that the ports weren’t free. This didn’t make sense because nothing else was using those ports.

A quick glance at netstat revealed that the ports had been used by outbound connections and were sitting in a TIME_WAIT state (indicating that the connection had closed already; the entry sits in the kernel table for a preconfigured time that defaults to about 2 minutes in linux but configurable through /proc/sys/net/ipv4/tcp_fin_timeout). Some process was opening about 60,000 connections in a matter of seconds — effectively blocking every available port and launching a DOS attack on itself. So it was time to do a little investigation. We ended up finally cracking the case this morning and I think that the troubleshooting process we went through might be informative since it could help troubleshoot other similar problems.

So where did we start? I’m not going to be terribly verbose but just show you the commands I used to track it down and show you the output. You can adapt these methods to your problem. :)
Read more

Well the pic hardly does this place justice. I found a great little coffeeshop in downtown Raleigh called Helios to hang out at tonight. They just hung a bunch of new art while I was sitting here. I also met another guy named Jeremy who had grown up in Chicago. They have free wifi and good drinks. And you can’t beat the atmosphere; chill; lots of people sitting around chatting… the perfect place to just hang out. I’m looking forward to my next trip to Raleigh!

Hey it turns out that there’s swing dancing in Raleigh and I found it! And if you want to see why lindy hoppers are so awesome then read the thread where I posted on Hackzapoppin.

So I got there a bit late due to a few extra hours troubleshooting an issue for the client I’m working with this week… but once I got there I had an awesome time. There were lots of great dancers and some great music. My only complaint was that they waited until right when I wanted to leave to start playing all the awesome music… which made me stay much later than I should have… that’s why it’s quarter to one and I’m just now going to bed. Oops…

By the way, the place was packed all night - I took this picture right before I left at like 12:30am which is why there are only a few couples on the floor.

This week I’m working with a client to help them get an Oracle Applications environment up and running on RAC. We started with metalink note 362135.1 as a general guide for the process. Although I am much more familiar with RMAN this note recommended using rconfig to convert the database to RAC+ASM. (rconfig is the tool used by OEM on the backend to do conversions and is documented in Appendix D of the Clusterware Install Manual.)

rconfig is a pretty cool utility. Although it does not allow you to control much of the conversion process it is very slick and easy to use. However we did run into one rather odd problem…
Read more

How ironic that just this afternoon I read James Morle’s recent whitepaper about Connection Management in an Oracle RAC Configuration. One of the first things that James unearthed during his testing was a bug in how Oracle’s assistants configured RAC networking settings - specifically in how they don’t correctly set the LOCAL_LISTENER parameter. Of course you don’t need to use DBCA or NETCA to have this problem - you can set it incorrectly or forget to set it yourself just as easily when manually creating a database.

The reason it’s ironic that I read his paper today is because about two hours later I ran into a very similar problem myself - I was receiving ORA-12545 from the client every time I tried to connect despite the fact that my TNSNAMES file was exactly correct. Although my problem was slightly different from James’ they both had the same solution: set the LOCAL_LISTENER correctly.

Let’s recap exactly how I received the ORA-12545 error.
Read more