• 100% class met expectations, would recommend to others

• 66% easy to follow, could use skills in working environment

• 100% already familiar with oracle, 90% use oracle daily

• 0 negative reviews of instructor (phew!)

• 1 negative review of curriculum: said practice exercises weren’t relevant but training manual was still above average.

• 0 negative “comments”

There were several positive comments such as this: “I would recommend this class to others. This setup is perfect to pick up new skills and expose what ifs w/out worrying about pressing the wrong button.”

There were also some suggestions for improvements which I’ve taken note of. The most common suggestion was this: “More time to practice / explore new concepts.”

What’s Next

The RAC Attack curriculum has been around for over three years now – and I’m still getting requests to put it on.  Seems that a number of people have felt that it’s a decent quality class and handbook.  I talked to several other Oracle instructors and speakers at Collaborate this year about the RAC Attack curriculum – including an in-depth conversation with Arup Nanda.

As a result of these conversations, I did some careful thinking and I’ve been very busy for the past few weeks since Collaborate.  And I now have three major announcements about RAC Attack:

1. Last week, I published an updated RAC Attack Lab Handbook.  The handbook is now completely self-contained for home use.  I’ve added significant new content:
   • Detailed hardware requirements (taking into consideration optimal I/O and memory configuration)
   • How to install and setup VMware environment (including networking and storage)
   • How to create the RAC Attack Classroom DVD (this is actually scripted now)

This means that you can now (finally) setup RAC Attack on your own computer – without needing any external assistance!

2. Although I created most of the labs for RAC Attack, some labs were also contributed by Dan Norris and Parto Jalili.  Last week I also obtained their permission to move all RAC Attack content under a Creative Commons Attribution-ShareAlike license – which you will see in the latest handbook.The RAC Attack lab handbook has now been donated to the Oracle professional community with a solid copyleft license.
   • New DBAs are invited to use this material at home.
   • Experienced DBAs are invited to make corrections and enhancements to the textbook.  Also you can add your own labs which work on the RAC Attack “platform”.
   • Instructors are invited to use this material partially or completely in their own classes, as they wish. You can even run your own class based solely on this material – without needing anyone’s permission. Also, you can charge for these classes – there isn’t a “non-commercial” limitation on the terms of our license.

On a related note, all code related to RAC Attack (including the jumpstart scripts and supplemental DVD build script) is now under the GPL and has been published on github for easy downloading, copying and branching.

3. Finally… how are you supposed to make corrections, enhancements and additions if you’ve only got a PDF version?  Right now the textbook sources are in OpenOffice format (with a master document and about 50 lab documents).  Although this is very well suited to print media, it’s difficult to coordinate with very many authors or editors.  I’ve decided that the best way to truly encourage collaboration with other teachers & instructors is to use the best platform on the web for collaborative open-content textbook projects: the RAC Attack lab textbook has a new home on wikibooks.

   http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home

   I’m just getting started now – but I’m hoping to have all existing RAC Attack content on this wiki textbook by the end of the month.  Please have a look at the project page and consider getting involved!

Feedback

I consider these to be some major changes.  I’ve put some careful thought into them, and I’m excited about the new direction for this class.  Let me know what you think.

Everything started with this short, innocent-seeming instant message:

there is something wrong with a 10g server config. connecting to it from windows works, the linux instant client returns ORA-28547. the same linux client can connect to 9g installation though. google is not helpful, does anyone have an idea?

Seems like a simple question! But we ended up chatting on-and-off throughout the whole day as I helped troubleshoot the problem. We finally found the problem at the end of the day – and I promise you that it would have taken Oracle Support weeks (or even months) to find this problem. Yikes!

Troubleshooting Summary

The client was php on a linux box. This client was connecting to the database server over a network connection which includes a VPN tunnel. The database is being upgraded from 9i to 10g – and the new database is already running on a new server. The client can connect to the 9i database but it can’t connect to the 10g database. The developer also has a windows box with an oracle client; this windows box can connect to the 10g database over the network link just fine. TNSNAMES is not used, and all connect strings are identical except for the IP addresses. The 11g instant client for Linux is being used.

Our investigation process involved a lot of trial-and-error. We repeated this cycle: gather data, postulate a theory, suggest an action. We knew that we were close because often the error messages would change – but they didn’t go away! For posterity (and google), here are the errors we encountered:

• ORA-28547: connection to server failed, probable Oracle Net admin error
• ORA-12170: TNS:Connect timeout occurred
• ORA-03113: end-of-file on communication channel
• WARNING: inbound connection timed out (ORA-3136)
• Fatal NI connect error 12170.
• TNS-12535: TNS:operation timed out / ns secondary err code: 12606
• ntt2err:Read unexpected EOF ERROR on 7
• ORA-12537 or TNS-12537: TNS: connection closed / ns2: 12560

We started with simple changes before looking into deep logs or tracefiles. I didn’t have any access to the actual system, just a chat connection to the developer. And of course I was hoping for a quick fix… though it didn’t end up happening. But FWIW, here’s everything we tried:

• Double-check that TNSNAMES or connection string is correct
• If RAC, then check REMOTE SERVER connect string in LSNRCTL output (this comes from local_listener/remote_listener settings)
• Add server hostname to local client DNS. (Connect string used IP, but server might pass back hostname for re-connect during connection process.)
• Check if DB server is behind a firewall, client seeing IP address different than server – see metalink note 270096.1 (can possibly address on 10g+ with DIRECT_HANDOFF_TTC_ in listener.ora or USE_SHARED_SOCKET in windows registry)
• Attempt to disable shared-socket/direct-handoff. (Trace revealed that the working 9i connection didn’t bequeath its connection but broken 10g connection did.)
• Downgraded instant client from 11g to 10g (to match target database)
• Check if there’s a software firewall active on the database server

To gather data, of course we first just captured error messages from the client. Later, we checked the server sqlnet.log, alert log, bdump/udump, and windows OS event log for messages. The developer I was working with was very savvy, so I actually had him generate some client trace files fairly early on – from both working and non-working connections.

We heavily leveraged the two working connections (linux->9i and windows->10g) by comparing tracefiles. In the end, these tracefiles tipped us off to the real problem.

Infuriating

I expected a quick, easy answer – but it wasn’t. It drove me crazy all day.

From the beginning it was obviously some kind of networking problem. The database was running on a windows server, so my first thought was that it was related to the re-connect that happens when the listener hands off the connection to the database process. But the trace file revealed that 10g was actually bequeathing (or rather, sharing) the connection – shame on me for not keeping up to speed with Oracle/Windows developments!! (I usually work with unix environments.)

Generating a client trace is easy – you just edit (or create) the text file sqlnet.ora and add two lines TRACE_DIRECTORY_CLIENT and TRACE_LEVEL_CLIENT (detailed description). If you’re using the instant client then you can set the environment variable TNS_ADMIN to the directory containing this file.

Here are a few lines from the client trace we generated:

2010-09-07 20:41:00.451918 : nscall:connecting...
2010-09-07 20:41:00.452854 : nsopen:opening transport...
2010-09-07 20:41:00.453086 : nttcni:Tcp conn timeout = 60000 (ms)
2010-09-07 20:41:00.504377 : nttcni:connected on ipaddr 10.10.0.97
2010-09-07 20:41:00.506948 : nscon:doing connect handshake...
2010-09-07 20:41:00.506979 : nscon:sending NSPTCN packet
2010-09-07 20:41:00.507113 : nttwr:socket 7 had bytes written=233
2010-09-07 20:41:00.567169 : nttrd:socket 7 had bytes read=8
2010-09-07 20:41:00.567434 : nscon:got NSPTRS packet
2010-09-07 20:41:00.567692 : nscon:sending NSPTCN packet
2010-09-07 20:41:00.567817 : nttwr:socket 7 had bytes written=233
2010-09-07 20:41:00.617139 : nttrd:socket 7 had bytes read=32
2010-09-07 20:41:00.617362 : nscon:got NSPTAC packet
2010-09-07 20:41:00.617453 : nscon:connect handshake is complete
2010-09-07 20:41:00.620093 : nsdofls:sending NSPTDA packet
2010-09-07 20:41:00.620253 : nttwr:socket 7 had bytes written=156
2010-09-07 20:41:00.667129 : nttrd:socket 7 had bytes read=127
2010-09-07 20:41:00.667359 : nsrdr:got NSPTDA packet
2010-09-07 20:41:00.670112 : nttfpwr:socket 7 had bytes written=47
2010-09-07 20:41:00.717287 : nttfprd:socket 7 had bytes read=181
2010-09-07 20:41:00.719612 : nttfpwr:socket 7 had bytes written=2011
2010-09-07 20:41:00.720401 : nttfpwr:socket 7 had bytes written=666
2010-09-07 20:42:01.577217 : ntt2err:Read unexpected EOF ERROR on 7

The presence of NSPTRS indicates that the connection is being bequeathed/shared on the server. Metalink notes 270096.1 and 1007807.1 are very helpful for understanding this handshake process. Here’s a brief outline of what’s happening:

One other important piece of information we eventually unearthed was this error in sqlnet.log on the server:

Fatal NI connect error 12170.
...
  Tns error struct:
    ns main err code: 12535
    TNS-12535: TNS:operation timed out
    ns secondary err code: 12606

Both the server and client were seeing the connection spontaneously die! Yet the client connects fine to a 9i database, and a different client connects fine to this database!! Right about now, I was thinking of two possible root causes:

1. Rogue firewall somewhere that’s killing the connection for some unknown reason. But the developer is pretty sure that there’s no firewall killing connections.
2. Bug?!

Breakthrough

At this point I asked the developer to run a level 16 client trace (which includes packet dumps) and I started reading through the dump. Then I noticed two important things.

1. There is a 60 second pause before the client detects the dead connection. This is equal to the TCP connection timeout. I suddenly realized that the client “EOF” error was not from the server terminating the connection – the client was timing out while waiting for the server. And then I realized that the server was doing the same thing – waiting for the client!
2. I looked a lot closer at a certain section of the tracefile that immediately precedes this timeout. In particular, I looked at the packet contents and sizes – and I compared with the working 9i connection.

Here’s a snip from the diff between the working 9i connection and the broken 10g connection:

 nttfpwr:socket 7 had bytes written=47
-nttfprd:socket 7 had bytes read=164
-nscontrol:Vect I/O support: 02010-09-07 20:48:06.447554 : nscontrol:cmd=44, lcl=0x0
-nttfpwr:socket 7 had bytes written=1080
-nttfprd:socket 7 had bytes read=949
-nttfpwr:socket 7 had bytes written=213
-nttfprd:socket 7 had bytes read=22
+nttfprd:socket 7 had bytes read=181
+nscontrol:Vect I/O support: 02010-09-07 20:52:12.267498 : nscontrol:cmd=44, lcl=0x0
+nttfpwr:socket 7 had bytes written=2011
+nttfpwr:socket 7 had bytes written=666
+ntt2err:Read unexpected EOF ERROR on 7

When I looked at the packet contents for the broken connection, it was immediately evident that those TWO packets (2011 bytes and 666 bytes) were ONE message. The message was too large for one packet so it had been fragmented.

However the first big message sent to the 9i database is 1080 bytes. It gets sent in a single packet.

I asked the developer, what’s the max packet size on the Linux box? He replied that the MTU was 1456.

WILD guess… I wondered, was the big packet getting dropped somewhere? (Windows client trace shows the large packet getting through without any problem, but still…)

Oracle is Better than MySQL

…ok, maybe the standard MySQL client has this functionality too – I actually don’t know. I just wrote that title because it’s provocative!! But when it comes to tweaks, Oracle really does shine. It has more switches and knobs than you’d believe.

It’s very simple to reduce the max packet size in the Oracle client: just add “(SDU=####)” to the connect string. The default is 2048 – which explains the fragment sizes we observed.

Here’s what transpired next in my chat (please excuse the language):

f**k me
just tried with 1024 and it worked

Seems that we identified the problem! And most importantly, we have a working connection so that this developer can get back to what’s important – coding.

Of course, we all know what happened after that – virtualization.  It didn’t take long before my home-built test clusters were running on VMware.  (Personally, I think that virtualization really started because of those NES and SNES emulators.  Most great achievements start with a geek who wants to play more video games.)  There are lots of people now who run RAC on virtual environments and it’s easy to find tutorials on the web for many different OS and VM combinations.

Low-Memory Linux

Something I haven’t seen many other people do is RAC with a very small memory configuration.  Like 760M of memory per server. (!)  Of course you’d only do this for a hobby setup – never on a system where you want any kind of support.  But I’m kinda cheap… and running RAC on these small VMs means that I don’t have to go buy an expensive new home computer.  My current gateway laptop with Vista Home does the job quite nicely!

10.2 and 11.1 RAC will install and run on servers with 760M of memory. But things were a little unstable at first. Now I’m the curious type… I like to fiddle with things… so I investigated a little bit.

Basic Unix Investigation

There are two basic investigation scenarios:

In this particular case, I noticed pretty quickly from the top utility that one process was consuming over 30% of the system’s memory!  (Note: in top, you can press the ‘<’ and ‘>’ keys to move the sort column left and right.  The initial sort column is %CPU.  I moved it one column to the right, sorting by %MEM.)

top - 18:41:13 up  5:28,  3 users,  load average: 0.04, 0.35, 0.60
Tasks: 180 total,   2 running, 178 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  3.7%sy,  0.0%ni, 89.4%id,  6.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    767020k total,   754296k used,    12724k free,     7084k buffers
Swap:  1540088k total,   654696k used,   885392k free,   361800k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17435 oracle    RT   0  230m 229m  31m S  0.3 30.6   0:26.21 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
 7765 oracle    15   0  464m 111m 102m S  0.0 14.9   0:04.52 ora_smon_RAC1
 7743 oracle    -2   0  445m  90m  83m S  0.0 12.1   0:12.01 ora_lms0_RAC1
 7783 oracle    15   0  440m  70m  66m S  0.0  9.4   0:05.35 ora_mmon_RAC1
20321 oracle    15   0  438m  48m  45m S  0.0  6.5   0:00.88 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8676 oracle    16   0  437m  46m  45m S  0.0  6.2   0:02.69 oracleRAC1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 8801 oracle    15   0  440m  44m  41m S  0.0  6.0   0:04.57 ora_cjq0_RAC1

This process (ocssd) is Oracle’s Cluster Synchronization Services Daemon.  It’s the process that sends and receives heartbeats from other nodes. Any delays sending or receiving those heartbeats can cause node evictions (a.k.a. server reboots) – so it’s a pretty important process! That’s why it runs with realtime (RT) scheduling priority, as you can see in the above output from top.

I was surprised that CSS uses so much physical memory – usually Linux is very good at memory management. In top, the VIRT column shows how much total memory each process is using, while the RES column shows how much actual physical memory Linux has allocated to it. It’s clear that Linux is pretty actively managing the physical memory for other processes.

A quick glance at vmstat shows that although we are actively swapping, it seems under control. This is about what I’d expect when we’re idle and all of the processes except CSS are sharing only 500M of memory:

collabn1:/home/oracle[RAC1]$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 604144  14024  34948 300632    0    0    32    86 1044 1797  1  8 84  7  0
 2  1 603912  11224  34968 303968   87    0   763    81 1083 1902  3 11 56 30  0
 0  0 604840  12660  34912 303904   25    0   253    38 1050 1975  2 11 69 18  0
 0  0 604840  12728  34924 303900    0    0    68   106 1043 1975  1  9 82  8  0
 0  0 604784  14536  34936 304140   19    0    89    71 1043 2042  1  5 81 13  0
 1  0 604752  14168  34944 304496    6    0   119    21 1040 2016  2  8 80 11  0
 0  1 604736  18732  35020 306212    0    0   384    73 1050 2489  1 10 70 19  0
 1  1 604736  11144  35232 311252  104    0  1209   128 1074 2024  2 12 38 47  0
 3  1 607900   8056  30788 307352   77 1500   504  1661 1055 2642  9 16 56 19  0
 0  0 607900   9836  30800 307360    0    0    71    70 1031 1798  1  6 85  8  0
 1  0 607884  10536  30812 307400    8    0    44    93 1031 1832  1  4 87  8  0

The SO column tells us when memory is written to disk (and removed from physical). The SI column tells us when memory is read from disk (and put back in physical). On a side note, remember that on a healthy Unix system the free memory is always small. Sometimes this is confusing at first.

Linux Process Memory Investigation

Nonetheless, I’m not happy that CSS is using 30% of my physical memory in this highly-constrained hobby environment. Why is Linux allowing this? The first clue comes simply from the output of the familiar unix ps utility:

collabn1:/home/oracle[RAC1]$ ps v -C ocssd.bin
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
17435 ?        SLl    0:08      7   588 235331 234840 30.6 /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin

On Linux, the “v” flag tells ps to give information relevant to virtual memory. TRS and DRS tell me how much physical (resident) memory is used for machine executable code (text) and data, respectively. But more importantly – the STAT column gives some informative BSD-style flags about the process. That capital-L indicates that CSS has some pages that are locked into physical memory. Bingo.

If I have root access, then I can get a very detailed report on process memory usage with the pmap command. The output was a little long, so I’ve abbreviated it here:

[root@collabn1 ~]# pmap -x 17435
17435:   /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin
Address   Kbytes     RSS    Anon  Locked Mode   Mapping
00110000     656       -       -       - r-x--  libhasgen11.so
001b4000       8       -       -       - rwx--  libhasgen11.so
  ... 37 more library blocks
02a0d000     100       -       -       - rwx--    [ anon ]
02a26000       4       -       -       - --x--    [ anon ]
02a27000   10240       -       -       - rwx--    [ anon ]
03427000       4       -       -       - --x--    [ anon ]
03428000   10240       -       -       - rwx--    [ anon ]
  ... 10 more anonymous blocks, half are 10240K
08048000     592       -       -       - r-x--  ocssd.bin
080dc000       4       -       -       - rwx--  ocssd.bin
  ... 30 more anonymous blocks, half are 10240K
bfe40000     148       -       -       - rwx--    [ stack ]
bfe65000       8       -       -       - rw---    [ anon ]
-------- ------- ------- ------- -------
total kB  235920       -       -       -

Interestingly, the linux pmap utility does not indicate any locked memory! I don’t know whether that output column is non-functional or if it refers only to some particular kind of locking. But at any rate, I know something is locked. I couldn’t think of anything better, so the next place I looked was in the Linux /proc pseudo-filesystem.

collabn1:/home/oracle[RAC1]$ grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Now we’re talking. The process has a total of 235920 kB of memory – and it’s ALL locked. On a normal RAC system you’d want this. Generally, important realtime processes should be locked so that they are never delayed by paging or swapping. (Remember how that could cause node reboots?)

But I personally doubt that all of the memory really NEEDS to be locked, and I think that Linux will actually do a decent job of not swapping the most important parts. And my highly constrained hobby environment will probably run much smoother if Linux has more flexibility when managing a measly 760M of memory.

Unlocking Linux Process Memory

But is it actually possible to unlock the process memory? As far as I know, Oracle provides no option to disable CSS memory locking. (For good reason.) There is the system call munlockall() – which unlocks all of a particular processes’ memory. But the CSS process itself would have to call this function. And of course it will not. Or will it?

If you’ve got root, then there’s a hacker-back-door way of doing this. Remember, you’d be crazy to try this anywhere besides a dark closet at home. And if you type too slow then CSS could reboot your machine.

But watch this…

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:    235920 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

[root@collabn1 ~]# gdb -p 17435 <<EOF
> call munlockall()
> quit
> EOF

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Attaching to process 17435
Reading symbols from /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin...done.
  ... 17 more Reading/Loading symbols.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f629f0 (LWP 17435)]
  ... 19 more New Threads.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libnsl.so.1...done.
  ... 13 more Loaded/Reading symbols.
0x008e7402 in __kernel_vsyscall ()
(gdb) $1 = 0
(gdb) The program is running.  Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
Detaching from program: /u01/crs/oracle/product/11.1.0/crs/bin/ocssd.bin, process 17435

[root@collabn1 ~]# grep Vm /proc/17435/status
VmPeak:   235924 kB
VmSize:   235920 kB
VmLck:         0 kB
VmHWM:    234844 kB
VmRSS:    234840 kB
VmData:   202272 kB
VmStk:       156 kB
VmExe:       592 kB
VmLib:     31960 kB
VmPTE:       268 kB

Ha. This is something you won’t find on metalink. After generating some activity on the system, the top utility shows me that Linux has significantly reduced the physical memory used by CSS.

These days I’ve actually scripted this for my home and classroom VM environments. I haven’t done a careful comparison or analysis, but it really has seemed to me that my low-memory Linux systems run noticeably smoother.

We’re internally debating whether or not we should include OCFS2 in this design right now, and I’m curious if anyone has arguments one way or the other to share. Our standard design on Solaris does utilize a cluster filesystem and we would welcome a similar design, but there are some concerns about the readiness, stability and future of OCFS2.

OCFS2 is being considered for these four use cases:

• database binaries (vs local files or NFS)
• diag top (11g) or admin tree (10g) (vs local files or NFS)
• archived logs
• backups

Other files will be stored in ASM.

I have seen mention in blogs such as http://bigdaveroberts.wordpress.com/ of something called ASMFS in 11gR2 and I’m wondering – will this feature (if included) have any impact on Oracle’s commitment to OCFS2 development? Could Oracle conceivably develop a whole new cluster filesystem and put their full weight behind it as they did for ASM storage, leaving OCFS2 as a lower priority for new features and improvements? Has Oracle demonstrated significant commitment to OCFS2 development and support in the past, and is this a mature enough technology for wide-scale deployment?

Just looking for opinions. :)

Actually it’s part of a bigger question of how we develop elegant software – other related design issues in this project are clear end-to-end traceability (requirement -> source code version -> released binary package), solid change control, integrated testing framework and readiness for automation (build and test). But the one I’m interested in right now is this: how can I develop a robust version numbering strategy?

First of all, there really isn’t a one-size-fits-all “correct” version numbering scheme. The best way to choose version numbers is influenced by your software development philosophy (and resultant development lifecycle), project management approach and development team. (Globally distributed informal volunteer network or local engineers on payroll?) So the best solution for my current project might not be the best solution for my next project.

But lets take a closer look at this project. First, a little background on the art of version numbering. In the open-source world, many projects follow a three-digit version: MAJOR.MINOR.PATCH. Also, it’s clear that the version number is closely tied to the development lifecycle. You can find plenty of example discussion from open-source projects such as the Linux kernel, Linux desktop distros, Eclipse, Apache APR and FarCry CMS. Before the 2.6 series the linux kernel used an even-odd scheme to distinguish between unstable and stable releases but that has now been abandoned and most people seem to regard this as a good thing.

The best overview of the topic that I could find was the wikipedia article on Software Versioning. The section about pre-release versions does a good job of describing one requirement of my current project. I also found the section about internal version numbers interesting; I’ve worked on projects in the past with that strategy.

I found an article on linux.com written by Nathan Willis a few years ago offering some principles. I agree with his first and third suggestions: “Pick a numbering scheme, then don’t change it. Ever.” and “Make friends with infinity.” Best to think this thing out at the front end of a project (before 1.1) and you shouldn’t be bothered by big version numbers. However I don’t buy his second point about the meaning of the decimal; I think most everyone today is used to the meaning of the decimal in version numbers.

For some alternate viewpoints I came across something Zack Weinberg wrote back in 2002. He states strongly that the decimal point isn’t mathematical, which I agree with. However contrary to his article, in our development lifecycle the test releases do need version numbers. But he has an interesting suggesting of using date-based numbers for development snapshots – that’s an interesting idea which I’ll have to think about. His list if ways to do it wrong has a good point about distinguishing between version x and version x.0 although I’m not in full agreement with the other list items. I actually find the GNU suggestion for numbering test versions very interesting – I’ll have to give that some thouht. (Tests versions for 4.6 are numbered 4.5.90 – 4.5.99)

Ultimately I don’t think either of these two sets of guidelines is robust enough for me – especially for repackaging the oracle database binaries. Some of these might seem obvious, but I thought a good place to start might be by listing what I would consider to be requirements for an elegant and robust version numbering scheme:

1. elegant, clear, understandable version numbers that are as simple as possible
2. support for single-digit or double-digit versions (1,2,3,4 or 1.0, 1.1, 1.2 ,2.0)
3. versions for pre-release builds such as test builds (for the quality process) and early-access beta builds.
4. very clear difference between pre-release and release versions, so someone doesn’t mistakenly install a pre-release build when they want a production build.
5. names need to sort correctly – pre-release before release. it’s critical that RPM and RHN sort them right; it would be nice if unix “ls” did too.
6. support for continuing development. after release 1.0, the 2.0-beta version should sort after release 1.0 and before release 2.0
7. support for special branch versions – version 1.0+patch12345, where patch12345 is not included in version 2.0 which has different patches
8. clear difference between pre-release and release versions on special branches.

I have to be honest – this is tough. I’ll be giving it some thought over the next few weeks. Let me know if you have any suggestions. :)

I’ve been doing a lot of scripting work lately. Although I can’t write about everything I’m doing, I would like to post a pattern that I thought could be useful to other folks. I needed to update some properties in the LISTENER.ORA file from a shell script. The problem is that basically you have to clobber it or properly parse it! Even though I initialized this listener file myself, there is always the possibility that a DBA could manually edit it – so I didn’t want to clobber the contents. That means that I had to parse it.

So how do you parse a listener file from a shell script? I came up with a nifty way to tokenize and parse the file through a simple awk script combined with sed – and it seemed rather useful so I refactored it a few times to apply to more general use cases. I think someone else could use something like this too so I’m posting it – do me a favor and keep the reference to ardentperf in there if you copy the code.

There’s two functions that are basically the same pattern: the first will read an attribute and the second will set one. I’ve included some examples at the bottom of the post. I’ve tried to make the awk code portable but so far I’ve only tested it with gawk and mawk on a few flavors of Linux.

Reading an Attribute

# get a property from stdin (listener.ora format) for a specific listener and
# print it on stdout. this awk script will tokenize the listener file so that
# it can find the property no matter what format the file is in. it will only
# print the first match (for example an address).
#   param 1: top-level property filter (e.g. listener name)
#   param 2: leaf property to get (e.g. host, port, protocol)
# by Jeremy Schneider - ardentperf.com
getlistenerproperty() {
  sed -e 's/=/`=/g' -e 's/(/`(/g' -e 's/)/`)/g'|awk 'BEGIN{level=1} {
    wrote=0
    split($0,tokens,"`")
    i=1; while(i in tokens) {
      if(tokens[i]~"^[(]") level++
      if(tokens[i]~"^[)]") level--
      if(level==1&&i==1&&tokens[i]~"[A-Za-z]") TOP=tokens[i]
      if(toupper(TOP)~toupper("^[ \t]*'"$1"'[ \t]*$")) {
        if(propertylvl) {
          if(level>=propertylvl) {
            if(tokens[i]~"^="&&level==propertylvl) printf substr(tokens[i],2)
              else printf tokens[i]
            wrote=1
          } else propertylvl=0
          found=1
        }
        if(!found&&toupper(tokens[i])~toupper("^[(]?[ \t]*'"$2"'[ \t]*$")) propertylvl=level
      }
      i++
    }
    if(wrote) printf "\n"
  }'
}

Setting an Attribute

This uses essentially the same pattern but it sets an attribute instead of reading one. It outputs the new LISTENER.ORA file on stdout.

# read stdin (listener.ora format) and change a property for a specific listener,
# writing the updated file on stdout. the function will tokenize the listener file
# on stdin so that it can find the property no matter what format the file is in.
# it will only replace the first match (for example an address).
#   param 1: top-level property filter (e.g. listener name)
#   param 2: leaf property to set (e.g. host, port, protocol, address, etc)
#   param 3: new value of the property
# by Jeremy Schneider - ardentperf.com
setlistenerproperty() {
  sed -e 's/=/`=/g' -e 's/(/`(/g' -e 's/)/`)/g'|awk 'BEGIN{level=1} {
    wrote=0
    split($0,tokens,"`")
    i=1; while(i in tokens) {
      if(tokens[i]~"^[(]") level++
      if(tokens[i]~"^[)]") level--
      if(level==1&&i==1&&tokens[i]~"[A-Za-z]") TOP=tokens[i]
      if(toupper(TOP)~toupper("^[ \t]*'"$1"'[ \t]*$")) {
        if(propertylvl) {
          if(level>=propertylvl) {
            if(!found) {
              if(tokens[i]~"^="&&level==propertylvl) printf "= '$3'"
                else printf "'$3'"
              wrote=1
            }
          } else { printf tokens[i]; wrote=1; propertylvl=0 }
          found=1
        } else { printf tokens[i]; wrote=1 }
        if(!found&&toupper(tokens[i])~toupper("^[(]?[ \t]*'"$2"'[ \t]*$")) propertylvl=level
      } else { printf tokens[i]; wrote=1 }
      i++
    }
    if(wrote||i==1) printf "\n"
  }'
}

Examples

First I’ll setup a sample LISTENER.ORA file that I can use to demonstrate a few different uses:

jeremy@debianvm:~/lsnr$ cat list-ardent.ora
TRACE_LEVEL_ARDENTL1 = ADMIN
TRACE_FILE_ARDENTL1 = listener
TRACE_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/trace
LOG_FILE_ARDENTL1 = listener
LOG_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/log
LOGGING_ARDENTL1 = ON

#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )

SID_LIST_ARDENTL1 =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = JTEST1)
      (SID_NAME = JTEST1)
      (ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
    )
  )

WALLET_LOCATION = (
  SOURCE=
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY=/home/oracle/Wallet_server))
  )

Now lets look at a few examples of reading properties. Note that – by design – it only reads the first match.

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 host
 10.9.8.33
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 port
 1522
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 address
 (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522)
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 description

    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 ardentl1

  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty log_directory_ardentl1 log_directory_ardentl1
 /u01/app/oracle/product/10.2.0/db_1/network/log
jeremy@debianvm:~/lsnr$ 

And finally, a few examples of changing properties. Note that the entire file is always displayed but I’ve only shown the relevant excerpts.

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|setlistenerproperty ardentl1 host 1.2.3.4
...
#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 1.2.3.4)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
...

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|setlistenerproperty ardentl1 address "(HOST=1.2.3.4)(PORT=999)"
...
#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (HOST=1.2.3.4)(PORT=999))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
...

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|setlistenerproperty logging_ardentl1 logging_ardentl1 OFF
TRACE_LEVEL_ARDENTL1 = ADMIN
TRACE_FILE_ARDENTL1 = listener
TRACE_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/trace
LOG_FILE_ARDENTL1 = listener
LOG_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/log
LOGGING_ARDENTL1 = OFF

#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
...

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|
> setlistenerproperty ardentl1 host 1.2.3.4|
> setlistenerproperty ardentl1 port 9999
...
#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 1.2.3.4)(PORT = 9999))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
...

And that’s about it. Leave me a comment and let me know if you find this useful. :)

Any software engineer who has worked on large projects – industry or community – can tell you the importance of solid change control processes. So… since everything for the build had to be checked into clearcase… yes, we checked Oracle 10g into the repo. A 2G tarball. And whenever there was a 1K patch to the oracle install? A brand new 2G tarball. The clearcase guys loved me.

And that was how I started thinking about an automated build process. I don’t need to check B24792-01_1of5.zip into the repository because it’s straight off edelivery. I can skip p7150622_10203_Linux-x86-64.zip since it’s direct from metalink. The only thing missing was a solid, simple, flexible program for automating the oracle install, patchset, CPUs and oneoffs – taking Oracle’s official bits as input.

Anyone else out there who could use a program like this? How about for rapid provisioning of servers? (Like all that grid buzz.) Grid Control does some basic stuff (if you can make it work) and there are advanced kits for the big data centers – but there have to be more people than just me who would love to have this program.

Proposing orainstall

That’s why I’m proposing to write a bit of software called orainstall. It would be script-based and cross-platform and intended for community use. I have some ideas for a design but I’m really interested to hear how you might use something like this and what features might be useful to you. Do you think it’s a good design? I’d also be interested in hearing if you’d be interested to use it or to help test it.

The program would be launched by running the main executable “orainstall”. The syntax could look something like this:

Command line syntax:
  ./orainstall <stepfile> [-oh <ORACLE_HOME>] [-tmp <ORAINST_TMP>]

Example:
  ./orainstall ora10203.sf -oh /u01/app/oracle/product/10.2.0/db_1 -tmp /u01/app/oracle/tmp

Return Value:
  0 - success
  1 - invalid stepfile
  2 - installation error
  3 - patching error

The Stepfile

What I’m thinking is that the install process is defined in an input file called the stepfile. The stepfile needs to accomodate a large variety of organizational requirements. Here’s my first draft for what a stepfile would look like:

# This stepfile instructs the oracle auto-installer how to setup the oracle
# home. Each line is executed in order from top to bottom.
#
# There are ten defined directives for this stepfile:
#   ORACLE_HOME - required
#     This will cause a search-replace of each response file to explicitly
#     define the oracle home. It must be defined before any open, install or
#     patch directives and it can only be defined once per stepfile.
#       Order of precedence:
#         1. command line
#         2. stepfile
#         3. environment variable
#   ORAINST_TMP - optional
#     Top-level temp directory for extracting contents of Oracle installers
#     and patches. Each installer or patch will be extracted into a
#     subdirectory of this temp location. It must be defined before any
#     open or patch directives. It can be defined multiple times in a
#     stepfile.
#       Order of precedence:
#         1. command line
#         2. stepfile
#         3. environment variable
#         4. "/tmp/orainstall"
#   OPEN /path/to/zip
#     Path to a zip file with database installer or major patchset which
#     requires use if OUI. The file is unzipped into a temporary location.
#     If several INSTALL directives are encountered in a row then they are
#     all unzipped into the same location; if there is another directive in
#     between then the next INSTALL is unzipped into a new temporary
#     location.
#   INSTALLPATH Disk1/runInstaller
#   INSTALLRSP install.rsp
#      "runInstaller -waitForCompletion -silent -responseFile <install.rsp>"
#     INSTALLPATH is optional - if not specified then the script will search
#     for it in the most recently created temp location. If there is more
#     than one runInstaller then it will use the first one it finds; search
#     order is undefined.
#     If no path is specified for the response file then orainstall will
#     search for it in this order:
#       1. current working directory (pwd) when orainstall was launched
#       2. same directory as orainstall binary (dirname $0)
#   OPATCH/path/p8000.zip
#     Path to special "OPatch" update - contents are unzipped directly to
#     ORACLE_HOME.
#   PATCH /path/p2345.zip
#     Path to a patch. Contents are unzipped to a temporary location then
#     applied by calling "opatch" from the ORACLE_HOME.
#   NPATCH /path/p1234.zip
#     Path to a patchset with multiple subpatches which need to be applied
#     with "napply" (such as a CPU). Contents are unzipped to temp location
#     then applied properly to the ORACLE_HOME.
#   CLEAN_TMP
#     Deletes all files from the current top-level temp directory.
#   REMOVE /path/to/file
#     Runs "rm -rf $ORACLE_HOME/path/to/file" - use carefully! Some sites
#     may require  removal of components for security requirements.
#
# If there are any errors (install or patch) then the process will fail.
#
#oracle_home /u01/app/oracle/product/10.2.0/db_1
orainst_tmp /u01/app/oracle/installers.tmp
open /net/rh5lab12/u10/stage/ora10201/B24792-01_1of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_2of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_3of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_4of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_5of5.zip
installpath database/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/database10201.rsp
installpath companion/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/companion10201.rsp
open /net/rh5lab12/u10/stage/ora10203/p5337014_10203_Linux-x86-64.zip
installrsp /net/rh5lab12/u10/stage/rsp/patchset10203.rsp
clean_tmp
orainst_tmp /u01/app/oracle/patches
opatch /net/rh5lab12/u10/stage/patch10203/p6880880_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p5892355_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p6455161_10203_Linux-x86-64.zip
npatch /net/rh5lab12/u10/stage/patch10203/p7150622_10203_Linux-x86-64.zip
remove /htmldb

Program Flow

The program flow would be pretty basic; it would validate the stepfile then execute it one line at a time. I thought of six things to validate before starting execution:

1. can read stepfile
2. one oracle_home, defined before open/install/patch, can create/write
3. orainst_tmp defined before open/install/patch, can create/write each
4. clean_tmp not defined before orainst_tmp
5. can read all open/install/patch input files
6. free space in oracle_home is greater than size of input files

Feedback

Maybe there aren’t as many people as I think who would use this. After all – most DBAs don’t install the database software very often; they just want to keep it running! But please let me know if it interests you. I’ll factor in the feedback I receive as I’m developing it.

Everybody knows that asmlib was very useful when it was first introduced with Oracle 10.1 to simplify a host of issues on Linux: direct async device access without raw devices, file permissions & ownership without custom code, and persistent device naming without devlabel.

But I’m now involved in setting some standards to be used across a large organization for Oracle 10.2 on RHEL5 and I’m wondering if there’s still a case for using asmlib. So I did a little trolling for info – which was suprisingly sparse. Had a hard time finding much, but after a lot of digging I think I’ve compiled a useful bit of information about benefits and drawbacks.

Benefits

It seems to me that the ASMLIB API was originally introduced to do more than just simplify file permissions – sounds like it was an alternative I/O API to the standard unix one, allowing ASM to access the underlying storage more efficiently and completely. I don’t think it’s just an “extra layer” – it’s an alternative code path to the std unix I/O libs. Like an ODM for block devices – and the idea was that there could be additional vendor implementations. And Oracle released an initial generic implementation on Linux under the GPL.

Some theoretical benefits of ASMLIB API:

• always uses direct, async i/o
• solves persistent device naming, even if underlying device moves across reboots
• solves file permissions and ownership
• reduced user mode to kernel mode context switches during I/O, possibly reducing CPU usage
• reduced file handle usage
• pass metadata such as I/O prioritization to storage device (don’t think this is implemented in the Linux version)

Drawbacks

However from what I can tell, the ASMLIB API didn’t quite catch on. Wim recommends it (as of Apr ’07) but Closson doesn’t really talk about it on his blog but has one disparaging off-hand reference about pitying people who use it. There have been no vendor implementations that I could find. (But I wonder what’s going on now with exadata and “iDB” which has I/O prioritization and predicate pre-filtering – could they have used the API here?) Wim’s post on Oracle Forums says that there isn’t “much of an io performance benefit” added by the linux ASMLIB implementation. And the OTN page hasn’t been updated for almost two years.

A RedHat consultant I’m working with recommended against using it and one of his reasons was that it introduces an unnecessary additional layer and dependency in the kernel. There have been a small number of past bugs in asmlib and it doesn’t seem to be frequently updated (once this year according to the changelogs)… but then it’s very simple code (mostly in a single small file).

Conclusions

After all of this, I’m not convinced that it’s worth installing ASMLIB if you’re comfortable with udev. On Linux, udev is definitely a better solution to handle persistent device naming and permissions and I don’t see enough benefits in asmlib to outweigh the very slight additional overhead.

So right now my preference would be to only install asmlib on old systems – Oracle 10.1 or Linux 2.4 – but to use udev and block devices for 10.2 and newer on 2.6 kernels. But the case isn’t closed and I’d love to hear from anyone who agrees or disagrees… thoughts?

References:

• Wim’s brief remarks on Oracle Forums – http://forums.oracle.com/forums/thread.jspa?threadID=498215
• In-depth information from ASM reference book – http://books.google.co.uk/books?id=HB453L86Q6AC&pg=RA1-PA142&lpg=RA1-PA142&dq=asmlib+api&source=web&ots=bEP4epFbp5&sig=47aLUWW9o73Snr1tcnvz_1h3prU&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PRA1-PA142,M1
• Uses submit_bio() call – source code – http://oss.oracle.com/viewvc/oracleasm/trunk/kernel/driver.c?view=markup
• cdos newsgroup thread with speculation about performance – http://groups.google.com/group/comp.databases.oracle.server/browse_thread/thread/905296d0aba00b84?pli=1
• tim hall was going to investigate though he didn’t publish results – http://www.oracle-base.com/blog/2006/05/01/asm-with-asmlib-or-raw-devices/
• kevin closson doesn’t say much but makes this off-hand remark – http://kevinclosson.wordpress.com/2006/11/02/dbwr-efficiency-aio-io-libraries-with-asm/
• short anonymous comment possibly by closson – http://archives.devshed.com/forums/databases-124/asmlib-1157660.html
• OTN official site for ASMLIB – http://www.oracle.com/technology/tech/linux/asmlib/index.html

First was a panel debate “To RAC or Not To RAC: What’s Best for HA.” Dan Norris invited me to participate in this panel along with Alex Gorbachev (Pythian), Neil Greene (Predictive Technologies) and Matt Zito (GridApp) – I certainly felt privileged to take part. Here are a few of the things that I took away from the debate… feel free to discuss:

• There are no silver bullets. And if there were, RAC wouldn’t be one. :) Many RAC implementation problems can be traced to poorly set expectations; knowing your requirements and RAC’s capabilities will go a long way towards making any project successful.

• RAC really shines when solving scalability problems. The simple act of moving an application from single-instance to RAC is more likely to improve throughput than service time. (Of course, improvement isn’t guaranteed: as Dan pointed out during our conversation it depends on where the bottlenecks are.)

• There are two major application design areas that need to be addressed when moving to RAC: (1) scalability and (2) connectivity. (There are other issues too but these are probably the biggest.) Regarding scalability, small inefficiencies will become major bottlenecks when you move your app to a cluster. Look out for any points of contention or serialization whether they’re logical (like an ID field that must be sequential) or physical (like free space management). On the connectivity front, there is both initial connection and load balancing to worry about.

• When it comes to HA, I personally think that failover clusters should get more serious consideration then they do. I think that there are essentially two differences between a failover cluster and RAC: (1) a few minutes of downtown each year, and (2) the prices for licenses – probably tens or hundreds of thousands of dollars minimum. You can even still use TAF with a failover cluster – it’s not much different than using it with Data Guard. The few apps that need extremely fast failover often can’t even wait a few minutes – they commonly run parallel systems in tandem to avoid failover altogether. Most other apps can tolerate an extra 60 seconds during failover, if we’re honest.

At any rate, I thought that the debate was informative and that Alex, Neil and Matt were each very well spoken. Alex also has a good write-up on his Pythian blog.

After the debate panel I slipped over to hear Alex’s 11g New Features presentation (which was actually Christo‘s presentation but Alex delivered it since Christo wasn’t able to fly over from Dubai). The presentation went great even though Alex hadn’t written it himself and I snapped this pic with my cell phone while I was there.

Alex’s session was the last one of the conference, so afterwards we grabbed a bite to eat and spent some time catching up. Dan joined us a bit later and then we made our way to the airport to have dinner and catch our respective flights. Thanks for the great company guys!

RAC 11g/VMware Lab

One last thing: on Wednesday we had a lab session where we went through the step-by-step process of setting up 11g RAC in a VMware Server virtual environment with OEL5 and ASM. I have uploaded the class to the publications section of this website.

If you have a 2Ghz processor, 2GB of memory and 20G of disk space, then you can download all of the software for free and try out this lab for yourself:

• VMware Server – http://www.vmware.com/download/server/
• Oracle Enterprise Linux (OEL) 5 – http://edelivery.oracle.com/linux
• Oracle Database Enterprise Edition 11g – http://www.oracle.com/technology/software/products/database/index.html

Next week sometime I expect to upload the instructions from today’s Hands-on Lab (11g RAC on VMware with ASM and OEL5). I want to clean it up a bit first. For anyone who didn’t hear the story, I heard on Sunday evening that they had a room with about 50 computers which was going to sit empty during a few timeslots. And to me that seemed like a tragedy – how often do I wish I could have a chance to try something new with a bit of guidance and without worrying about hosing my laptop?! So I decided to write a hands-on lab for 11gRAC/VMware. I pretty much spent all day yesterday putting it together but it came out great!

Also I’m thinking about repeating that RAC/VMware lab around Chicago sometime… is there anyone around the Chicago area who might be interested in something like this?

Guess I should also mention that I’ve been rather enjoying myself at Collaborate so far too! (Even though I spent pretty much all day yesterday making that Hands-on Lab…) Got a chance to meet Peter Scott Monday night – that was fun! Somehow he spotted my name badge and then I got to finally put a face to someone I’d only known as a blogger. :)