A Short History

We first ran the RAC Attack last year at Collaborate in Denver. On the first day of Collaborate we heard that one of the computer labs was going to be sitting empty for several hours on Wednesday. Being such a fervent believer in hands-on learning, this sounded like a terrible waste! So at the last minute we arranged to deliver the RAC Attack hands-on lab during the open slot. If I remember right, we had something like 20 people and everyone seemed to really come away having learned something new. The lab focused on getting a running system.

In August when we ran RAC Attack as a standalone workshop in Chicago. In addition to having several excellent speakers, we dramatically expanded the scope of the lab to include topics like backup & recovery, services, and clusterware. During the full day allocated, only a few people completed every possible exercise.

Five Reasons To Sign Up

This year at Collaborate we’re doing it again. The lab will again be expanded and improved when we bring it to Orlando. Furthermore, I’m taking requests – if there’s a topic you’d like to learn about then leave a comment and I’ll try to get it included! I’m not sure how quickly the session will fill, but I know that there are a limited number of seats – so you should sign up soon.

Here are five reasons I think you should come:

1. Get exposure to and practice on all of the latest software, including VMware Server 2.0.1, Oracle Enterprise Linux (same as RedHat) 5.3, and Oracle RAC 11.1.0.7

2. There are hands-on challenges for every experience level. Meticulous step-by-step guide for beginners (with the choice to skip difficult steps), start-from-scratch or instant running environment for masters.

3. Learn from professionals with years of RAC experience. Several of us will be on-hand to answer questions and assist with the labs. If you have never touched a cluster database then we are here to help you complete the labs and understand what you’re doing!

4. Choose your own adventure. ASM or OCFS2. Parallel SQL or clusterware callouts. Got something specific you want to learn about? Leave a comment and I’ll try to get it included.

5. Walk out with a lab binder that tells you how to do it all on your own laptop after you get home. It won’t be possible to finish all the labs during the timeslot we have at Collaborate, but you can finish them later!

Finances are tight and many companies are cutting training budgets – but remember that generally speaking, Collaborate is one of the best value opportunities for Oracle training that’s available in the United States. You have world-class educational sessions and networking opportunities, complimented by top notch deep-dive university seminars such as RAC Attack. (Well… I’m a little biased about this session!) And with a slightly greater degree of independence you won’t just hear the corporate messages – it’ll be a little easier to find real-life users telling the great, the good, the bad and the ugly.

Hope to see you there!

But it turns out that ASM actually handles this quite nicely – the “usable space” is used only for display and alerting. If a datafile needs to autoextend or if an archivelog needs to be created then ASM will keep using space until it completely exhausts everything physically available. It will go ahead and use the “reserved space” and just report a negative amount of free space! Here’s the output of my test confirming this behavior:

SQL> alter database datafile
2>  '+TEST01/js1_rh5lab20/datafile/filler.256.681667743' resize 18G;

Database altered.

.

rh5lab20$ asmcmd dgs
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB
MOUNTED  NORMAL  N         512   4096  1048576     38561     1529

  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
             7844           -3157              0  TEST01/

.

rh5lab20$ asmdf
Filesystem        SizeKB     UsedKB    AvailKB    Use% Mounted on
+TEST01         15727104   18959872   -3232768 120.56% TEST01

There is also a second way to get negative usable space in ASM. If you do experience a failure, then ASM will restore your redundancy level making new copies of the data that was on the failed disk(s). It will use the “reserved space” on your disks. After the rebuild is complete, ASM will recalculate the amount of reserved space it needs to protect from a second failure. If you were using most of the space before the first failure then ASM might report a negative amount of usable space after the rebuild.

It’s important for you to remember that negative free space doesn’t mean your data isn’t safe – you just won’t be able to restore redundancy after the failure. In other words, if you have negative free space then you cannot physically survive two subsequent failures without data loss (or three with high redundancy/triple mirroring). But ASM allows the negative numbers because it’s smart enough not to hang your database when it can be avoided (for example if archivelogs fill up the reported usable space).

This is an worked-through example of how ASM reports free space, taking into consideration “hot spare capacity” that is reserved to restore redundancy after the largest failure group dies.

ASM has 46 local disks attached to 6 controllers
- two controllers have 6 disks, other four have 8
- ASM has failure groups by controller

===========================================================================

Each disk is partitioned. Get the size of the ASM partition according to
the Linux kernel. All partitions are the same size.

# sfdisk -s /dev/sdav1
401624968

Now get the size info from ASM.

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB
MOUNTED  NORMAL  N         512   4096  1048576  18041706  18041326
Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
        3137688         7451819              0  DATA01/

===========================================================================

-- size of individual disk in MB is kernel-reported KB divided by 1024
401624968 /1024 = 392211.8828125 M

-- ASM can't use partial AU's so truncate and we get Total_MB
392211 * 46 = 18041706

-- largest failure group [controller] has 8 disks so get Req_mir_free_MB
     (amount of "hot spare" capacity that is reserved to restore redundancy
     after a controller failure)
392211 * 8 = 3137688

-- asm reports that some space is used; use "Free_MB" to see total free space
= 18041326

-- with normal-mirroring, usable space is (totalfree-spare)/2
(18041326 -  3137688) / 2 = 7451819

We’re internally debating whether or not we should include OCFS2 in this design right now, and I’m curious if anyone has arguments one way or the other to share. Our standard design on Solaris does utilize a cluster filesystem and we would welcome a similar design, but there are some concerns about the readiness, stability and future of OCFS2.

OCFS2 is being considered for these four use cases:

• database binaries (vs local files or NFS)
• diag top (11g) or admin tree (10g) (vs local files or NFS)
• archived logs
• backups

Other files will be stored in ASM.

I have seen mention in blogs such as http://bigdaveroberts.wordpress.com/ of something called ASMFS in 11gR2 and I’m wondering – will this feature (if included) have any impact on Oracle’s commitment to OCFS2 development? Could Oracle conceivably develop a whole new cluster filesystem and put their full weight behind it as they did for ASM storage, leaving OCFS2 as a lower priority for new features and improvements? Has Oracle demonstrated significant commitment to OCFS2 development and support in the past, and is this a mature enough technology for wide-scale deployment?

Just looking for opinions. :)

ASM Extents

However the manual states that the opposite is true – striping based on AU instead of data extent – in two different places on that page. First, just before the illustration:

The ASM coarse striping is always equal to the disk group AU size, but fine striping size always remains 128KB in any configuration (not shown in the figure). The AU size is determined at creation time with the allocation unit size (AU_SIZE) disk group attribute. The values can be 1, 2, 4, 8, 16, 32, and 64 MB.

And again right after it:

To stripe data, ASM separates files into stripes and spreads data evenly across all of the disks in a disk group. The stripes are equal in size to the effective AU. The coarse-grained stripe size is always equal to the AU size. The fine-grained stripe size always equals 128 KB; this provides lower I/O latency for small I/O operations such as redo log writes.

As far as I can tell, it seems to be a contradiction in the manual. I commented on the documentation page and I’ve opened an SR; I’ll post an update when I get a reply.

Update 4:45pm:

The SR got answered very quickly (less than an hour) – and said that striping is based on the data extent size. The usage of the term “allocation unit” was what I found confusing (even in the SR resolution)… but at least we have an answer. :)

Suppose you have a diskgroup with 1MB AU size. After a particular file has 20000 extents, the allocation unit for that file becomes 8MB.

allocation unit = extent size
40000>extent>20000 => FILE_AU=DISKGROUP_AU*8.

Here’s the definitive quote from the SR: “[After 20000 extents] the 8meg AU will be striped, with the first 8meg AU on disk one, the second 8meg AU on disk 2 and so on.”

This would apply for both striping and mirroring. My gut feeling was right; I was just a bit confused by the definitions of extents and allocation units in the manual (which are incorrect).

Actually it’s part of a bigger question of how we develop elegant software – other related design issues in this project are clear end-to-end traceability (requirement -> source code version -> released binary package), solid change control, integrated testing framework and readiness for automation (build and test). But the one I’m interested in right now is this: how can I develop a robust version numbering strategy?

First of all, there really isn’t a one-size-fits-all “correct” version numbering scheme. The best way to choose version numbers is influenced by your software development philosophy (and resultant development lifecycle), project management approach and development team. (Globally distributed informal volunteer network or local engineers on payroll?) So the best solution for my current project might not be the best solution for my next project.

But lets take a closer look at this project. First, a little background on the art of version numbering. In the open-source world, many projects follow a three-digit version: MAJOR.MINOR.PATCH. Also, it’s clear that the version number is closely tied to the development lifecycle. You can find plenty of example discussion from open-source projects such as the Linux kernel, Linux desktop distros, Eclipse, Apache APR and FarCry CMS. Before the 2.6 series the linux kernel used an even-odd scheme to distinguish between unstable and stable releases but that has now been abandoned and most people seem to regard this as a good thing.

The best overview of the topic that I could find was the wikipedia article on Software Versioning. The section about pre-release versions does a good job of describing one requirement of my current project. I also found the section about internal version numbers interesting; I’ve worked on projects in the past with that strategy.

I found an article on linux.com written by Nathan Willis a few years ago offering some principles. I agree with his first and third suggestions: “Pick a numbering scheme, then don’t change it. Ever.” and “Make friends with infinity.” Best to think this thing out at the front end of a project (before 1.1) and you shouldn’t be bothered by big version numbers. However I don’t buy his second point about the meaning of the decimal; I think most everyone today is used to the meaning of the decimal in version numbers.

For some alternate viewpoints I came across something Zack Weinberg wrote back in 2002. He states strongly that the decimal point isn’t mathematical, which I agree with. However contrary to his article, in our development lifecycle the test releases do need version numbers. But he has an interesting suggesting of using date-based numbers for development snapshots – that’s an interesting idea which I’ll have to think about. His list if ways to do it wrong has a good point about distinguishing between version x and version x.0 although I’m not in full agreement with the other list items. I actually find the GNU suggestion for numbering test versions very interesting – I’ll have to give that some thouht. (Tests versions for 4.6 are numbered 4.5.90 – 4.5.99)

Ultimately I don’t think either of these two sets of guidelines is robust enough for me – especially for repackaging the oracle database binaries. Some of these might seem obvious, but I thought a good place to start might be by listing what I would consider to be requirements for an elegant and robust version numbering scheme:

1. elegant, clear, understandable version numbers that are as simple as possible
2. support for single-digit or double-digit versions (1,2,3,4 or 1.0, 1.1, 1.2 ,2.0)
3. versions for pre-release builds such as test builds (for the quality process) and early-access beta builds.
4. very clear difference between pre-release and release versions, so someone doesn’t mistakenly install a pre-release build when they want a production build.
5. names need to sort correctly – pre-release before release. it’s critical that RPM and RHN sort them right; it would be nice if unix “ls” did too.
6. support for continuing development. after release 1.0, the 2.0-beta version should sort after release 1.0 and before release 2.0
7. support for special branch versions – version 1.0+patch12345, where patch12345 is not included in version 2.0 which has different patches
8. clear difference between pre-release and release versions on special branches.

I have to be honest – this is tough. I’ll be giving it some thought over the next few weeks. Let me know if you have any suggestions. :)

I’ve been doing a lot of scripting work lately. Although I can’t write about everything I’m doing, I would like to post a pattern that I thought could be useful to other folks. I needed to update some properties in the LISTENER.ORA file from a shell script. The problem is that basically you have to clobber it or properly parse it! Even though I initialized this listener file myself, there is always the possibility that a DBA could manually edit it – so I didn’t want to clobber the contents. That means that I had to parse it.

So how do you parse a listener file from a shell script? I came up with a nifty way to tokenize and parse the file through a simple awk script combined with sed – and it seemed rather useful so I refactored it a few times to apply to more general use cases. I think someone else could use something like this too so I’m posting it – do me a favor and keep the reference to ardentperf in there if you copy the code.

There’s two functions that are basically the same pattern: the first will read an attribute and the second will set one. I’ve included some examples at the bottom of the post. I’ve tried to make the awk code portable but so far I’ve only tested it with gawk and mawk on a few flavors of Linux.

Reading an Attribute

# get a property from stdin (listener.ora format) for a specific listener and
# print it on stdout. this awk script will tokenize the listener file so that
# it can find the property no matter what format the file is in. it will only
# print the first match (for example an address).
#   param 1: top-level property filter (e.g. listener name)
#   param 2: leaf property to get (e.g. host, port, protocol)
# by Jeremy Schneider - ardentperf.com
getlistenerproperty() {
  sed -e 's/=/`=/g' -e 's/(/`(/g' -e 's/)/`)/g'|awk 'BEGIN{level=1} {
    wrote=0
    split($0,tokens,"`")
    i=1; while(i in tokens) {
      if(tokens[i]~"^[(]") level++
      if(tokens[i]~"^[)]") level--
      if(level==1&&i==1&&tokens[i]~"[A-Za-z]") TOP=tokens[i]
      if(toupper(TOP)~toupper("^[ \t]*'"$1"'[ \t]*$")) {
        if(propertylvl) {
          if(level>=propertylvl) {
            if(tokens[i]~"^="&&level==propertylvl) printf substr(tokens[i],2)
              else printf tokens[i]
            wrote=1
          } else propertylvl=0
          found=1
        }
        if(!found&&toupper(tokens[i])~toupper("^[(]?[ \t]*'"$2"'[ \t]*$")) propertylvl=level
      }
      i++
    }
    if(wrote) printf "\n"
  }'
}

Setting an Attribute

This uses essentially the same pattern but it sets an attribute instead of reading one. It outputs the new LISTENER.ORA file on stdout.

# read stdin (listener.ora format) and change a property for a specific listener,
# writing the updated file on stdout. the function will tokenize the listener file
# on stdin so that it can find the property no matter what format the file is in.
# it will only replace the first match (for example an address).
#   param 1: top-level property filter (e.g. listener name)
#   param 2: leaf property to set (e.g. host, port, protocol, address, etc)
#   param 3: new value of the property
# by Jeremy Schneider - ardentperf.com
setlistenerproperty() {
  sed -e 's/=/`=/g' -e 's/(/`(/g' -e 's/)/`)/g'|awk 'BEGIN{level=1} {
    wrote=0
    split($0,tokens,"`")
    i=1; while(i in tokens) {
      if(tokens[i]~"^[(]") level++
      if(tokens[i]~"^[)]") level--
      if(level==1&&i==1&&tokens[i]~"[A-Za-z]") TOP=tokens[i]
      if(toupper(TOP)~toupper("^[ \t]*'"$1"'[ \t]*$")) {
        if(propertylvl) {
          if(level>=propertylvl) {
            if(!found) {
              if(tokens[i]~"^="&&level==propertylvl) printf "= '$3'"
                else printf "'$3'"
              wrote=1
            }
          } else { printf tokens[i]; wrote=1; propertylvl=0 }
          found=1
        } else { printf tokens[i]; wrote=1 }
        if(!found&&toupper(tokens[i])~toupper("^[(]?[ \t]*'"$2"'[ \t]*$")) propertylvl=level
      } else { printf tokens[i]; wrote=1 }
      i++
    }
    if(wrote||i==1) printf "\n"
  }'
}

Examples

First I’ll setup a sample LISTENER.ORA file that I can use to demonstrate a few different uses:

jeremy@debianvm:~/lsnr$ cat list-ardent.ora
TRACE_LEVEL_ARDENTL1 = ADMIN
TRACE_FILE_ARDENTL1 = listener
TRACE_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/trace
LOG_FILE_ARDENTL1 = listener
LOG_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/log
LOGGING_ARDENTL1 = ON

#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )

SID_LIST_ARDENTL1 =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = JTEST1)
      (SID_NAME = JTEST1)
      (ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
    )
  )

WALLET_LOCATION = (
  SOURCE=
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY=/home/oracle/Wallet_server))
  )

Now lets look at a few examples of reading properties. Note that – by design – it only reads the first match.

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 host
 10.9.8.33
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 port
 1522
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 address
 (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522)
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 description

    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty ardentl1 ardentl1

  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.9.8.33)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
jeremy@debianvm:~/lsnr$ cat list-ardent.ora|getlistenerproperty log_directory_ardentl1 log_directory_ardentl1
 /u01/app/oracle/product/10.2.0/db_1/network/log
jeremy@debianvm:~/lsnr$ 

And finally, a few examples of changing properties. Note that the entire file is always displayed but I’ve only shown the relevant excerpts.

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|setlistenerproperty ardentl1 host 1.2.3.4
...
#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 1.2.3.4)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
...

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|setlistenerproperty ardentl1 address "(HOST=1.2.3.4)(PORT=999)"
...
#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (HOST=1.2.3.4)(PORT=999))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
...

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|setlistenerproperty logging_ardentl1 logging_ardentl1 OFF
TRACE_LEVEL_ARDENTL1 = ADMIN
TRACE_FILE_ARDENTL1 = listener
TRACE_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/trace
LOG_FILE_ARDENTL1 = listener
LOG_DIRECTORY_ARDENTL1 = /u01/app/oracle/product/10.2.0/db_1/network/log
LOGGING_ARDENTL1 = OFF

#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
...

jeremy@debianvm:~/lsnr$ cat list-ardent.ora|
> setlistenerproperty ardentl1 host 1.2.3.4|
> setlistenerproperty ardentl1 port 9999
...
#SSL_CLIENT_AUTHENTICATION = FALSE
ARDENTL1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 1.2.3.4)(PORT = 9999))
    (ADDRESS = (PROTOCOL = TCPS)(HOST = 10.9.8.33)(PORT = 2485))
  )
...

And that’s about it. Leave me a comment and let me know if you find this useful. :)

Any software engineer who has worked on large projects – industry or community – can tell you the importance of solid change control processes. So… since everything for the build had to be checked into clearcase… yes, we checked Oracle 10g into the repo. A 2G tarball. And whenever there was a 1K patch to the oracle install? A brand new 2G tarball. The clearcase guys loved me.

And that was how I started thinking about an automated build process. I don’t need to check B24792-01_1of5.zip into the repository because it’s straight off edelivery. I can skip p7150622_10203_Linux-x86-64.zip since it’s direct from metalink. The only thing missing was a solid, simple, flexible program for automating the oracle install, patchset, CPUs and oneoffs – taking Oracle’s official bits as input.

Anyone else out there who could use a program like this? How about for rapid provisioning of servers? (Like all that grid buzz.) Grid Control does some basic stuff (if you can make it work) and there are advanced kits for the big data centers – but there have to be more people than just me who would love to have this program.

Proposing orainstall

That’s why I’m proposing to write a bit of software called orainstall. It would be script-based and cross-platform and intended for community use. I have some ideas for a design but I’m really interested to hear how you might use something like this and what features might be useful to you. Do you think it’s a good design? I’d also be interested in hearing if you’d be interested to use it or to help test it.

The program would be launched by running the main executable “orainstall”. The syntax could look something like this:

Command line syntax:
  ./orainstall <stepfile> [-oh <ORACLE_HOME>] [-tmp <ORAINST_TMP>]

Example:
  ./orainstall ora10203.sf -oh /u01/app/oracle/product/10.2.0/db_1 -tmp /u01/app/oracle/tmp

Return Value:
  0 - success
  1 - invalid stepfile
  2 - installation error
  3 - patching error

The Stepfile

What I’m thinking is that the install process is defined in an input file called the stepfile. The stepfile needs to accomodate a large variety of organizational requirements. Here’s my first draft for what a stepfile would look like:

# This stepfile instructs the oracle auto-installer how to setup the oracle
# home. Each line is executed in order from top to bottom.
#
# There are ten defined directives for this stepfile:
#   ORACLE_HOME - required
#     This will cause a search-replace of each response file to explicitly
#     define the oracle home. It must be defined before any open, install or
#     patch directives and it can only be defined once per stepfile.
#       Order of precedence:
#         1. command line
#         2. stepfile
#         3. environment variable
#   ORAINST_TMP - optional
#     Top-level temp directory for extracting contents of Oracle installers
#     and patches. Each installer or patch will be extracted into a
#     subdirectory of this temp location. It must be defined before any
#     open or patch directives. It can be defined multiple times in a
#     stepfile.
#       Order of precedence:
#         1. command line
#         2. stepfile
#         3. environment variable
#         4. "/tmp/orainstall"
#   OPEN /path/to/zip
#     Path to a zip file with database installer or major patchset which
#     requires use if OUI. The file is unzipped into a temporary location.
#     If several INSTALL directives are encountered in a row then they are
#     all unzipped into the same location; if there is another directive in
#     between then the next INSTALL is unzipped into a new temporary
#     location.
#   INSTALLPATH Disk1/runInstaller
#   INSTALLRSP install.rsp
#      "runInstaller -waitForCompletion -silent -responseFile <install.rsp>"
#     INSTALLPATH is optional - if not specified then the script will search
#     for it in the most recently created temp location. If there is more
#     than one runInstaller then it will use the first one it finds; search
#     order is undefined.
#     If no path is specified for the response file then orainstall will
#     search for it in this order:
#       1. current working directory (pwd) when orainstall was launched
#       2. same directory as orainstall binary (dirname $0)
#   OPATCH/path/p8000.zip
#     Path to special "OPatch" update - contents are unzipped directly to
#     ORACLE_HOME.
#   PATCH /path/p2345.zip
#     Path to a patch. Contents are unzipped to a temporary location then
#     applied by calling "opatch" from the ORACLE_HOME.
#   NPATCH /path/p1234.zip
#     Path to a patchset with multiple subpatches which need to be applied
#     with "napply" (such as a CPU). Contents are unzipped to temp location
#     then applied properly to the ORACLE_HOME.
#   CLEAN_TMP
#     Deletes all files from the current top-level temp directory.
#   REMOVE /path/to/file
#     Runs "rm -rf $ORACLE_HOME/path/to/file" - use carefully! Some sites
#     may require  removal of components for security requirements.
#
# If there are any errors (install or patch) then the process will fail.
#
#oracle_home /u01/app/oracle/product/10.2.0/db_1
orainst_tmp /u01/app/oracle/installers.tmp
open /net/rh5lab12/u10/stage/ora10201/B24792-01_1of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_2of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_3of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_4of5.zip
open /net/rh5lab12/u10/stage/ora10201/B24792-01_5of5.zip
installpath database/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/database10201.rsp
installpath companion/runInstaller
installrsp /net/rh5lab12/u10/stage/rsp/companion10201.rsp
open /net/rh5lab12/u10/stage/ora10203/p5337014_10203_Linux-x86-64.zip
installrsp /net/rh5lab12/u10/stage/rsp/patchset10203.rsp
clean_tmp
orainst_tmp /u01/app/oracle/patches
opatch /net/rh5lab12/u10/stage/patch10203/p6880880_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p5892355_10203_Linux-x86-64.zip
patch /net/rh5lab12/u10/stage/patch10203/p6455161_10203_Linux-x86-64.zip
npatch /net/rh5lab12/u10/stage/patch10203/p7150622_10203_Linux-x86-64.zip
remove /htmldb

Program Flow

The program flow would be pretty basic; it would validate the stepfile then execute it one line at a time. I thought of six things to validate before starting execution:

1. can read stepfile
2. one oracle_home, defined before open/install/patch, can create/write
3. orainst_tmp defined before open/install/patch, can create/write each
4. clean_tmp not defined before orainst_tmp
5. can read all open/install/patch input files
6. free space in oracle_home is greater than size of input files

Feedback

Maybe there aren’t as many people as I think who would use this. After all – most DBAs don’t install the database software very often; they just want to keep it running! But please let me know if it interests you. I’ll factor in the feedback I receive as I’m developing it.

Everybody knows that asmlib was very useful when it was first introduced with Oracle 10.1 to simplify a host of issues on Linux: direct async device access without raw devices, file permissions & ownership without custom code, and persistent device naming without devlabel.

But I’m now involved in setting some standards to be used across a large organization for Oracle 10.2 on RHEL5 and I’m wondering if there’s still a case for using asmlib. So I did a little trolling for info – which was suprisingly sparse. Had a hard time finding much, but after a lot of digging I think I’ve compiled a useful bit of information about benefits and drawbacks.

Benefits

It seems to me that the ASMLIB API was originally introduced to do more than just simplify file permissions – sounds like it was an alternative I/O API to the standard unix one, allowing ASM to access the underlying storage more efficiently and completely. I don’t think it’s just an “extra layer” – it’s an alternative code path to the std unix I/O libs. Like an ODM for block devices – and the idea was that there could be additional vendor implementations. And Oracle released an initial generic implementation on Linux under the GPL.

Some theoretical benefits of ASMLIB API:

• always uses direct, async i/o
• solves persistent device naming, even if underlying device moves across reboots
• solves file permissions and ownership
• reduced user mode to kernel mode context switches during I/O, possibly reducing CPU usage
• reduced file handle usage
• pass metadata such as I/O prioritization to storage device (don’t think this is implemented in the Linux version)

Drawbacks

However from what I can tell, the ASMLIB API didn’t quite catch on. Wim recommends it (as of Apr ‘07) but Closson doesn’t really talk about it on his blog but has one disparaging off-hand reference about pitying people who use it. There have been no vendor implementations that I could find. (But I wonder what’s going on now with exadata and “iDB” which has I/O prioritization and predicate pre-filtering – could they have used the API here?) Wim’s post on Oracle Forums says that there isn’t “much of an io performance benefit” added by the linux ASMLIB implementation. And the OTN page hasn’t been updated for almost two years.

A RedHat consultant I’m working with recommended against using it and one of his reasons was that it introduces an unnecessary additional layer and dependency in the kernel. There have been a small number of past bugs in asmlib and it doesn’t seem to be frequently updated (once this year according to the changelogs)… but then it’s very simple code (mostly in a single small file).

Conclusions

After all of this, I’m not convinced that it’s worth installing ASMLIB if you’re comfortable with udev. On Linux, udev is definitely a better solution to handle persistent device naming and permissions and I don’t see enough benefits in asmlib to outweigh the very slight additional overhead.

So right now my preference would be to only install asmlib on old systems – Oracle 10.1 or Linux 2.4 – but to use udev and block devices for 10.2 and newer on 2.6 kernels. But the case isn’t closed and I’d love to hear from anyone who agrees or disagrees… thoughts?

References:

• Wim’s brief remarks on Oracle Forums – http://forums.oracle.com/forums/thread.jspa?threadID=498215
• In-depth information from ASM reference book – http://books.google.co.uk/books?id=HB453L86Q6AC&pg=RA1-PA142&lpg=RA1-PA142&dq=asmlib+api&source=web&ots=bEP4epFbp5&sig=47aLUWW9o73Snr1tcnvz_1h3prU&hl=en&sa=X&oi=book_result&resnum=1&ct=result#PRA1-PA142,M1
• Uses submit_bio() call – source code – http://oss.oracle.com/viewvc/oracleasm/trunk/kernel/driver.c?view=markup
• cdos newsgroup thread with speculation about performance – http://groups.google.com/group/comp.databases.oracle.server/browse_thread/thread/905296d0aba00b84?pli=1
• tim hall was going to investigate though he didn’t publish results – http://www.oracle-base.com/blog/2006/05/01/asm-with-asmlib-or-raw-devices/
• kevin closson doesn’t say much but makes this off-hand remark – http://kevinclosson.wordpress.com/2006/11/02/dbwr-efficiency-aio-io-libraries-with-asm/
• short anonymous comment possibly by closson – http://archives.devshed.com/forums/databases-124/asmlib-1157660.html
• OTN official site for ASMLIB – http://www.oracle.com/technology/tech/linux/asmlib/index.html

First was a panel debate “To RAC or Not To RAC: What’s Best for HA.” Dan Norris invited me to participate in this panel along with Alex Gorbachev (Pythian), Neil Greene (Predictive Technologies) and Matt Zito (GridApp) – I certainly felt privileged to take part. Here are a few of the things that I took away from the debate… feel free to discuss:

• There are no silver bullets. And if there were, RAC wouldn’t be one. :) Many RAC implementation problems can be traced to poorly set expectations; knowing your requirements and RAC’s capabilities will go a long way towards making any project successful.

• RAC really shines when solving scalability problems. The simple act of moving an application from single-instance to RAC is more likely to improve throughput than service time. (Of course, improvement isn’t guaranteed: as Dan pointed out during our conversation it depends on where the bottlenecks are.)

• There are two major application design areas that need to be addressed when moving to RAC: (1) scalability and (2) connectivity. (There are other issues too but these are probably the biggest.) Regarding scalability, small inefficiencies will become major bottlenecks when you move your app to a cluster. Look out for any points of contention or serialization whether they’re logical (like an ID field that must be sequential) or physical (like free space management). On the connectivity front, there is both initial connection and load balancing to worry about.

• When it comes to HA, I personally think that failover clusters should get more serious consideration then they do. I think that there are essentially two differences between a failover cluster and RAC: (1) a few minutes of downtown each year, and (2) the prices for licenses – probably tens or hundreds of thousands of dollars minimum. You can even still use TAF with a failover cluster – it’s not much different than using it with Data Guard. The few apps that need extremely fast failover often can’t even wait a few minutes – they commonly run parallel systems in tandem to avoid failover altogether. Most other apps can tolerate an extra 60 seconds during failover, if we’re honest.

At any rate, I thought that the debate was informative and that Alex, Neil and Matt were each very well spoken. Alex also has a good write-up on his Pythian blog.

After the debate panel I slipped over to hear Alex’s 11g New Features presentation (which was actually Christo’s presentation but Alex delivered it since Christo wasn’t able to fly over from Dubai). The presentation went great even though Alex hadn’t written it himself and I snapped this pic with my cell phone while I was there.

Alex’s session was the last one of the conference, so afterwards we grabbed a bite to eat and spent some time catching up. Dan joined us a bit later and then we made our way to the airport to have dinner and catch our respective flights. Thanks for the great company guys!

RAC 11g/VMware Lab

One last thing: on Wednesday we had a lab session where we went through the step-by-step process of setting up 11g RAC in a VMware Server virtual environment with OEL5 and ASM. I have uploaded the class to the publications section of this website.

If you have a 2Ghz processor, 2GB of memory and 20G of disk space, then you can download all of the software for free and try out this lab for yourself:

• VMware Server – http://www.vmware.com/download/server/
• Oracle Enterprise Linux (OEL) 5 – http://edelivery.oracle.com/linux
• Oracle Database Enterprise Edition 11g – http://www.oracle.com/technology/software/products/database/index.html