Last week Dan broke the news that we’re bringing RAC Attack back to Collaborate! We’ve run this workshop several times now: it’s gotten better every time and we’ve always received overwhelmingly positive feedback. This is going to be a great workshop in Orlando that you don’t want to miss!

A Short History

We first ran the RAC Attack last year at Collaborate in Denver. On the first day of Collaborate we heard that one of the computer labs was going to be sitting empty for several hours on Wednesday. Being such a fervent believer in hands-on learning, this sounded like a terrible waste! So at the last minute we arranged to deliver the RAC Attack hands-on lab during the open slot. If I remember right, we had something like 20 people and everyone seemed to really come away having learned something new. The lab focused on getting a running system.

In August when we ran RAC Attack as a standalone workshop in Chicago. In addition to having several excellent speakers, we dramatically expanded the scope of the lab to include topics like backup & recovery, services, and clusterware. During the full day allocated, only a few people completed every possible exercise.

Five Reasons To Sign Up

This year at Collaborate we’re doing it again. The lab will again be expanded and improved when we bring it to Orlando. Furthermore, I’m taking requests - if there’s a topic you’d like to learn about then leave a comment and I’ll try to get it included! I’m not sure how quickly the session will fill, but I know that there are a limited number of seats - so you should sign up soon.

Here are five reasons I think you should come:

1. Get exposure to and practice on all of the latest software, including VMware Server 2.0.1, Oracle Enterprise Linux (same as RedHat) 5.3, and Oracle RAC 11.1.0.7

2. There are hands-on challenges for every experience level. Meticulous step-by-step guide for beginners (with the choice to skip difficult steps), start-from-scratch or instant running environment for masters.

3. Learn from professionals with years of RAC experience. Several of us will be on-hand to answer questions and assist with the labs. If you have never touched a cluster database then we are here to help you complete the labs and understand what you’re doing!

4. Choose your own adventure. ASM or OCFS2. Parallel SQL or clusterware callouts. Got something specific you want to learn about? Leave a comment and I’ll try to get it included.

5. Walk out with a lab binder that tells you how to do it all on your own laptop after you get home. It won’t be possible to finish all the labs during the timeslot we have at Collaborate, but you can finish them later!

Finances are tight and many companies are cutting training budgets - but remember that generally speaking, Collaborate is one of the best value opportunities for Oracle training that’s available in the United States. You have world-class educational sessions and networking opportunities, complimented by top notch deep-dive university seminars such as RAC Attack. (Well… I’m a little biased about this session!) And with a slightly greater degree of independence you won’t just hear the corporate messages - it’ll be a little easier to find real-life users telling the great, the good, the bad and the ugly.

Hope to see you there!

In the last post I showed mathematically how ASM calculates the free usable space that it displays. However one of the first questions I received after presenting that internally was from someone concerned about all of the unusable space being saved just in case we lose a disk. (In my worked example it was 1/6 of the disk capacity! And what if we don’t care about restoring redundancy before we replace the failed disk?)

But it turns out that ASM actually handles this quite nicely - the “usable space” is used only for display and alerting. If a datafile needs to autoextend or if an archivelog needs to be created then ASM will keep using space until it completely exhausts everything physically available. It will go ahead and use the “reserved space” and just report a negative amount of free space! Here’s the output of my test confirming this behavior:

SQL> alter database datafile
2>  '+TEST01/js1_rh5lab20/datafile/filler.256.681667743' resize 18G;

Database altered.

.

rh5lab20$ asmcmd dgs
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB
MOUNTED  NORMAL  N         512   4096  1048576     38561     1529

  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
             7844           -3157              0  TEST01/

.

rh5lab20$ asmdf
Filesystem        SizeKB     UsedKB    AvailKB    Use% Mounted on
+TEST01         15727104   18959872   -3232768 120.56% TEST01

There is also a second way to get negative usable space in ASM. If you do experience a failure, then ASM will restore your redundancy level making new copies of the data that was on the failed disk(s). It will use the “reserved space” on your disks. After the rebuild is complete, ASM will recalculate the amount of reserved space it needs to protect from a second failure. If you were using most of the space before the first failure then ASM might report a negative amount of usable space after the rebuild.

It’s important for you to remember that negative free space doesn’t mean your data isn’t safe - you just won’t be able to restore redundancy after the failure. In other words, if you have negative free space then you cannot physically survive two subsequent failures without data loss (or three with high redundancy/triple mirroring). But ASM allows the negative numbers because it’s smart enough not to hang your database when it can be avoided (for example if archivelogs fill up the reported usable space).

This is quick and dirty. I hope to get a more polished write-up of what it means and how it works, but since I’m so busy right now I don’t know if I’ll get to it soon. In the meantime someone just might find this useful or informative so I’m just going to put it out there as it is.

This is an worked-through example of how ASM reports free space, taking into consideration “hot spare capacity” that is reserved to restore redundancy after the largest failure group dies.

ASM has 46 local disks attached to 6 controllers
- two controllers have 6 disks, other four have 8
- ASM has failure groups by controller

===========================================================================

Each disk is partitioned. Get the size of the ASM partition according to
the Linux kernel. All partitions are the same size.

# sfdisk -s /dev/sdav1
401624968

Now get the size info from ASM.

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB
MOUNTED  NORMAL  N         512   4096  1048576  18041706  18041326
Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
        3137688         7451819              0  DATA01/

===========================================================================

-- size of individual disk in MB is kernel-reported KB divided by 1024
401624968 /1024 = 392211.8828125 M

-- ASM can't use partial AU's so truncate and we get Total_MB
392211 * 46 = 18041706

-- largest failure group [controller] has 8 disks so get Req_mir_free_MB
     (amount of "hot spare" capacity that is reserved to restore redundancy
     after a controller failure)
392211 * 8 = 3137688

-- asm reports that some space is used; use "Free_MB" to see total free space
= 18041326

-- with normal-mirroring, usable space is (totalfree-spare)/2
(18041326 -  3137688) / 2 = 7451819

At the company where I’m working right now, I’m part of an architecture effort to come up with our standard design for RAC on Linux across the firm. There will be dozens or possibly hundreds of deployments globally using the design we settle on.

We’re internally debating whether or not we should include OCFS2 in this design right now, and I’m curious if anyone has arguments one way or the other to share. Our standard design on Solaris does utilize a cluster filesystem and we would welcome a similar design, but there are some concerns about the readiness, stability and future of OCFS2.

OCFS2 is being considered for these four use cases:

• database binaries (vs local files or NFS)
• diag top (11g) or admin tree (10g) (vs local files or NFS)
• archived logs
• backups

Other files will be stored in ASM.

I have seen mention in blogs such as http://bigdaveroberts.wordpress.com/ of something called ASMFS in 11gR2 and I’m wondering - will this feature (if included) have any impact on Oracle’s commitment to OCFS2 development? Could Oracle conceivably develop a whole new cluster filesystem and put their full weight behind it as they did for ASM storage, leaving OCFS2 as a lower priority for new features and improvements? Has Oracle demonstrated significant commitment to OCFS2 development and support in the past, and is this a mature enough technology for wide-scale deployment?

Just looking for opinions. :)

Right now I’m working on some internal documentation and referencing Oracle 11g’s Storage Administrators Guide (ASM manual). And I think there’s a bit of a contradiction here about the stripe size. ASM uses a variable “data extent” size when allocating space for files (like any sane filesystem) - the first 20000 data extents are equal to the diskgroup allocation unit (AU), the next 20000 are AU*8, etc. But it’s a little unclear to me how the stripe size is determined. Seems to me that you’d stripe “data extents” rather than allocation units and I think I’ve even heard before that “extents are mirrored [and presumably striped]” in ASM. In fact the illustration in the manual shows the stripes in exactly that manner:

ASM Extents

However the manual states that the opposite is true - striping based on AU instead of data extent - in two different places on that page. › Read more

This article isn’t directly database-related, but I think it’s a great software engineering topic so it seemed worth writing about. Right now I’m involved in a project that involves releasing software packages of a few different flavors. Some of them are other people’s software that we’re re-packaging (like oracle database binaries) and some are code that was completely written and is maintained in-house. And as we’ve been working through the software development process this question has recently come to mind: how should we assign version numbers?

Actually it’s part of a bigger question of how we develop elegant software - other related design issues in this project are clear end-to-end traceability (requirement -> source code version -> released binary package), solid change control, integrated testing framework and readiness for automation (build and test). But the one I’m interested in right now is this: how can I develop a robust version numbering strategy?

First of all, there really isn’t a one-size-fits-all “correct” version numbering scheme. The best way to choose version numbers is influenced by your software development philosophy (and resultant development lifecycle), project management approach and development team. (Globally distributed informal volunteer network or local engineers on payroll?) So the best solution for my current project might not be the best solution for my next project.

But lets take a closer look at this project. First, a little background on the art of version numbering. › Read more

Happy Thanksgiving to everyone in the US! And happy belated thanksgiving to everyone in Canada since you celebrated back at the beginning of October. :)

I’ve been doing a lot of scripting work lately. Although I can’t write about everything I’m doing, I would like to post a pattern that I thought could be useful to other folks. I needed to update some properties in the LISTENER.ORA file from a shell script. The problem is that basically you have to clobber it or properly parse it! Even though I initialized this listener file myself, there is always the possibility that a DBA could manually edit it - so I didn’t want to clobber the contents. That means that I had to parse it.

So how do you parse a listener file from a shell script? I came up with a nifty way to tokenize and parse the file through a simple awk script combined with sed - and it seemed rather useful so I refactored it a few times to apply to more general use cases. I think someone else could use something like this too so I’m posting it - do me a favor and keep the reference to ardentperf in there if you copy the code.

There’s two functions that are basically the same pattern: the first will read an attribute and the second will set one. I’ve included some examples at the bottom of the post. I’ve tried to make the awk code portable but so far I’ve only tested it with gawk and mawk on a few flavors of Linux.
› Read more

Before I started consulting, I was an Oracle engineer in a very large software development organization. The company had a number of major products and the one I worked with was used by hospitals and radiology offices world-wide. (These guys are one of the biggest companies worldwide in the field.) Our product included the hardware and software; you could order it with Sun or HP hardware (Solaris or Linux). It had an Oracle backend and a web-based middle tier built with lots of C++ and Java code.

Any software engineer who has worked on large projects - industry or community - can tell you the importance of solid change control processes. So… since everything for the build had to be checked into clearcase… yes, we checked Oracle 10g into the repo. A 2G tarball. And whenever there was a 1K patch to the oracle install? A brand new 2G tarball. The clearcase guys loved me.

And that was how I started thinking about an automated build process. I don’t need to check B24792-01_1of5.zip into the repository because it’s straight off edelivery. I can skip p7150622_10203_Linux-x86-64.zip since it’s direct from metalink. The only thing missing was a solid, simple, flexible program for automating the oracle install, patchset, CPUs and oneoffs - taking Oracle’s official bits as input.

Anyone else out there who could use a program like this? How about for rapid provisioning of servers? (Like all that grid buzz.) Grid Control does some basic stuff (if you can make it work) and there are advanced kits for the big data centers - but there have to be more people than just me who would love to have this program.

Proposing orainstall

That’s why I’m proposing to write a bit of software called orainstall. It would be script-based and cross-platform and intended for community use. I have some ideas for a design but I’m really interested to hear how you might use something like this and what features might be useful to you. Do you think it’s a good design? I’d also be interested in hearing if you’d be interested to use it or to help test it.

The program would be launched by running › Read more

Is asmlib obsolete on a modern Linux system? I’m still undecided but starting to lean toward “yes”.

Everybody knows that asmlib was very useful when it was first introduced with Oracle 10.1 to simplify a host of issues on Linux: direct async device access without raw devices, file permissions & ownership without custom code, and persistent device naming without devlabel.

But I’m now involved in setting some standards to be used across a large organization for Oracle 10.2 on RHEL5 and I’m wondering if there’s still a case for using asmlib. So I did a little trolling for info - which was suprisingly sparse. Had a hard time finding much, but after a lot of digging I think I’ve compiled a useful bit of information about benefits and drawbacks.

Benefits

It seems to me that the ASMLIB API was originally introduced to do more than just simplify file permissions - sounds like it was an alternative I/O API to the standard unix one, allowing ASM to access the underlying storage more efficiently and completely. I don’t think it’s just an “extra layer” - it’s an alternative code path to the std unix I/O libs. Like an ODM for block devices - and the idea was that there could be additional vendor implementations. And Oracle released an initial generic implementation on Linux under the GPL.
› Read more

So Collaborate is over and I’m back in Chicago… home sweet home. I thoroughly enjoyed the week in Denver, in spite of the snow! Thursday, the last day, was especially fun.

First was a panel debate “To RAC or Not To RAC: What’s Best for HA.” Dan Norris invited me to participate in this panel along with Alex Gorbachev (Pythian), Neil Greene (Predictive Technologies) and Matt Zito (GridApp) - I certainly felt privileged to take part. Here are a few of the things that I took away from the debate… feel free to discuss:
› Read more